在 jupyter/ipython notebook 中将命令行参数传递给 argv

Passing command line arguments to argv in jupyter/ipython notebook

提问by justadampaul

I'm wondering if it's possible to populate sys.argv(or some other structure) with command line arguments in a jupyter/ipython notebook, similar to how it's done through a python script.

我想知道是否可以sys.argv在 jupyter/ipython 笔记本中使用命令行参数填充（或其他结构），类似于通过 python 脚本完成的方式。

For instance, if I were to run a python script as follows:

例如，如果我要按如下方式运行 python 脚本：

python test.py False

python test.py False

Then sys.argvwould contain the argument False. But if I run a jupyter notebook in a similar manner:

然后sys.argv将包含参数False。但是如果我以类似的方式运行 jupyter notebook：

jupyter notebook test.ipynb False

jupyter notebook test.ipynb False

Then the command line argument gets lost. Is there any way to access this argument from within the notebook itself?

然后命令行参数丢失。有没有办法从笔记本内部访问这个参数？

采纳答案by Steve Molly

After a lot of looking around I found very cumbersome, custom libraries, but solved it with a few lines of code which I thought was pretty slick. I used nbconvert to end up with an html report as output that contains all graphics and markdown from the notebook, but accepts command line parameters just as always through a minimal python wrapper:

环顾四周后，我发现非常麻烦的自定义库，但我用几行代码解决了它，我认为这些代码非常巧妙。我使用 nbconvert 最终得到一个 html 报告作为输出，其中包含笔记本中的所有图形和降价，但像往常一样通过最小的 python 包装器接受命令行参数：

The python file test_args.py (which takes command line params as normal):

python 文件 test_args.py（它正常使用命令行参数）：

import sys,os
IPYNB_FILENAME = 'test_argv.ipynb'
CONFIG_FILENAME = '.config_ipynb'

def main(argv):
    with open(CONFIG_FILENAME,'w') as f:
        f.write(' '.join(argv))
    os.system('jupyter nbconvert --execute {:s} --to html'.format(IPYNB_FILENAME))
    return None

if __name__ == '__main__':
    main(sys.argv)

The notebook contains:

笔记本包含：

import sys,os,argparse
from IPython.display import HTML
CONFIG_FILE = '.config_ipynb'
if os.path.isfile(CONFIG_FILE):
    with open(CONFIG_FILE) as f:
        sys.argv = f.read().split()
else:
    sys.argv = ['test_args.py', 'input_file', '--int_param', '12']

parser = argparse.ArgumentParser()
parser.add_argument("input_file",help="Input image, directory, or npy.")
parser.add_argument("--int_param", type=int, default=4, help="an optional integer parameter.")
args = parser.parse_args()
p = args.int_param
print(args.input_file,p)

and I can run the python notebook with arguments parsed as usual:

我可以像往常一样运行带有解析参数的python笔记本：

python test_args.py my_input_file --int_param 12

I tend to paste the block with argparse calls into the python wrapper so that command line errors are caught by the python script and -h works properly.

我倾向于将带有 argparse 调用的块粘贴到 python 包装器中，以便 python 脚本捕获命令行错误并且 -h 正常工作。

回答by Piotr Czapla

There are two projects I've found that do what you ask for

我发现有两个项目可以满足您的要求

• Papermill, will add a cell to your notebook with arguments that you pass to it on the command line. So this is quite straightforward, you define your defaults in the first cell (the should have parameterstag)
• nbparameteriseit is a similar concept but you don't tag your cell with defaults, it has to be first.

• Papermill，将向您的笔记本添加一个单元格，其中包含您在命令行中传递给它的参数。所以这很简单，你在第一个单元格中定义你的默认值（应该有parameters标签）
• nbparameterise它是一个类似的概念，但你没有用默认值标记你的单元格，它必须是第一个。

Here is a good resource discussing the issue: https://github.com/jupyter/help/issues/218

这是讨论该问题的好资源：https: //github.com/jupyter/help/issues/218

回答by Spawnrider

I think this Gist may help you : https://gist.github.com/gbishop/acf40b86a9bca2d571fa

我认为这个要点可以帮助你：https: //gist.github.com/gbishop/acf40b86a9bca2d571fa

This is an attempt at a simple argument parser for mostly key=value pairs that can be used both on the command line and in IPython notebooks. It support query parameters in notebook URLs and a Run command for notebooks.

这是一个简单的参数解析器的尝试，主要用于键=值对，可以在命令行和 IPython 笔记本中使用。它支持笔记本 URL 中的查询参数和笔记本的运行命令。

回答by sngjuk

If you use iPython for testing, transforming argparse into class format can be a quick dummy solution like this.

如果您使用 iPython 进行测试，那么将 argparse 转换为类格式可能是一个像这样的快速虚拟解决方案。

class Args:
  data = './data/penn'
  model = 'LSTM'
  emsize = 200
  nhid = 200

args=Args()

Github pageoffers web transformation service. http://35.192.144.192:8000/arg2cls.html
Hope that it would be helpful for your testing. Jan 9/19 many bugs are fixed.

Transform argparse module into class format. Python3 is required.

Github 页面提供网络转换服务。http://35.192.144.192:8000/arg2cls.html
希望对您的测试有所帮助。2019 年 1 月 9 日修复了许多错误。

将 argparse 模块转换为类格式。需要 Python3。

python3 [arg2cls.py] [argparse_script.py]

then copy & paste class format to replace argparse functions.

然后复制和粘贴类格式以替换 argparse 函数。

#!/usr/bin/env python3
from collections import OrderedDict
import sys
import re
DBG = False

#add_argument(), set_defaults() only available.
ListStartPatt = re.compile(r'\s*\[.*')
ListStartPatt2 = re.compile(r'\).*\[.*') # list out of function scope.
ListPatt = re.compile(r'(\[.*?\])')
GbgPatt = re.compile(r'(.*?)\)[^\)]+') # for float('inf') cmplx.
GbgPatt2 = re.compile(r'(.*?)\).*') # general gbg, ? for non greedy.
LpRegex = re.compile(r'\({1,}\s{0,}')
RpRegex = re.compile(r'\s{0,}\){1,}')
PrRegex = re.compile(r'\((.*)(\))(?!.*\))') # from \( to last \).
CmRegex = re.compile(r'\s{0,},\s{0,}')
StrRegex = re.compile(r'\'(.*?)\'')

# Argument dict : {arg_name : value}
argDct=OrderedDict()

# process 'default=' value.
def default_value(tval, dtype=''):
  # string pattern.
  regres = StrRegex.match(tval) 
  if regres and not re.search('int|float|long|bool|complex', dtype):
    if DBG:
      print('default_value: str patt found')
    tval = regres.group(0)
    return tval

  # typed pattern.
  CommaSeparated = CmRegex.split(tval)[0]
  if DBG:
    print('comma sepearated value:', CommaSeparated)

  if ListStartPatt.match(CommaSeparated) and not ListStartPatt2.match(CommaSeparated):
    lres = ListPatt.search(tval)
    if lres:
      tval = lres.group(1)
    if DBG:
      print('list patt exist tval: ', tval)
  else :
    tval = CmRegex.split(tval)[0]
    if DBG:
      print('no list format tval: ', tval)

  # if default value is not like - int('inf') , remove characters after ')' garbage chars.
  ires = RpRegex.split(tval)[0]
  if not (re.search('int|float|long|bool|complex', ires) and re.search(r'[a-z]+\(',ires)):
    if DBG:
      print('not int("inf") format. Rp removed tval : ', tval)
    tval = re.split(r'\s{0,}\){1,}',tval)[0]
    gbg = GbgPatt2.search(tval)
    if gbg:
      tval = gbg.group(1)  
      if DBG:
        print('garbage exist & removed. tval : ', tval)

  # int('inf') patt.
  else:
    if DBG:
      print('type("inf") value garbaging!')
    gbg = GbgPatt.search(tval)
    if gbg:
      if DBG:
        print('garbage found, extract!')
      tval = gbg.group(1)

  return tval

# Handling add_argument()
def add_argument(arg_line):
  global argDct
  if DBG:
    print('\nin add_argument : **Pre regex: ', arg_line)

  '''    
  argument name
  '''
  # argname = DdRegex.split(arg_line)[1] # Dash or regex for arg name.
  argname = re.search('\'--(.*?)\'', arg_line)
  if not argname:
    argname = re.search('\'-+(.*?)\'', arg_line)

  # dest= keyword handling.
  dest = re.search(r',\s*dest\s*=(.*)', arg_line)
  if dest:
    dval = dest.group(1)
    dval = default_value(dval)
    argname = StrRegex.search(dval)

  # hyphen(-) to underscore(_)
  if argname:
    argname = argname.group(1).replace('-', '_')
  else :
    # naive str argname.
    sres = StrRegex.match(arg_line)
    if sres:
      argname = sres.group(1)
    if not argname:
      return # no argument name 

  '''
  check for syntaxes (type=, default=, required=, action=, help=, choices=)
  '''
  dtype = ''
  dres = re.search(r',\s*type\s*=\s*(.*)', arg_line)
  if dres:
    dtype = dres.group(1)
    dtype = CmRegex.split(dtype)[0]

  dfult = re.search(r',\s*default\s*=\s*(.*)', arg_line)
  rquird = re.search(r',\s*required\s*=\s*(.*)', arg_line)
  action = re.search(r',\s*action\s*=\s*(.*)', arg_line)
  hlp = re.search(r',\s*help\s*=\s*(.*)', arg_line)
  chice = re.search(r',\s*choices\s*=\s*(.*)', arg_line)

  # help message
  hlp_msg = ''
  if hlp:
    thl = hlp.group(1)
    if DBG:
      print('handling help=')
    hlp_msg = default_value(thl)
    if hlp_msg:
      hlp_msg = 'help='+hlp_msg

  # choice message
  choice_msg = ''
  if chice:
    tch = chice.group(1)
    if DBG:
      print('handling choices=')
    choice_msg = default_value(tch)
    if choice_msg:
      choice_msg = 'choices='+choice_msg+' '

  '''
  argument value
  '''
  # tval: argument value.
  tval = ''
  # default exist.
  if dfult:
    tval = dfult.group(1)
    tval = default_value(tval, dtype)
    if DBG:
      print('value determined : ', tval)

  # action or required syntaxes exist.
  elif action or rquird:
    if DBG:
      print('in action/required handling')
    msg_str = ''
    if action:
      tval = action.group(1)
      msg_str = 'action'
    elif rquird:
      tval = rquird.group(1)
      msg_str = 'required'

    tval = default_value(tval)
    tval = ' ** ' + msg_str + ' '+tval+'; '+choice_msg+ hlp_msg

  # no default, action, required.
  else : 
    argDct[argname] = ' ** default not found; '+choice_msg+ hlp_msg

  # value found.
  if tval:
    argDct[argname] = tval

# Handling set_defaults()
def set_defaults(arg_line):
  global argDct
  if DBG:
    print('\nin set_defaults arg_line: ', arg_line)

  # arguments to process.
  tv='' 
  # arguments of set_default()
  SetPatt = re.compile(r'(.+=.+\)?)')
  sres = SetPatt.match(arg_line)
  if sres:
    tv = sres.group(1)
    if DBG:
      print("setPatt res: ", tv)
    tv = re.sub(r'\s+','', tv)
    if DBG:
      print('\nset_default values: ', tv)

  # one arguemnt regex.
  SetArgPatt = re.compile(r',?([^=]+=)[^=,]+,?')
  # handling multiple set_default() arguments. (may have a bug)
  while True:
    tname=''
    tval =''
    tnv=''
    # func closed.
    if re.match(r',*\).*',tv):
      tv=''
      break
    if DBG:
      print('set_default remaining: ', tv)

    nres = SetArgPatt.match(tv)
    if nres:
      tname = nres.group(1)
      if len(tv.split(tname, 1)) > 1:
        tval = tv.split(tname,1)[1]
        tval = default_value(tval)
        tnv=tname+tval
        tname = tname.rsplit('=',1)[0]

      if DBG:
        print('set_default tnam: ', tname)
        print('set_default tval: ', tval)
      if tname:
        argDct[tname] = tval

      # split with processed argument.
      tv = tv.split(tnv)
      if len(tv) > 1:
        tv = tv[1]
      # no more value to process
      else:
        break

    # no arg=value pattern found.
    else:
      break

# Remove empty line & Concatenate line-separated syntax.
def preprocess(fname):
  try :
    with open(fname, 'r', encoding='UTF8') as f:
      txt = f.read()
      t = txt.splitlines(True)
      t = list( filter(None, t) )

      # remove empty line
      t = [x for x in t if not re.match(r'\s{0,}\n',x)]
      # concatenate multiple lined arguments.
      # empl : lines to be deleted from t[].
      empl = []
      for i in range(len(t)-1, 0, -1):
        if not re.search('add_argument|set_defaults', t[i]):
          t[i-1] += t[i]
          t[i-1]=re.sub(r'\n{0,}','',t[i-1])
          t[i-1]=re.sub(r'\s{1,}',' ',t[i-1])
          empl.append(t[i])

      for d in empl:
        t.remove(d)
      for i, line in enumerate(t):
        t[i] = line.replace('\"', '\'').split('parse_args()')[0]
      return t

  except IOError:
    print('IOError : no such file.', fname)
    sys.exit()

def transform(fname):
  # t : list() contains add_argument|set_defaults lines.
  arg_line_list = preprocess(fname)

  for i, arg_line in enumerate(arg_line_list):
    t = PrRegex.search(arg_line)

    if t:
      t = t.group(1) # t: content of add_argument Parentheses.
    else :
      continue # nothing to parse.

    if re.search(r'add_argument\s*\(', arg_line):
      add_argument(t)
    elif re.search(r'set_defaults\s*\(',arg_line):
      set_defaults(t)
    else :
      # Nothing to parse.
      continue

  print('\nclass Args:')
  for i in argDct:
    print(' ',i, '=', argDct[i])
  print()
  print('args=Args()')

def main():
  if len(sys.argv) <2:
    print('Usage : python arg2cls.py [target.py] [target2.py(optional)] ...')
    sys.exit(0)
  sys.argv.pop(0)

  #handling multiple file input.
  for fname in sys.argv:
    transform(fname)

if(__name__ == "__main__"):
  main()

回答by crytting

sys.argvyields a list, so I used

sys.argv产生 a list，所以我用

sys.argv.append('hello')

in a jupyter notebook, which allowed me to append extra members and pretend like I'm passing in arguments from the command line.

在 jupyter 笔记本中，它允许我附加额外的成员并假装我是从命令行传递参数。

回答by Thomas G.

A workaround is to make the jupyter notebook read the arguments from a file. From the command line, modify the file and run the notebook.

解决方法是让 jupyter notebook 从文件中读取参数。从命令行修改文件并运行笔记本。

回答by mataney

you can use Jupyter build-in magic command %runwithin the notebook.

您可以%run在笔记本中使用 Jupyter 内置魔术命令。

From this link, you can use:

从此链接，您可以使用：

%run -p [prof_opts] filename.py [args to program]

%run -p [prof_opts] filename.py [args to program]

Or something like %run -i script.py False

或者类似的东西 %run -i script.py False

Or if you are parsing the arguments %run -i script.py --flag1 False --flag2 True

或者，如果您正在解析参数 %run -i script.py --flag1 False --flag2 True