bash 如何使使用 call/Popen 调用的子进程继承环境变量

how to make subprocess called with call/Popen inherit environment variables

提问by Katie

First off, apologies for what I'm sure will be obvious is my rudimentary understanding of bash and shells and subprocesses.

首先，很抱歉，我对 bash、shell 和子进程的基本理解是显而易见的。

I am trying to use Python to automate calls to a program called Freesurfer (actually, the subprogram I'm calling is called recon-all.)

我正在尝试使用 Python 自动调用一个名为 Freesurfer 的程序（实际上，我正在调用的子程序称为 recon-all。）

If I were doing this directly at the command line, I'd "source" a script called mySetUpFreeSurfer.sh that does nothing but set three environment variables, and then "source" another script, FreeSurferEnv.sh. FreesurferEnv.sh doesn't seem to me to do anything but set a lot of environment variables and echo some stuff to the terminal, but it's more complicated than the other bash script, so I'm not sure of that.

如果我直接在命令行执行此操作，我会“获取”一个名为 mySetUpFreeSurfer.sh 的脚本，该脚本除了设置三个环境变量之外什么都不做，然后“获取”另一个脚本 FreeSurferEnv.sh。FreesurferEnv.sh 在我看来除了设置很多环境变量并将一些内容回显到终端之外并没有做任何事情，但它比其他 bash 脚本更复杂，所以我不确定。

Here is what I have right now:

这是我现在所拥有的：

from subprocess import Popen, PIPE, call, check_output
import os

root = "/media/foo/"

#I got this function from another Stack Overflow question.

def source(script, update=1):
    pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)
    data = pipe.communicate()[0]
    env = dict((line.split("=", 1) for line in data.splitlines()))
    if update:
        os.environ.update(env)
    return env

source('~/scripts/mySetUpFreeSurfer.sh')
source('/usr/local/freesurfer/FreeSurferEnv.sh')

for sub_dir in os.listdir(root):
    sub = "s" + sub_dir[0:4]
    anat_dir = os.path.join(root, sub_dir, "anatomical")
    for directory in os.listdir(anat_dir):
        time_dir = os.path.join(anat_dir, directory)
        for d in os.listdir(time_dir):
            dicoms_dir = os.path.join(time_dir, d, 'dicoms')
            dicom_list = os.listdir(dicoms_dir)
            dicom = dicom_list[0]
            path = os.path.join(dicoms_dir, dicom)
            cmd1 = "recon-all -i " + path + " -subjid " + sub
            check_output(cmd1, shell=True)
            call(cmd1, shell=True)
            cmd2 = "recon-all -all -subjid " + sub,
            call(cmd2, shell=True)

This is failing:

这是失败的：

Traceback (most recent call last):
     File "/home/katie/scripts/autoReconSO.py", line 28, in <module>
        check_output(cmd1, shell=True)
      File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    CalledProcessError: Command 'recon-all -i /media/foo/bar -subjid s1001' returned non-zero exit status 127

I maybe understand why this is. My "calls" later in the script are raising new subprocesses that do not inherit environment variables from the processes that are raised by invocation of the source() function. I have done a number of things to try to confirm my understanding. One example -- I put these lines:

我也许明白这是为什么。我稍后在脚本中的“调用”正在引发新的子进程，这些子进程不从通过调用 source() 函数引发的进程继承环境变量。我做了很多事情来试图确认我的理解。一个例子——我把这些行：

mkdir ~/testFreeSurferEnv
export TEST_ENV_VAR=~/testFreeSurferEnv

in the FreeSurferEnv.sh script. The directory gets made just fine, but in the Python script this:

在 FreeSurferEnv.sh 脚本中。该目录制作得很好，但在 Python 脚本中：

cmd = 'mkdir $TEST_ENV_VAR/test'
check_output(cmd, shell=True)

fails like this:

像这样失败：

File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'mkdir $TEST_ENV_VAR/test' returned non-zero exit status 1

QUESTION:

题：

How can I make the subprocess that runs "recon-all" inherit the environment variables it needs? Or how can I do everything I need to do -- run the scripts to set the environment variables, and call recon-all, in the same process? Or should I approach the problem another way? Or do I likely misunderstand the problem?

如何让运行“recon-all”的子进程继承它需要的环境变量？或者我怎样才能做我需要做的一切——运行脚本来设置环境变量，并在同一个过程中调用 recon-all？或者我应该以另一种方式解决问题？或者我可能误解了这个问题？

采纳答案by unutbu

Regarding

关于

If I were doing this directly at the command line, I'd "source" a script called mySetUpFreeSurfer.sh that does nothing but set three environment variables, and then "source" another script, FreeSurferEnv.sh.

If I were doing this directly at the command line, I'd "source" a script called mySetUpFreeSurfer.sh that does nothing but set three environment variables, and then "source" another script, FreeSurferEnv.sh.

I think you would be better off using Python to automate the process of writing a shell script newscript.sh, and then calling this script with onecall subprocess.check_output(instead of many calls to Popen, check_output, call, etc.):

我想你会过得更好用Python编写自动化shell脚本的过程newscript.sh，然后调用该脚本与一个电话 subprocess.check_output（而不是多次调用Popen，check_output， call，等）：

newscript.sh:

新脚本.sh：

#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
recon-all -i /media/foo/bar -subjid s1001
...

and then calling

然后打电话

subprocess.check_output(['newscript.sh'])

---

---

import subprocess
import tempfile
import os
import stat


with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
    f.write('''\
#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
''')
    root = "/media/foo/"
    for sub_dir in os.listdir(root):
        sub = "s" + sub_dir[0:4]
        anat_dir = os.path.join(root, sub_dir, "anatomical")
        for directory in os.listdir(anat_dir):
            time_dir = os.path.join(anat_dir, directory)
            for d in os.listdir(time_dir):
                dicoms_dir = os.path.join(time_dir, d, 'dicoms')
                dicom_list = os.listdir(dicoms_dir)
                dicom = dicom_list[0]
                path = os.path.join(dicoms_dir, dicom)
                cmd1 = "recon-all -i {}  -subjid {}\n".format(path, sub)
                f.write(cmd1)
                cmd2 = "recon-all -all -subjid {}\n".format(sub)
                f.write(cmd2)

filename = f.name
os.chmod(filename, stat.S_IRUSR | stat.S_IXUSR)
subprocess.call([filename])
os.unlink(filename)

---

---

By the way,

顺便一提，

def source(script, update=1):
    pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)
    data = pipe.communicate()[0]
    env = dict((line.split("=", 1) for line in data.splitlines()))
    if update:
        os.environ.update(env)
    return env

is broken. For example, if scriptcontains something like

被打破。例如，如果script包含类似

VAR=`ls -1`
export VAR

then

然后

. script; env

may return output like

可能会返回类似的输出

VAR=file1
file2
file3

which will result in source(script)raising a ValueError:

这将导致source(script)提高一个ValueError：

env = dict((line.split("=", 1) for line in data.splitlines()))
ValueError: dictionary update sequence element #21 has length 1; 2 is required

---

---

There is a way to fix source: have envseparate environment variables with a zero byte instead of the ambiguous newline:

有一种方法可以解决source：env使用零字节而不是不明确的换行符来使用单独的环境变量：

def source(script, update=True):
    """
    http://pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (Miki Tebeka)
    http://stackoverflow.com/questions/3503719/#comment28061110_3505826 (ahal)
    """
    import subprocess
    import os
    proc = subprocess.Popen(
        ['bash', '-c', 'set -a && source {} && env -0'.format(script)], 
        stdout=subprocess.PIPE, shell=False)
    output, err = proc.communicate()
    output = output.decode('utf8')
    env = dict((line.split("=", 1) for line in output.split('\x00') if line))
    if update:
        os.environ.update(env)
    return env

Fixable or not, however, you are still probably better off constructing a conglomerate shell script (as shown above) than you would be parsing envand passing envdicts to subprocesscalls.

然而，无论是否可以修复，与解析env并将envdicts传递给subprocess调用相比，构建一个联合 shell 脚本（如上所示）可能仍然更好。

回答by abarnert

If you look at the docs for Popen, it takes an envparameter:

如果您查看 docs for Popen，它需要一个env参数：

If envis not None, it must be a mapping that defines the environment variables for the new process; these are used instead of inheriting the current process' environment, which is the default behavior.

如果env不是None，则它必须是定义新进程的环境变量的映射；使用这些而不是继承当前进程的环境，这是默认行为。

You've written a function that extracts the environment you want from your sourced scripts and puts it into a dict. Just pass the result as the envto the scripts you want to use it. For example:

您已经编写了一个函数，可以从源脚本中提取所需的环境并将其放入dict. 只需将结果作为 传递env给您要使用它的脚本。例如：

env = {}
env.update(os.environ)
env.update(source('~/scripts/mySetUpFreeSurfer.sh'))
env.update(source('/usr/local/freesurfer/FreeSurferEnv.sh'))

# …

check_output(cmd, shell=True, env=env)