Python 子进程：打开的文件太多
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/16526783/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Python Subprocess: Too Many Open Files
提问by Vahid Mirjalili
I am using subprocess to call another program and save its return values to a variable. This process is repeated in a loop, and after a few thousands times the program crashed with the following error:
我正在使用子进程调用另一个程序并将其返回值保存到变量中。这个过程在一个循环中重复，在几千次之后程序崩溃并出现以下错误：
Traceback (most recent call last):
File "./extract_pcgls.py", line 96, in <module>
SelfE.append( CalSelfEnergy(i) )
File "./extract_pcgls.py", line 59, in CalSelfEnergy
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1166, in _execute_child
errpipe_read, errpipe_write = _create_pipe()
OSError: [Errno 24] Too many open files
Any idea how to solve this issue is much appreciated!
非常感谢如何解决这个问题的任何想法！
Code supplied from comments:
从评论提供的代码：
cmd = "enerCHARMM.pl -parram=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
采纳答案by Vahid Mirjalili
I guess the problem was due to the fact that I was processing an open file with subprocess:
我想问题是由于我正在使用子进程处理打开的文件：
cmd = "enerCHARMM.pl -par param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
Here the cmd variable contain the name of a file that has just been created but not closed. Then the subprocess.Popencalls a system command on that file. After doing this for many times, the program crashed with that error message.
这里的 cmd 变量包含刚刚创建但未关闭的文件的名称。然后subprocess.Popen对该文件调用系统命令。多次执行此操作后，程序因该错误消息而崩溃。
So the message I learned from this is
所以我从中学到的信息是
Close the file you have created, then process it
关闭您创建的文件，然后对其进行处理
回答by Matt Sweeney
You can try raising the open file limit of the OS:
您可以尝试提高操作系统的打开文件限制：
ulimit -n 2048
ulimit -n 2048
回答by imp
opens file in subprocess. It is blocking call.
在子进程中打开文件。它正在阻止呼叫。
ss=subprocess.Popen(tempFileName,shell=True)
ss.communicate()
回答by gogasca
In Mac OSX (El Capitan) See current configuration:
在 Mac OSX (El Capitan) 中查看当前配置：
#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
Set open filesvalue to 10K :
将打开的文件值设置为 10K ：
#ulimit -Sn 10000
Verify results:
验证结果：
#ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 10000
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
回答by jfs
A child process created by Popen()may inherit open file descriptors (a finite resource) from the parent. Use close_fds=Trueon POSIX (default since Python 3.2), to avoid it. Also, "PEP 0446 -- Make newly created file descriptors non-inheritable" deals with some remaining issues (since Python 3.4).
创建的子进程Popen()可以从父进程继承打开的文件描述符（有限资源）。close_fds=True在 POSIX 上使用（自 Python 3.2 起默认），以避免它。此外，“PEP 0446 - 使新创建的文件描述符不可继承”处理一些遗留问题（自 Python 3.4 起）。
回答by Soniku
As others have noted, raise the limit in /etc/security/limits.conf and also file descriptors was an issue for me personally, so I did
正如其他人所指出的，提高 /etc/security/limits.conf 中的限制以及文件描述符对我个人来说是一个问题，所以我做了
sudo sysctl -w fs.file-max=100000
And added a line with fs.file-max = 100000 to /etc/sysctl.conf (reload with sysctl -p)
并在 /etc/sysctl.conf 中添加了一行 fs.file-max = 100000（使用 sysctl -p 重新加载）
Also if you want to make sure that your process is not affected by anything else (which mine was), use
此外，如果您想确保您的流程不受其他任何东西（我的）的影响，请使用
cat /proc/{process id}/limits
to find out what the actual limits of your process are, as for me the software running the python scripts also had its limits applied which have overridden the system wide settings.
找出你的进程的实际限制是什么，对我来说，运行 python 脚本的软件也应用了它的限制，这些限制覆盖了系统范围的设置。
Posting this answer here after resolving my particular issue with this error and hopefully it helps someone.
在解决了我与此错误有关的特定问题后，在此处发布此答案，希望对某人有所帮助。
回答by Jyogo
Maybe you are invoking the command multiple times. If so, each time you're doing stdout=subprocess.PIPE. Between each call try doing p.stdout.close().
也许您多次调用该命令。如果是这样，每次您执行stdout=subprocess.PIPE. 在每次通话之间尝试做p.stdout.close()。
