Python IOError: [Errno 24] 打开的文件太多:
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/18280612/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
IOError: [Errno 24] Too many open files:
提问by learner
I have a huge file that I am writing into approximately 450 files. I am getting error as too many files open. I searched the web and found some solution but it is not helping.
我有一个巨大的文件，我正在写入大约 450 个文件。我收到错误为too many files open. 我在网上搜索并找到了一些解决方案，但没有帮助。
import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (1000,-1))
>>> len(pureResponseNames) #Filenames
434
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(1000, 9223372036854775807)
>>> output_files = [open(os.path.join(outpathDirTest, fname) + ".txt", "w") for fname in pureResponseNames]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 24] Too many open files: 'icd9_737.txt'
>>>
I also changed ulimitfrom the command line as below:
我也ulimit从命令行更改如下：
$ ulimit -n 1200
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1200
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
$
I am still getting the same error. PS: I also restarted my system and run the program but with no success.
我仍然遇到同样的错误。PS：我也重新启动了系统并运行了程序，但没有成功。
回答by Stefan Bollmann
A minimal working example would be nice. I got the same results like ron.rothman using the following script with Python 3.3.2, GCC 4.2.1 on mac 10.6.8. Do you get errors using it?
一个最小的工作示例会很好。我在 mac 10.6.8 上使用以下脚本和 Python 3.3.2、GCC 4.2.1 得到了与 ron.rothman 相同的结果。使用它会出错吗？
import os, sys
import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (1000,-1))
pureResponseNames = ['f'+str(i) for i in range(434)]
try:
os.mkdir("testCase")
except:
print('Maybe the folder is already there.')
outpathDirTest="testCase/"
output_files = [open(os.path.join(outpathDirTest, fname) + ".txt", "w") for fname in pureResponseNames]
for i in range(len(output_files)):
output_files[i].write('This is a test of file nr.'+str(i))
output_files[i].close()
回答by publysher
"Too many open files" errors are always tricky –?you not only have to twiddle with ulimit, but you also have to check system-wide limits and OSX-specifics. This SO post gives more information on open files in OSX.(Spoiler alert: the default is 256).
“打开的文件太多”错误总是很棘手——你不仅要摆弄ulimit，还必须检查系统范围的限制和 OSX 特定的。这篇 SO 帖子提供了有关 OSX 中打开文件的更多信息。（剧透警告：默认值为 256）。
However, it is often easy to limit the number of files that have to be open at the same time. If we look at Stefan Bollman's example, we can easily change that to:
但是，限制必须同时打开的文件数通常很容易。如果我们看看 Stefan Bollman 的例子，我们可以轻松地将其更改为：
pureResponseNames = ['f'+str(i) for i in range(434)]
outpathDirTest="testCase/"
output_files = [os.path.join(outpathDirTest, fname) + ".txt" for fname in pureResponseNames]
for filename in range(output_files):
with open(filename, 'w') as f:
f.write('This is a test of file nr.'+str(i))
回答by naveenkumar.s
You should try $ ulimit -n 50000instead of 1200.
你应该尝试$ ulimit -n 50000而不是1200.
回答by Fruit
In case you can't close the file for some reasons(e.g. you're using 3rd party module), you may consider to set based on hardmaximum limit instead of predefined hard-coded limit (It will throws ValueErrorif you try to set hard+1):
如果由于某些原因您无法关闭文件（例如您正在使用 3rd 方模块），您可以考虑根据hard最大限制而不是预定义的硬编码限制ValueError进行设置（如果您尝试设置，它将抛出hard+1）：
import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard))
And I want to make it clear that even you manually delete the files created while python process still running, it will still throws such error later.
而且我想明确指出，即使您手动删除在 python 进程仍在运行时创建的文件，它稍后仍会抛出此类错误。
回答by Q Caron
I strongly discourage you from increasing the ulimit.
我强烈建议您不要增加ulimit.
- For example, your database may grow a lot and result in generating many more files than it used to, so much that it would become greater than the limit you fixed and thought was enough.
- It's a time-consuming/error-prone maintenance task because you would have to make sure that every environment/server has that limit properly set and never changed.
- 例如，您的数据库可能会增长很多并导致生成的文件比以前多得多，以至于它会变得大于您固定并认为足够的限制。
- 这是一项耗时/容易出错的维护任务，因为您必须确保每个环境/服务器都正确设置了该限制并且永不更改。
You should ensure that openis used in combination with closeor that the withstatement is used (which is more pythonic).
您应该确保open结合使用close或使用该with语句（更pythonic）。
Third-party libraries might give you issues (for example, pyPDF2 PdfFileMerger.appendkeeps files open until the writemethod is called on it).
The way I tracked this is pretty ugly but trying a couple of things on the server while monitoring the number of open files did the trick (my local development computer runs under Mac OS X and server is CentOs):
第三方库可能会给您带来问题（例如，pyPDF2PdfFileMerger.append会保持文件打开，直到对其write调用该方法）。我跟踪它的方式非常难看，但是在监视打开文件数量的同时在服务器上尝试了几件事就成功了（我的本地开发计算机在 Mac OS X 下运行，服务器是 CentOs）：
watch 'lsof | grep "something-created-filenames-have-in-common" | wc -l'
