Javascript 节点和错误：EMFILE，打开的文件太多
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/8965606/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
node and Error: EMFILE, too many open files
提问by xaverras
For some days I have searched for a working solution to an error
几天来，我一直在寻找解决错误的可行解决方案
Error: EMFILE, too many open files
Error: EMFILE, too many open files
It seems that many people have the same problem. The usual answer involves increasing the number of file descriptors. So, I've tried this:
似乎很多人都有同样的问题。通常的答案涉及增加文件描述符的数量。所以，我试过这个：
sysctl -w kern.maxfiles=20480,
sysctl -w kern.maxfiles=20480,
The default value is 10240. This is a little strange in my eyes, because the number of files I'm handling in the directory is under 10240. Even stranger, I still receive the same error after I've increased the number of file descriptors.
默认值是10240。这在我看来有点奇怪，因为我在目录中处理的文件数在10240以下。更奇怪的是，我增加文件描述符数后仍然收到同样的错误.
Second question:
第二个问题：
After a number of searches I found a work around for the "too many open files" problem:
经过多次搜索，我找到了解决“打开文件太多”问题的方法：
var requestBatches = {};
function batchingReadFile(filename, callback) {
// First check to see if there is already a batch
if (requestBatches.hasOwnProperty(filename)) {
requestBatches[filename].push(callback);
return;
}
// Otherwise start a new one and make a real request
var batch = requestBatches[filename] = [callback];
FS.readFile(filename, onRealRead);
// Flush out the batch on complete
function onRealRead() {
delete requestBatches[filename];
for (var i = 0, l = batch.length; i < l; i++) {
batch[i].apply(null, arguments);
}
}
}
function printFile(file){
console.log(file);
}
dir = "/Users/xaver/Downloads/xaver/xxx/xxx/"
var files = fs.readdirSync(dir);
for (i in files){
filename = dir + files[i];
console.log(filename);
batchingReadFile(filename, printFile);
Unfortunately I still recieve the same error. What is wrong with this code?
不幸的是，我仍然收到同样的错误。这段代码有什么问题？
One last question (I'm new to javascript and node), I'm in the process of developping a web application with a lot of requests for about 5000 daily users. I've many years of experience in programming with other languages like python and java. so originally I thought to developp this application with django or play framework. Then I discovered node and I must say that the idea of non-blocking I/O model is really nice, seductive, and most of all very fast!
最后一个问题（我是 javascript 和 node 的新手），我正在开发一个 Web 应用程序，每天有大约 5000 个用户的大量请求。我有多年使用其他语言（如 python 和 java）编程的经验。所以最初我想用 django 或 play 框架开发这个应用程序。然后我发现了 node，我必须说非阻塞 I/O 模型的想法非常好、诱人，而且最重要的是非常快！
But what kind of problems should I expect with node? Is it a production proven web server? What are your experiences?
但是我应该期待 node 出现什么样的问题？它是经过生产验证的 Web 服务器吗？你有什么经验？
回答by blak3r
For when graceful-fsdoesn't work... or you just want to understand where the leak is coming from. Follow this process.
当优雅的 fs不起作用时……或者您只想了解泄漏的来源。按照这个过程。
(e.g. graceful-fs isn't gonna fix your wagon if your issue is with sockets.)
（例如，如果您的问题与套接字有关，graceful-fs 不会修复您的旅行车。）
From My Blog Article: http://www.blakerobertson.com/devlog/2014/1/11/how-to-determine-whats-causing-error-connect-emfile-nodejs.html
来自我的博客文章：http: //www.blakerobertson.com/devlog/2014/1/11/how-to-determine-whats-causing-error-connect-emfile-nodejs.html
How To Isolate
如何隔离
This command will output the number of open handles for nodejs processes:
此命令将输出 nodejs 进程的打开句柄数：
lsof -i -n -P | grep nodejs
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
...
nodejs 12211 root 1012u IPv4 151317015 0t0 TCP 10.101.42.209:40371->54.236.3.170:80 (ESTABLISHED)
nodejs 12211 root 1013u IPv4 151279902 0t0 TCP 10.101.42.209:43656->54.236.3.172:80 (ESTABLISHED)
nodejs 12211 root 1014u IPv4 151317016 0t0 TCP 10.101.42.209:34450->54.236.3.168:80 (ESTABLISHED)
nodejs 12211 root 1015u IPv4 151289728 0t0 TCP 10.101.42.209:52691->54.236.3.173:80 (ESTABLISHED)
nodejs 12211 root 1016u IPv4 151305607 0t0 TCP 10.101.42.209:47707->54.236.3.172:80 (ESTABLISHED)
nodejs 12211 root 1017u IPv4 151289730 0t0 TCP 10.101.42.209:45423->54.236.3.171:80 (ESTABLISHED)
nodejs 12211 root 1018u IPv4 151289731 0t0 TCP 10.101.42.209:36090->54.236.3.170:80 (ESTABLISHED)
nodejs 12211 root 1019u IPv4 151314874 0t0 TCP 10.101.42.209:49176->54.236.3.172:80 (ESTABLISHED)
nodejs 12211 root 1020u IPv4 151289768 0t0 TCP 10.101.42.209:45427->54.236.3.171:80 (ESTABLISHED)
nodejs 12211 root 1021u IPv4 151289769 0t0 TCP 10.101.42.209:36094->54.236.3.170:80 (ESTABLISHED)
nodejs 12211 root 1022u IPv4 151279903 0t0 TCP 10.101.42.209:43836->54.236.3.171:80 (ESTABLISHED)
nodejs 12211 root 1023u IPv4 151281403 0t0 TCP 10.101.42.209:43930->54.236.3.172:80 (ESTABLISHED)
....
Notice the: 1023u (last line)- that's the 1024th file handle which is the default maximum.
请注意： 1023u（最后一行）- 这是默认最大值的第 1024 个文件句柄。
Now, Look at the last column. That indicates which resource is open. You'll probably see a number of lines all with the same resource name. Hopefully, that now tells you where to look in your code for the leak.
现在，看看最后一列。这表明哪个资源是开放的。您可能会看到许多行都具有相同的资源名称。希望现在可以告诉您在代码中查找泄漏的位置。
If you don't know multiple node processes, first lookup which process has pid 12211. That'll tell you the process.
如果您不知道多个节点进程，请先查找哪个进程的 pid 为 12211。它会告诉您进程。
In my case above, I noticed that there were a bunch of very similar IP Addresses. They were all 54.236.3.###By doing ip address lookups, was able to determine in my case it was pubnub related.
在我上面的例子中，我注意到有一堆非常相似的 IP 地址。他们都是54.236.3.###通过进行 ip 地址查找，在我的情况下能够确定它与 pubnub 相关。
Command Reference
命令参考
Use this syntax to determine how many open handles a process has open...
使用此语法来确定进程打开了多少个打开的句柄...
To get a count of open files for a certain pid
获取某个 pid 的打开文件数
I used this command to test the number of files that were opened after doing various events in my app.
我使用此命令来测试在我的应用程序中执行各种事件后打开的文件数。
lsof -i -n -P | grep "8465" | wc -l
# lsof -i -n -P | grep "nodejs.*8465" | wc -l
28
# lsof -i -n -P | grep "nodejs.*8465" | wc -l
31
# lsof -i -n -P | grep "nodejs.*8465" | wc -l
34
What is your process limit?
你的进程限制是多少？
ulimit -a
The line you want will look like this:
您想要的行将如下所示：
open files (-n) 1024
Permanently change the limit:
永久更改限制：
- tested on Ubuntu 14.04, nodejs v. 7.9
- 在 Ubuntu 14.04、nodejs v. 7.9 上测试
In case if you are expecting to open many connections (websockets is a good example), you can permanently increase the limit:
如果您希望打开许多连接（websockets 就是一个很好的例子），您可以永久增加限制：
file: /etc/pam.d/common-session (add to the end)
session required pam_limits.sofile: /etc/security/limits.conf (add to the end, or edit if already exists)
root soft nofile 40000 root hard nofile 100000restart your nodejs and logout/login from ssh.
- this may not work for older NodeJS you'll need to restart server
- use instead of if your node runs with different uid.
文件：/etc/pam.d/common-session （添加到最后）
session required pam_limits.so文件：/etc/security/limits.conf （添加到最后，如果已经存在则编辑）
root soft nofile 40000 root hard nofile 100000重新启动您的 nodejs 并从 ssh 注销/登录。
- 这可能不适用于较旧的 NodeJS，您需要重新启动服务器
- 如果您的节点使用不同的 uid 运行，请使用代替。
回答by Myrne Stol
Using the graceful-fsmodule by Isaac Schlueter (node.js maintainer) is probably the most appropriate solution. It does incremental back-off if EMFILE is encountered. It can be used as a drop-in replacement for the built-in fsmodule.
使用graceful-fsIsaac Schlueter（node.js 维护者）的模块可能是最合适的解决方案。如果遇到 EMFILE，它会进行增量回退。它可以用作内置fs模块的直接替代品。
回答by bh4r4th
I am not sure whether this will help anyone, I started working on a big project with lot of dependencies which threw me the same error. My colleague suggested me to install watchmanusing brew and that fixed this problem for me.
我不确定这是否会帮助任何人，我开始从事一个有很多依赖项的大项目，这给了我同样的错误。我的同事建议我watchman使用 brew安装，这为我解决了这个问题。
brew update
brew install watchman
Edit on 26 June 2019: Github link to watchman
2019 年 6 月 26 日编辑： Github 链接到守望者
回答by Tim P.
You're reading too many files. Node reads files asynchronously, it'll be reading all files at once. So you're probably reading the 10240 limit.
您正在阅读太多文件。节点异步读取文件，它将一次读取所有文件。因此，您可能正在阅读 10240 限制。
See if this works:
看看这是否有效：
var fs = require('fs')
var events = require('events')
var util = require('util')
var path = require('path')
var FsPool = module.exports = function(dir) {
events.EventEmitter.call(this)
this.dir = dir;
this.files = [];
this.active = [];
this.threads = 1;
this.on('run', this.runQuta.bind(this))
};
// So will act like an event emitter
util.inherits(FsPool, events.EventEmitter);
FsPool.prototype.runQuta = function() {
if(this.files.length === 0 && this.active.length === 0) {
return this.emit('done');
}
if(this.active.length < this.threads) {
var name = this.files.shift()
this.active.push(name)
var fileName = path.join(this.dir, name);
var self = this;
fs.stat(fileName, function(err, stats) {
if(err)
throw err;
if(stats.isFile()) {
fs.readFile(fileName, function(err, data) {
if(err)
throw err;
self.active.splice(self.active.indexOf(name), 1)
self.emit('file', name, data);
self.emit('run');
});
} else {
self.active.splice(self.active.indexOf(name), 1)
self.emit('dir', name);
self.emit('run');
}
});
}
return this
};
FsPool.prototype.init = function() {
var dir = this.dir;
var self = this;
fs.readdir(dir, function(err, files) {
if(err)
throw err;
self.files = files
self.emit('run');
})
return this
};
var fsPool = new FsPool(__dirname)
fsPool.on('file', function(fileName, fileData) {
console.log('file name: ' + fileName)
console.log('file data: ', fileData.toString('utf8'))
})
fsPool.on('dir', function(dirName) {
console.log('dir name: ' + dirName)
})
fsPool.on('done', function() {
console.log('done')
});
fsPool.init()
回答by Trey Griffith
I ran into this problem today, and finding no good solutions for it, I created a module to address it. I was inspired by @fbartho's snippet, but wanted to avoid overwriting the fs module.
我今天遇到了这个问题，没有找到好的解决方案，我创建了一个模块来解决它。我受到@fbartho 片段的启发，但想避免覆盖 fs 模块。
The module I wrote is Filequeue, and you use it just like fs:
我写的模块是Filequeue，你可以像 fs 一样使用它：
var Filequeue = require('filequeue');
var fq = new Filequeue(200); // max number of files to open at once
fq.readdir('/Users/xaver/Downloads/xaver/xxx/xxx/', function(err, files) {
if(err) {
throw err;
}
files.forEach(function(file) {
fq.readFile('/Users/xaver/Downloads/xaver/xxx/xxx/' + file, function(err, data) {
// do something here
}
});
});
回答by Plaute
Like all of us, you are another victim of asynchronous I/O. With asynchronous calls, if you loop around a lot of files, Node.js will start to open a file descriptor for each file to read and then will wait for action until you close it.
像我们所有人一样，您也是异步 I/O 的受害者。对于异步调用，如果你循环了很多文件，Node.js 将开始为每个要读取的文件打开一个文件描述符，然后等待操作直到你关闭它。
File descriptor remains open until resource is available on your server to read it. Even if your files are small and reading or updating is fast, it takes some time, but in the same time your loop don't stop to open new files descriptor. So if you have too many files, the limit will be soon reached and you get a beautiful EMFILE.
文件描述符保持打开状态，直到您的服务器上有资源可以读取它。即使您的文件很小并且读取或更新速度很快，也需要一些时间，但同时您的循环不会停止打开新的文件描述符。所以如果你有太多的文件，很快就会达到限制，你会得到一个漂亮的EMFILE。
There is one solution, creating a queue to avoid this effect.
有一个解决方案，创建一个队列来避免这种影响。
Thanks to people who wrote Async, there is a very useful function for that. There is a method called Async.queue, you create a new queue with a limit and then add filenames to the queue.
感谢编写Async 的人，有一个非常有用的功能。有一个名为Async.queue的方法，您可以创建一个具有限制的新队列，然后将文件名添加到队列中。
Note: If you have to open many files, it would be a good idea to store which files are currently open and don't reopen them infinitely.
注意：如果您必须打开许多文件，最好存储当前打开的文件并且不要无限地重新打开它们。
const fs = require('fs')
const async = require("async")
var q = async.queue(function(task, callback) {
console.log(task.filename);
fs.readFile(task.filename,"utf-8",function (err, data_read) {
callback(err,task.filename,data_read);
}
);
}, 4);
var files = [1,2,3,4,5,6,7,8,9,10]
for (var file in files) {
q.push({filename:file+".txt"}, function (err,filename,res) {
console.log(filename + " read");
});
}
You can see that each file is added to the queue (console.log filename), but only when the current queue is under the limit you set previously.
您可以看到每个文件都添加到队列中（console.log 文件名），但仅当当前队列低于您之前设置的限制时。
async.queue get information about availability of the queue through a callback, this callback is called only when data file is read and any action you have to do is achieved. (see fileRead method)
async.queue 通过回调获取有关队列可用性的信息，只有在读取数据文件并且您必须执行的任何操作都完成时才会调用此回调。（见 fileRead 方法）
So you cannot be overwhelmed by files descriptor.
所以你不会被文件描述符淹没。
> node ./queue.js
0.txt
1.txt
2.txt
0.txt read
3.txt
3.txt read
4.txt
2.txt read
5.txt
4.txt read
6.txt
5.txt read
7.txt
1.txt read (biggest file than other)
8.txt
6.txt read
9.txt
7.txt read
8.txt read
9.txt read
回答by fbartho
I just finished writing a little snippet of code to solve this problem myself, all of the other solutions appear way too heavyweight and require you to change your program structure.
我刚刚写了一小段代码来自己解决这个问题，所有其他解决方案都显得过于重量级，需要您更改程序结构。
This solution just stalls any fs.readFile or fs.writeFile calls so that there are no more than a set number in flight at any given time.
此解决方案只是停止任何 fs.readFile 或 fs.writeFile 调用，以便在任何给定时间都不会超过设定的数量。
// Queuing reads and writes, so your nodejs script doesn't overwhelm system limits catastrophically
global.maxFilesInFlight = 100; // Set this value to some number safeish for your system
var origRead = fs.readFile;
var origWrite = fs.writeFile;
var activeCount = 0;
var pending = [];
var wrapCallback = function(cb){
return function(){
activeCount--;
cb.apply(this,Array.prototype.slice.call(arguments));
if (activeCount < global.maxFilesInFlight && pending.length){
console.log("Processing Pending read/write");
pending.shift()();
}
};
};
fs.readFile = function(){
var args = Array.prototype.slice.call(arguments);
if (activeCount < global.maxFilesInFlight){
if (args[1] instanceof Function){
args[1] = wrapCallback(args[1]);
} else if (args[2] instanceof Function) {
args[2] = wrapCallback(args[2]);
}
activeCount++;
origRead.apply(fs,args);
} else {
console.log("Delaying read:",args[0]);
pending.push(function(){
fs.readFile.apply(fs,args);
});
}
};
fs.writeFile = function(){
var args = Array.prototype.slice.call(arguments);
if (activeCount < global.maxFilesInFlight){
if (args[1] instanceof Function){
args[1] = wrapCallback(args[1]);
} else if (args[2] instanceof Function) {
args[2] = wrapCallback(args[2]);
}
activeCount++;
origWrite.apply(fs,args);
} else {
console.log("Delaying write:",args[0]);
pending.push(function(){
fs.writeFile.apply(fs,args);
});
}
};
回答by Rohit Parte
I did all the stuff above mentioned for same problem but nothing worked. I tried below it worked 100%. Simple config changes.
我为同样的问题做了上面提到的所有东西，但没有任何效果。我在下面试过它 100% 工作。简单的配置更改。
Option 1 set limit (It won't work most of the time)
选项 1 设置限制（大多数情况下不起作用）
user@ubuntu:~$ ulimit -n 65535
check available limit
检查可用限制
user@ubuntu:~$ ulimit -n
1024
Option 2 To increase the available limit to say 65535
选项 2 增加可用限制为 65535
user@ubuntu:~$ sudo nano /etc/sysctl.conf
add the following line to it
向其中添加以下行
fs.file-max = 65535
run this to refresh with new config
运行它以使用新配置刷新
user@ubuntu:~$ sudo sysctl -p
edit the following file
编辑以下文件
user@ubuntu:~$ sudo vim /etc/security/limits.conf
add following lines to it
添加以下几行
root soft nproc 65535
root hard nproc 65535
root soft nofile 65535
root hard nofile 65535
edit the following file
编辑以下文件
user@ubuntu:~$ sudo vim /etc/pam.d/common-session
add this line to it
将此行添加到其中
session required pam_limits.so
logout and login and try the following command
注销并登录并尝试以下命令
user@ubuntu:~$ ulimit -n
65535
Option 3 Just add below line in
选项 3 只需添加以下行
DefaultLimitNOFILE=65535
to /etc/systemd/system.conf and /etc/systemd/user.conf
到 /etc/systemd/system.conf 和 /etc/systemd/user.conf
回答by James
Building on @blak3r's answer, here's a bit of shorthand I use in case it helps other diagnose:
以@blak3r 的回答为基础，这是我使用的一些速记，以防它有助于其他诊断：
If you're trying to debug a Node.js script that is running out of file descriptors here's a line to give you the output of lsofused by the node process in question:
如果您正在尝试调试文件描述符不足的 Node.js 脚本，这里有一行为您提供lsof相关节点进程使用的输出：
openFiles = child_process.execSync(`lsof -p ${process.pid}`);
This will synchronously run lsoffiltered by the current running Node.js process and return the results via buffer.
这将同步运行，lsof由当前运行的 Node.js 进程过滤并通过缓冲区返回结果。
Then use console.log(openFiles.toString())to convert the buffer to a string and log the results.
然后使用console.log(openFiles.toString())将缓冲区转换为字符串并记录结果。
回答by user1837639
With bagpipe, you just need change
有了风笛，你只需要改变
FS.readFile(filename, onRealRead);
=>
=>
var bagpipe = new Bagpipe(10);
bagpipe.push(FS.readFile, filename, onRealRead))
The bagpipe help you limit the parallel. more details: https://github.com/HymansonTian/bagpipe
风笛帮助您限制平行。更多详情：https: //github.com/HymansonTian/bagpipe
