nodejs的工作队列？

worker queue for nodejs?

提问by mkoryak

I am in the process of beginning to write a worker queue for node using node's cluster API and mongoose.

我正在开始使用节点的集群 API 和 mongoose 为节点编写工作队列。

I noticed that a lot of libs exist that already do this but using redis and forking. Is there a good reason to fork versus using the cluster API?

我注意到有很多库已经这样做了，但是使用 redis 和分叉。与使用集群 API 相比，是否有充分的理由分叉？

editand now i also find this: https://github.com/xk/node-threads-a-gogo-- too many options!

编辑，现在我也发现了这个：https: //github.com/xk/node-threads-a-gogo——选项太多了！

I would rather not add redis to the mix since I already use mongo. Also, my requirements are very loose, I would like persistence but could go without it for the first version.

我宁愿不将 redis 添加到组合中，因为我已经使用了 mongo。此外，我的要求非常宽松，我想要持久性，但在第一个版本中可以没有它。

Part two of the question: What are the most stable/used nodejsworker queue libs out there today?

问题的第二部分：当今最稳定/最常用的nodejs工作队列库是什么？

采纳答案by mkoryak

Wanted to follow up on this. My solution ended up being a roll your own cluster impl where some of my cluster workers are dedicated job workers (ie they just have code to work on jobs).

想跟进此事。我的解决方案最终成为滚动你自己的集群实现，其中我的一些集群工作人员是专门的工作工作人员（即他们只有代码来处理工作）。

I use agendafor job scheduling.

我使用议程进行作业调度。

Cron type jobs are scheduled by the cluster master. The rest of the jobs are created in the non-worker clusters as they are needed. (verification emails etc)

Cron 类型的作业由集群主机调度。其余的工作是根据需要在非工作器集群中创建的。（验证邮件等）

Before that I was using kuebut dropped it because the rest of my app uses mongodb and I didnt like having to use redis just for job scheduling.

在此之前，我使用kue但放弃了它，因为我的应用程序的其余部分使用 mongodb，而且我不喜欢仅使用 redis 进行作业调度。

回答by rafaelcastrocouto

Have u tried https://github.com/rvagg/node-worker-farm? It is very light weight and doesn't require a separate server.

你试过https://github.com/rvagg/node-worker-farm吗？它非常轻巧，不需要单独的服务器。

回答by ChrisCM

I personally am partial to cluster-master.

我个人偏爱集群主控。

https://github.com/isaacs/cluster-master

https://github.com/isaacs/cluster-master

The reason I like cluster master is because it does very little besides add in logic for forking your process, and give you the ability to manage the number of process you're running, and a little bit of logging/recovery to boot! I find overly bloated process management libraries tend to be unstable, and sometimes even slow things down.

我喜欢 cluster master 的原因是因为它除了添加用于分叉进程的逻辑之外几乎没有什么作用，并且让您能够管理正在运行的进程数量，以及一些日志记录/恢复来启动！我发现过度膨胀的进程管理库往往不稳定，有时甚至会减慢速度。

This library will be good for you if the following are true:

如果以下情况属实，这个库对你有好处：

• Your module is largely asynchronous
• You don't have a huge amount of different types of events triggering
• The events that fire have small amounts of work to do, but you have lots of similar events firing(things like web servers)

• 您的模块在很大程度上是异步的
• 您没有大量不同类型的事件触发
• 触发的事件有少量工作要做，但是您有很多类似的事件触发（例如 Web 服务器）

The reason for the above list, is the reason why threads-a-gogo may be good for you, for the opposite reasons. If you have a few spots in your code, where there is a lot of work to do within your event loop, something like threads-a-gogo that launches a "thread" specifically for this work is awesome, because you aren't determining ahead of time how many workers to spawn, but rather spawning them to do work when needed. Note: this can also be bad if there is the potential for a lot of them to spawn, if you start launching too many processes things can actually bog down, but I digress.

上面列表的原因，是线程-a-gogo 可能对你有好处的原因，出于相反的原因。如果你的代码中有几个地方，在你的事件循环中有很多工作要做，像threads-a-gogo这样专门为这项工作启动一个“线程”的东西很棒，因为你不确定提前产生多少工人，而是在需要时产生他们做工作。注意：如果有可能产生很多进程，这也可能很糟糕，如果你开始启动太多进程，事情实际上可能会陷入困境，但我离题了。

To summarize, if your module is largely asynchronous already, what you really want is a worker pool. To minimize the down time when your process is not listening for events, and to maximize the amount of processor you can use. Unless you have a very busy syncronous call, a single node event loop will have troubles taking advantage of even a single core of a processor. Under this circumstance, you are best off with cluster-master. What I recommend is doing a little benchmarking, and see how much of a single core your program can use under the "worst case scenario". Let's say this is 33% of one core. If you have a quad core machine, you then tell cluster master to launch you 12 workers.

总而言之，如果您的模块在很大程度上已经是异步的，那么您真正想要的是一个工作池。最大限度地减少您的进程不侦听事件时的停机时间，并最大限度地增加您可以使用的处理器数量。除非您有一个非常繁忙的同步调用，否则单个节点事件循环即使利用处理器的单个内核也会遇到麻烦。在这种情况下，最好使用 cluster-master。我建议做一些基准测试，看看你的程序在“最坏情况”下可以使用多少单核。假设这是一个核心的 33%。如果您有一台四核机器，则告诉集群主服务器启动 12 个工作线程。

Hope this helped!

希望这有帮助！