关于 Python/Django 和消息队列的建议

Advice on Python/Django and message queues

提问by Andy Hume

I have an application in Django, that needs to send a large number of emails to users in various use cases. I don't want to handle this synchronously within the application for obvious reasons.

我在 Django 中有一个应用程序，需要在各种用例中向用户发送大量电子邮件。出于显而易见的原因，我不想在应用程序中同步处理这个问题。

Has anyone any recommendations for a message queuing server which integrates well with Python, or they have used on a Django project? The rest of my stack is Apache, mod_python, MySQL.

有没有人对与 Python 集成良好的消息队列服务器有任何建议，或者他们已经在 Django 项目中使用过？我的堆栈的其余部分是 Apache、mod_python、MySQL。

采纳答案by max

So far I have found no "nice" solution for this. I have some more strict soft realtime requirements (taking a picture from a cardboard box being labeled) so probably one of the approaches is fast enough for you. I assume emails can wait for a few minutes.

到目前为止，我还没有找到“好的”解决方案。我有一些更严格的软实时要求（从贴有标签的纸板箱中拍照），因此其中一种方法可能对您来说足够快。我假设电子邮件可以等待几分钟。

• A "todo list" in the database processed by a cron job.
• A "todo list" in the database processed permanently beeing polled by a daemon.
• Using a custom daemon which gets notified by the webserver via an UDP packet (in Production today). Basically my own Queing system with the IP stack for handling the queue.
• Using ActiveMQ as a message broker- this didn't work out because of stability issues. Also to me Java Daemons are generally somewhat plump
• Using Update Triggers in CouchDB. Nice but Update Triggers are not meant to do heavy image processing, so no good fit for my problem.

• 由 cron 作业处理的数据库中的“待办事项列表”。
• 数据库中的“待办事项列表”由守护进程轮询永久处理。
• 使用自定义守护程序，该守护程序通过 UDP 数据包（今天在生产中）由网络服务器通知。基本上是我自己的队列系统，带有用于处理队列的 IP 堆栈。
• 使用 ActiveMQ 作为消息代理- 由于稳定性问题，这没有成功。对我来说，Java 守护进程通常也有些丰满
• 在 CouchDB 中使用更新触发器。不错，但更新触发器并不意味着进行繁重的图像处理，因此不适合我的问题。

So far I haven't tried RabbitMQ and XMPP/ejabebrd for handling the problem but they are on my list of next things to try. RabbitMQ got decent Python connectivity during 2008 and there are tons of XMPP libraries.

到目前为止，我还没有尝试过 RabbitMQ 和 XMPP/ejabebrd 来处理这个问题，但它们在我接下来要尝试的列表中。RabbitMQ 在 2008 年获得了不错的 Python 连接，并且有大量的 XMPP 库。

But perhaps all you need is a correctly configured mailserver on the local machine. This probably would allow you to dump mails synchronously into the local mailserver and thus make your whole software stack much more simple.

但也许您所需要的只是在本地机器上正确配置的邮件服务器。这可能允许您将邮件同步转储到本地邮件服务器，从而使您的整个软件堆栈更加简单。

回答by Van Gale

In your specific case, where it's just an email queue, I wold take the easy way out and use django-mailer. As a nice side bonues there are other pluggable projects that are smart enough to take advantage of django-mailer when they see it in the stack.

在您的特定情况下，它只是一个电子邮件队列，我会采取简单的方法并使用django-mailer。作为一个不错的附带奖励，还有其他可插拔的项目，当他们在堆栈中看到它时，它们足够聪明以利用 django-mailer。

As for more general queue solutions, I haven't been able to try any of these yet, but here's a list of ones that look more interesting to me:

至于更通用的队列解决方案，我还没有尝试过其中任何一个，但这里有一个我觉得更有趣的列表：

1. pybeanstalk/beanstalkd
2. python interface to gearman(which is probably much more interesting now with the release of the C version of gearman)
3. memcacheQ
4. stomp
5. Celery

1. 豆茎/豆茎
2. 到 gearman 的 python 接口（现在随着gearman的C 版本的发布，这可能更有趣）
3. 内存缓存Q
4. 跺脚
5. 芹菜

回答by Gruff

Stompserver is a good option. It's lightweight, easy to install and easy to use from Django/python.

Stompserver 是一个不错的选择。它是轻量级的，易于安装且易于从 Django/python 使用。

We have a system using stompserver in production for sending out emails and processing other jobs asynchronously.

我们有一个在生产中使用 stompserver 的系统，用于发送电子邮件和异步处理其他作业。

Django saves the emails to the database, a model.post_save handler in Django sends an event to stompserver and stompserver passes the event to a consumer process which does the asynchronous task (sends the email).

Django 将电子邮件保存到数据库中，Django 中的 model.post_save 处理程序向 stompserver 发送事件，然后 stompserver 将事件传递给执行异步任务（发送电子邮件）的消费者进程。

It scales up quite nicely because you can add consumer processes at runtime - two consumers can send twice as many emails, and the consumers can be on seperate machines. One slight complication is that each consumer needs its own named queue so Django needs to know how many consumers are available and send events to each queue in a round-robin way. (Two consumers listening on the same queue will both get each message = duplication). If you only want one consumer process then this isn't an issue.

它可以很好地扩展，因为您可以在运行时添加消费者进程 - 两个消费者可以发送两倍的电子邮件，并且消费者可以在不同的机器上。一个稍微复杂的地方是每个消费者都需要自己的命名队列，因此 Django 需要知道有多少消费者可用，并以循环方式将事件发送到每个队列。（在同一个队列上侦听的两个消费者都将获得每条消息 = 重复）。如果您只想要一个消费者进程，那么这不是问题。

We previously had processes which polled the database continuously for jobs but found that it was adding a lot of load to the system, even when nothing needed to be processed.

我们之前有一些进程会不断轮询数据库以获取作业，但发现它给系统增加了大量负载，即使没有需要处理的内容。

回答by dhruvbird

You might want to have a look at pymq. It's written in python, talks HTTP with it's clients and allows a host of monitoring and management options for queues.

你可能想看看pymq。它是用 python 编写的，与它的客户端进行 HTTP 通信，并允许对队列进行大量监控和管理选项。

回答by Sandip Bhattacharya

Is there anything wrong is solving this using the mail infrastructure? Like, every app server running their own mail daemons which will queue any locally submitted mail, which forward to a centralized mail server which can do the mail heavy lifting?

使用邮件基础设施解决这个问题有什么问题吗？就像，每个应用服务器都运行自己的邮件守护程序，它们会将任何本地提交的邮件排队，然后转发到可以处理邮件繁重工作的中央邮件服务器？

回答by David

http://www.snakemq.net/might work as well

http://www.snakemq.net/也可以

回答by nosklo

Just add the emails to a database, and then write another script ran by some task scheduler utility (cron comes to mind) to send the emails.

只需将电子邮件添加到数据库，然后编写由某个任务调度程序实用程序（想到 cron）运行的另一个脚本来发送电子邮件。

回答by James Brady

If you already have MySQL installed, you could create a table to use as a "todo list" of sorts.

如果您已经安装了 MySQL，您可以创建一个表用作各种“待办事项列表”。

Threads synchronously add jobs to the table, and a batched task removes jobs as they're completed.

线程同步地将作业添加到表中，批处理任务在作业完成时删除作业。

That way, there's no need to install and learn more software, and it should work fine as a persistent job store so long as you're not sending lotsof email (like >10/sec).

这样，就无需安装和学习更多软件，只要您不发送大量电子邮件（例如 >10/秒），它就可以作为持久作业存储正常工作。

回答by Seun Osewa

Here's a lazy but correct and adequate solution. Use the following database table as a queue.

这是一个懒惰但正确且充分的解决方案。使用以下数据库表作为队列。

drop table if exists mailqueue;
create table mailqueue (
    id bigint primary key,
    subject text not null,
    body mediumtext not null,
    from varchar(255) not null,
    to varchar(255) not null
);

Senders should insert new rows to the end of this table.

发件人应在此表的末尾插入新行。

Setup worker threads to pop mails one at a time from the other end (lowest id) and try to send them.

设置工作线程从另一端（最低 id）一次弹出一个邮件并尝试发送它们。