Python Redis 发布订阅和消息队列

Redis Pubsub and Message Queueing

提问by Marco Benvoglio

My overall question is: Using Redis for PubSub, what happens to messages when publishers push messages into a channel faster than subscribers are able to read them?

我的总体问题是：将 Redis 用于 PubSub，当发布者将消息推送到频道的速度比订阅者能够读取它们的速度快时，消息会发生什么？

For example, let's say I have:

例如，假设我有：

• A simple publisher publishing messages at the rate of 2 msg/sec.
• A simple subscriber reading messages at the rate of 1 msg/sec.

• 一个简单的发布者以 2 msg/sec 的速率发布消息。
• 一个简单的订阅者以 1 msg/sec 的速率阅读消息。

My naive assumption would be the subscriber would only see 50% of the messages published onto Redis. To test this theory, I wrote two scripts:

我天真的假设订阅者只会看到 50% 的消息发布到 Redis 上。为了测试这个理论，我写了两个脚本：

pub.py

发布文件

queue = redis.StrictRedis(host='localhost', port=6379, db=0)
channel = queue.pubsub()

for i in range(10): 
    queue.publish("test", i)
    time.sleep(0.5)

sub.py

子.py

r = redis.StrictRedis(host='localhost', port=6379, db=0)
p = r.pubsub()
p.subscribe('test')

while True:
    message = p.get_message()
    if message:
        print "Subscriber: %s" % message['data']
    time.sleep(1)

Results

结果

• When I ran sub.pyfirst, immediately followed by pub.py, I found that sub.pyactually displayed all the messages (1-10), one after another with a delay of 1 second in between. My initial assumption was wrong, Redis is queuing messages. More tests needed.
• When I ran pub.pyfirst, then waited 5 seconds before running sub.py, I found that sub.pyonly displayed the second half of the messages (5-10). I would have assumed this originally, but given my previous results, I would have thought messages were queued, which led me to the following conclusion...

• 当我先跑sub.py，紧接着是 时pub.py，我发现sub.py实际上显示了所有消息（1-10），一个接一个，中间有1秒的延迟。我最初的假设是错误的，Redis 正在排队消息。需要更多的测试。
• 当我先跑pub.py，然后等了5秒才跑的时候sub.py，发现sub.py只显示了后半部分的消息（5-10）。我最初会假设这一点，但考虑到我以前的结果，我会认为消息已排队，这使我得出以下结论......

Conclusions

结论

• Redis server appears to queue messages for each client, for each channel.
• As long as a client is listening, it doesn't matter how fast it reads messages. As long as it's connected, messages will remain queued for that client, for that channel.

• Redis 服务器似乎为每个客户端、每个通道排队消息。
• 只要客户端在监听，它读取消息的速度就无关紧要。只要它已连接，消息就会为该客户端、该通道保持排队状态。

Remaining Questions

剩余问题

• Are these conclusions valid?
• If so, how long will client/channel messages remained queued?
• If so, is there a redis-cli infocommand to see how many messages are queued (for each client/channel)?

• 这些结论有效吗？
• 如果是这样，客户端/通道消息将保持排队多长时间？
• 如果是这样，是否有redis-cli info命令可以查看排队的消息数（对于每个客户端/通道）？

采纳答案by Didier Spezia

The tests are valid, but the conclusions are partially wrong.

测试是有效的，但结论是部分错误的。

Redis does not queue anything on pub/sub channels. On the contrary, it tends to read the item from the publisher socket, and write the item in all the subscriber sockets, ideally in the same iteration of the event loop. Nothing is kept in Redis data structures.

Redis 不会在 pub/sub 频道上排队。相反，它倾向于从发布者套接字读取项目，并在所有订阅者套接字中写入项目，最好是在事件循环的同一迭代中。Redis 数据结构中不保留任何内容。

Now, as you demonstrated, there is still some kind of buffering. It is due to the usage of TCP/IP sockets, and Redis communication buffers.

现在，正如您所展示的，仍然存在某种缓冲。这是由于使用了 TCP/IP 套接字和 Redis 通信缓冲区。

Sockets have buffers, and of course, TCP comes with some flow control mechanisms. It avoids the loss of data when buffers are full. If a subscriber is not fast enough, data will accumulate in its socket buffer. When it is full, TCP will block the communication and prevents Redis to push more information in the socket.

套接字有缓冲区，当然，TCP 自带一些流量控制机制。它避免了缓冲区已满时的数据丢失。如果订阅者不够快，数据将在其套接字缓冲区中累积。当它已满时，TCP 会阻塞通信并阻止 Redis 在套接字中推送更多信息。

Redis also manages output communication buffers (on top of the ones of the sockets) to generate data formatted with the Redis protocol. So when the output buffer of the socket is full, the event loop will mark the socket as non writable, and data will remain in Redis output buffers.

Redis 还管理输出通信缓冲区（在套接字之上）以生成使用 Redis 协议格式化的数据。所以当套接字的输出缓冲区已满时，事件循环会将套接字标记为不可写，数据将保留在 Redis 输出缓冲区中。

Provided the TCP connection is still valid, data can remain in the buffers for a very long time. Now, both the socket and Redis output buffer are bound. If the subscribers are really too slow, and a lot of data accumulate, Redis will ultimately close the connection with subscribers (as a safety mechanism).

如果 TCP 连接仍然有效，数据可以在缓冲区中保留很长时间。现在，套接字和 Redis 输出缓冲区都已绑定。如果订阅者真的太慢了​​，大量数据积累，Redis 最终会关闭与订阅者的连接（作为一种安全机制）。

By default, for pub/sub, Redis has a soft limit at 8 MB, and a hard limit at 32 MB, per connection buffer. If the output buffer reaches the hard limit, or if it remains between the soft and hard limit for more than 60 seconds, the connection with the slow subscriber will be closed.

默认情况下，对于 pub/sub，Redis 每个连接缓冲区的软限制为 8 MB，硬限制为 32 MB。如果输出缓冲区达到硬限制，或者在软限制和硬限制之间保持超过 60 秒，将关闭与慢速订阅者的连接。

Knowing the number of pending messages is not easy. It can be evaluated by looking at the size of the pending information in the socket buffers, and the Redis output buffers.

知道待处理消息的数量并不容易。可以通过查看套接字缓冲区和 Redis 输出缓冲区中待处理信息的大小来评估它。

For Redis output buffers, you can use the CLIENT LIST command(from redis-cli). The size of the output buffer is returned in the obl and oll fields (in bytes).

对于 Redis 输出缓冲区，您可以使用CLIENT LIST 命令（来自 redis-cli）。输出缓冲区的大小在 obl 和 oll 字段中返回（以字节为单位）。

For socket buffers, there is no Redis command. However, on Linux, it is possible to build a script to interpret the content of the /proc/net/tcp file. See an example here. This script probably needs to be adapted to your system.

对于套接字缓冲区，没有 Redis 命令。但是，在 Linux 上，可以构建脚本来解释 /proc/net/tcp 文件的内容。请参阅此处的示例。该脚本可能需要适应您的系统。