java REST 服务前的请求队列

Request Queue in front of a REST Service

提问by TheWhiteRabbit

What is the best technology solution (framework/approach) to have a Request Queue in front of a REST service. so that i can increase the no of instances of REST service for higher availability and by placing Request queue in front to form a service/transaction boundary for the service client.

在 REST 服务前设置请求队列的最佳技术解决方案（框架/方法）是什么？这样我就可以增加 REST 服务实例的数量以获得更高的可用性，并通过将请求队列放在前面以形成服务客户端的服务/事务边界。

1. I need good and lightweight technology/framework choice for Request Queue (java)
2. Approach to implement a competing consumer with it.

1. 我需要为请求队列 (java) 选择好的轻量级技术/框架
2. 使用它实现竞争消费者的方法。

回答by Will Hartung

There's a couple of issues here, depending on your goals.

这里有几个问题，具体取决于您的目标。

First, it only promotes availability of the resources on the back end. Consider if you have 5 servers handling queue requests on the back end. If one of those servers goes down, then the queued request should fall back in to the queue, and be redelivered to one of the remaining 4 servers.

首先，它只促进后端资源的可用性。考虑是否有 5 个服务器在后端处理队列请求。如果其中一台服务器出现故障，则排队的请求应回落到队列中，并重新发送到其余 4 台服务器中的一台。

However, while those back end servers are processing, the front end servers are holding on to the actual, initiating requests. If one of those front end servers fails, then those connections are lost completely, and it will be up to the original client to resubmit the request.

然而，当这些后端服务器正在处理时，前端服务器会保留实际的发起请求。如果这些前端服务器之一出现故障，则这些连接将完全丢失，由原始客户端重新提交请求。

The premise perhaps is that simpler front end systems are at a lower risk for failure, and that's certainly true for software related failure. But networks cards, power supplies, hard drives, etc. are pretty agnostic to such false hopes of man and punish all equally. So, consider this when talking about overall availability.

前提可能是更简单的前端系统发生故障的风险较低，这对于与软件相关的故障来说当然是正确的。但是网卡、电源、硬盘驱动器等对人类的这种虚假希望完全不可知，并且一视同仁。因此，在谈论整体可用性时，请考虑这一点。

As to design, the back end is a simple process waiting upon a JMS message queue, and processing each message as they come. There are a multitude of examples of this available, and any JMS server will suit at a high level. All you need is to ensure that the message handling is transactional so that if a message processing fails, the message remains in the queue and can be redelivered to another message handler.

就设计而言，后端是一个简单的过程，等待 JMS 消息队列，并在每条消息到来时对其进行处理。有很多可用的示例，任何 JMS 服务器都适用于高级别。您所需要的只是确保消息处理是事务性的，这样如果消息处理失败，消息仍会保留在队列中，并且可以重新传送到另一个消息处理程序。

Your JMS queue's primary requirement is being clusterable. The JMS server itself is a single point of failure in the system. Lost the JMS server, and your system is pretty much dead in the water, so you'll need to be able to cluster the server and have the consumers and producers handle failover appropriately. Again, this is JMS server specific, most do it, but it's pretty routine in the JMS world.

您的 JMS 队列的主要要求是可集群化。JMS 服务器本身是系统中的单点故障。失去了 JMS 服务器，您的系统几乎完全陷入困境，因此您需要能够对服务器进行集群并让消费者和生产者适当地处理故障转移。同样，这是特定于 JMS 服务器的，大多数都这样做，但它在 JMS 世界中很常见。

The front end is where things get a little trickier, since the front end servers are the bridge from the synchronous world of the REST request to the asynchronous world of the back end processors. A REST request follows a typically RPC pattern of consuming the request payload from the socket, holding the connection open, processing the results, and delivering the results back down the originating socket.

前端是事情变得有点棘手的地方，因为前端服务器是从 REST 请求的同步世界到后端处理器的异步世界的桥梁。REST 请求遵循典型的 RPC 模式，即使用来自套接字的请求有效负载、保持连接打开、处理结果并将结果传递回原始套接字。

To manifest this hand off, you should take a look at the Asynchronous Servlet handling the Servlet 3.0 introduced, and is available in Tomcat 7, the latest Jetty (not sure what version), Glassfish 3.x, and others.

要证明这一点，您应该查看引入的 Servlet 3.0 处理异步 Servlet，并且在 Tomcat 7、最新的 Jetty（不确定哪个版本）、Glassfish 3.x 和其他版本中可用。

In this case what you would do is when the request arrives, you convert the nominally synchronous Servlet call in to an Asynchronous call using HttpServletRequest.startAsync(HttpServletRequest request, HttpServletResponse response).

在这种情况下，您要做的是在请求到达时，使用 将名义上同步的 Servlet 调用转换为异步调用HttpServletRequest.startAsync(HttpServletRequest request, HttpServletResponse response)。

This returns an AsynchronousContext, and once started, allows the server to free up the processing thread. You then do several things.

这将返回一个 AsynchronousContext，并且一旦启动，就允许服务器释放处理线程。然后你做几件事。

1. Extract the parameters from the request.
2. Create a unique ID for the request.
3. Create a new back end request payload from your parameters.
4. Associate the ID with the AsyncContext, and retain the context (such as putting it in to a application wide Map).
5. Submit the back end request to the JMS queue.

1. 从请求中提取参数。
2. 为请求创建唯一 ID。
3. 根据您的参数创建新的后端请求负载。
4. 将 ID 与 AsyncContext 关联，并保留上下文（例如将其放入应用程序范围的 Map）。
5. 将后端请求提交到 JMS 队列。

At this point, the initial processing is done, and you simply return from doGet (or service, or whatever). Since you have not called AsyncContext.complete(), the server will not close out the connection to the server. Since you have the AsyncContext store in the map by the ID, it's handy for safe keeping for the time being.

此时，初始处理完成，您只需从 doGet（或服务，或其他）返回。由于您没有调用 AsyncContext.complete()，服务器不会关闭与服务器的连接。由于您通过 ID 在地图中存储了 AsyncContext，因此暂时安全保存起来很方便。

Now, when you submitted the request to the JMS queue, it contained: the ID of the request (that you generated), any parameters for the request, and the identification of the actual server making the request. This last bit is important as the results of the processing needs to return to its origin. The origin is identified by the request ID and the server ID.

现在，当您向 JMS 队列提交请求时，它包含：请求的 ID（您生成的）、请求的任何参数以及发出请求的实际服务器的标识。最后一点很重要，因为处理的结果需要返回到它的原点。源由请求 ID 和服务器 ID 标识。

When your front end server started up, it also started a thread who's job it is to listen to a JMS response queue. When it sets up its JMS connection, it can set up a filter such as "Give me only messages for a ServerID of ABC123". Or, you could create a unique queue for each front end server and the back end server uses the server ID to determine the queue to return the reply to.

当您的前端服务器启动时，它还启动了一个线程，该线程的工作是侦听 JMS 响应队列。当它建立它的 JMS 连接时，它可以设置一个过滤器，例如“只给我一个服务器 ID 为 ABC123 的消息”。或者，您可以为每个前端服务器创建一个唯一的队列，后端服务器使用服务器 ID 来确定将回复返回到的队列。

When the back end processors consume the message, they're take the request ID, and parameters, perform the work, and then take the result and put them on to the JMS response Queue. When it puts it the result back, it'll add the originating ServerID and the original Request ID as properties of the message.

当后端处理器使用消息时，它们会获取请求 ID 和参数，执行工作，然后获取结果并将它们放入 JMS 响应队列。当它返回结果时，它将添加原始 ServerID 和原始请求 ID 作为消息的属性。

So, if you got the request originally for Front End Server ABC123, the back end processor will address the results back to that server. Then, that listener thread will be notified when it gets a message. The listener threads task is to take that message and put it on to an internal queue within the front end server.

因此，如果您最初收到前端服务器 ABC123 的请求，后端处理器会将结果寻址回该服务器。然后，该侦听器线程将在收到消息时收到通知。侦听器线程的任务是接收该消息并将其放入前端服务器内的内部队列中。

This internal queue is backed by a thread pool who's job is to send the request payloads back to the original connection. It does this by extracting the original request ID from the message, looking up the AsyncContext from that internal map discussed earlier, and then sending results down to the HttpServletResponse associated with the AsyncContext. At the end, it call AsyncContext.complete() (or a similar method) to tell the server that you're done and to allow it to release the connection.

这个内部队列由一个线程池支持，该线程池的工作是将请求有效负载发送回原始连接。它通过从消息中提取原始请求 ID，从前面讨论的内部映射中查找 AsyncContext，然后将结果向下发送到与 AsyncContext 关联的 HttpServletResponse 来完成此操作。最后，它调用 AsyncContext.complete()（或类似的方法）来告诉服务器您已完成并允许它释放连接。

For housekeeping, you should have another thread on the front end server who's job it is to detect when requests have been waiting in the map for too long. Part of the original message should have been a time the request started. This thread can wake up every second, scan the map for requests, and for any that have been there too long (say 30 seconds), it can put the request on to another internal queue, consumed by a collection of handlers designed to inform the client that the request timed out.

对于内务管理，您应该在前端服务器上有另一个线程，它的工作是检测请求何时在地图中等待的时间过长。原始消息的一部分应该是请求开始的时间。这个线程可以每秒唤醒一次，扫描地图上的请求，对于任何已经存在太长时间（比如 30 秒）的请求，它可以将请求放到另一个内部队列中，由一组处理程序消耗，这些处理程序旨在通知请求超时的客户端。

You want these internal queues so that the main processing logic isn't stuck waiting on the client to consume the data. It could be a slow connection or something, so you don't want to block all of the other pending requests to handle them one by one.

您需要这些内部队列，以便主处理逻辑不会卡在等待客户端使用数据。这可能是连接缓慢或其他原因，因此您不想阻止所有其他未决请求来一一处理它们。

Finally, you'll need to account that you may well get a message from the response queue for a request that no longer exists in your internal map. For one, the request may have timed out, so it should not be there any longer. For another, that front end server may have stopped and been restarted, so it internal map of pending request will simply be empty. At this point, if you detect you have a reply for a request that no longer exists, you should simply discard it (well, log it, then discard it).

最后，您需要考虑到您很可能会从响应队列中收到一条消息，用于处理内部映射中不再存在的请求。一方面，请求可能已超时，因此它不应再存在。另一方面，该前端服务器可能已停止并重新启动，因此挂起请求的内部映射将只是空的。此时，如果您检测到对不再存在的请求的回复，您应该简单地丢弃它（好吧，记录它，然后丢弃它）。

You can't reuse these requests, there's not such thing really as a load balancer going back to the client. If the client is allowing you to make callbacks via published end points, then, sure you can just have another JMS message handler make those requests. But that's not a REST kind of thing, REST at this level of discussion is more client/server/RPC.

你不能重用这些请求，真的没有负载均衡器返回客户端这样的事情。如果客户端允许您通过已发布的端点进行回调，那么请确保您可以让另一个 JMS 消息处理程序发出这些请求。但这不是 REST 类型的事情，在这个级别的讨论中 REST 更像是客户端/服务器/RPC。

As to which framework support Asynchronous Servlets at a higher level than a raw Servlet, (such as Jersey for JAX-RS or something like that), I can't say. I don't know what frameworks are supporting it at that level. Seems like this is a feature of Jersey 2.0, which is not out yet. There well may be others, you'll have to look around. Also, don't fixate on Servlet 3.0. Servlet 3.0 is simply a standardization of techniques used in individual containers for some time (Jetty notably), so you may want to look at container specific options outside of just Servlet 3.0.

至于哪个框架在比原始 Servlet 更高的级别上支持异步 Servlet（例如 JAX-RS 的 Jersey 或类似的东西），我不能说。我不知道在那个级别支持它的框架是什么。好像这是 Jersey 2.0 的一个特性，还没有出来。可能还有其他人，你必须环顾四周。另外，不要专注于 Servlet 3.0。Servlet 3.0 只是一段时间（尤其是 Jetty）在单个容器中使用的技术的标准化，因此您可能需要查看 Servlet 3.0 之外的容器特定选项。

But the concepts are the same. The big takeaway are the response queue listener with the filtered JMS connection, the internal request map to the AsyncContext, and the internal queues and thread pools to do the actual work within the application.

但是概念是一样的。最大的收获是响应队列侦听器和过滤后的 JMS 连接，内部请求映射到 AsyncContext，以及在应用程序中执行实际工作的内部队列和线程池。

回答by Steven Schlansker

If you relax your requirement that it must be in Java, you could consider HAProxy. It's very lightweight, very standard, and does a lot of good things (request pooling / keepalives / queueing) well.

如果您放宽必须使用 Java 的要求，则可以考虑使用 HAProxy。它非常轻量级，非常标准，并且在很多方面做得很好（请求池/保活/排队）。

Think twice before you implement request queueing, though. Unless your traffic is extremely bursty it will do nothing but hurt your system's performance under load.

不过，在实现请求排队之前要三思。除非您的流量非常突发，否则它只会在负载下损害您的系统性能。

Assume that your system can handle 100 requests per second. Your HTTP server has a bounded worker thread pool. The only way a request pool can help is if you are receiving more than 100 requests per second. After your worker thread pool is full, requests start to pile up in your load balancer pool. Since they are arriving faster than you can handle them, the queue gets bigger ... and bigger ... and bigger. Eventually either this pool fills too, or you run out of RAM and the load balancer (and thus the entire system) crashes hard.

假设您的系统每秒可以处理 100 个请求。您的 HTTP 服务器有一个有界工作线程池。请求池可以提供帮助的唯一方法是您每秒收到 100 个以上的请求。在您的工作线程池已满后，请求开始在您的负载均衡器池中堆积。由于它们到达的速度比您处理它们的速度快，队列变得越来越大……越来越大……越来越大。最终要么这个池也填满了，要么你的 RAM 用完了，负载平衡器（以及整个系统）就会严重崩溃。

If your web server is too busy, start rejecting requests and get some additional capacity online.

如果您的 Web 服务器太忙，请开始拒绝请求并在线获取一些额外容量。

Request pooling certainly can help if you can get additional capacity in time to handle the requests. It can also hurt you really badly. Think through the consequences before turning on a secondary request pool in front of your HTTP server's worker thread pool.

如果您能及时获得额外的容量来处理请求，请求池肯定会有所帮助。它也会严重伤害你。在您的 HTTP 服务器的工作线程池之前打开辅助请求池之前，请仔细考虑后果。

回答by moskiteau

The design we use is a a REST interface receiving all the request and dispatching them to a message queue (i.e. Rabbitmq)

我们使用的设计是一个 REST 接口，接收所有请求并将它们分派到消息队列（即 Rabbitmq）

Then workers listen to the messages and execute them following certain rules. If everything goes down you would still have the request in the MQ and if you have a high number of request you can just add workers...

然后工作人员听取消息并按照某些规则执行它们。如果一切都失败了，您仍然会在 MQ 中收到请求，如果您有大量请求，您可以添加工作人员...

Check this keynote, it kind of shows the power of this concept!

检查这个主题演讲，它展示了这个概念的力量！

http://www.springsource.org/SpringOne2GX2012

http://www.springsource.org/SpringOne2GX2012