Python 请求：不要等待请求完成

Python Requests: Don't wait for request to finish

提问by octosquidopus

In Bash, it is possible to execute a command in the background by appending &. How can I do it in Python?

在 Bash 中，可以通过附加&. 我怎样才能在 Python 中做到这一点？

while True:
    data = raw_input('Enter something: ') 
    requests.post(url, data=data) # Don't wait for it to finish.
    print('Sending POST request...') # This should appear immediately.

采纳答案by Andrew Gorcester

I use multiprocessing.dummy.Pool. I create a singleton thread pool at the module level, and then use pool.apply_async(requests.get, [params])to launch the task.

我用multiprocessing.dummy.Pool. 我在模块级别创建了一个单例线程池，然后用于pool.apply_async(requests.get, [params])启动任务。

This command gives me a future, which I can add to a list with other futures indefinitely until I'd like to collect all or some of the results.

这个命令给了我一个未来，我可以无限期地将它与其他期货一起添加到列表中，直到我想收集全部或部分结果。

multiprocessing.dummy.Poolis, against all logic and reason, a THREAD pool and not a process pool.

multiprocessing.dummy.Pool违反所有逻辑和理由，是线程池而不是进程池。

Example (works in both Python 2 and 3, as long as requests is installed):

示例（适用于 Python 2 和 3，只要安装了 requests）：

from multiprocessing.dummy import Pool

import requests

pool = Pool(10) # Creates a pool with ten threads; more threads = more concurrency.
                # "pool" is a module attribute; you can be sure there will only
                # be one of them in your application
                # as modules are cached after initialization.

if __name__ == '__main__':
    futures = []
    for x in range(10):
        futures.append(pool.apply_async(requests.get, ['http://example.com/']))
    # futures is now a list of 10 futures.
    for future in futures:
        print(future.get()) # For each future, wait until the request is
                            # finished and then print the response object.

The requests will be executed concurrently, so running all ten of these requests should take no longer than the longest one. This strategy will only use one CPU core, but that shouldn't be an issue because almost all of the time will be spent waiting for I/O.

这些请求将并发执行，因此运行所有 10 个请求的时间不应超过最长的一个。此策略将仅使用一个 CPU 内核，但这应该不是问题，因为几乎所有时间都将花费在等待 I/O 上。

回答by Romain Jouin

According to the doc, you should move to another library :

根据文档，您应该移动到另一个库：

Blocking Or Non-Blocking?

With the default Transport Adapter in place, Requests does not provide any kind of non-blocking IO. The Response.content property will block until the entire response has been downloaded. If you require more granularity, the streaming features of the library (see Streaming Requests) allow you to retrieve smaller quantities of the response at a time. However, these calls will still block.

If you are concerned about the use of blocking IO, there are lots of projects out there that combine Requests with one of Python's asynchronicity frameworks.

Two excellent examples are grequestsand requests-futures.

阻塞还是非阻塞？

使用默认传输适配器，请求不提供任何类型的非阻塞 IO。Response.content 属性将一直阻塞，直到整个响应被下载。如果您需要更多粒度，库的流功能（请参阅流请求）允许您一次检索更少量的响应。但是，这些调用仍然会阻塞。

如果您担心阻塞 IO 的使用，有很多项目将请求与 Python 的异步框架之一相结合。

两个很好的例子是 grequests和 requests-futures。

回答by Chosmos

If you can write the code to be executed separately in a separate python program, hereis a possible solution based on subprocessing.

如果可以在单独的python程序中编写单独执行的代码，这里有一个基于子处理的可能解决方案。

Otherwise you may find useful this questionand related answer: the trick is to use the threading library to start a separate thread that will execute the separated task.

否则你可能会发现这个问题和相关答案很有用：诀窍是使用线程库来启动一个单独的线程来执行分离的任务。

A caveat with both approach could be the number of items (that's to say the number of threads) you have to manage. If the items in parentare too many, you may consider halting every batch of items till at least some threads have finished, but I think this kind of management is non-trivial.

这两种方法的一个警告可能是您必须管理的项目数（即线程数）。如果itemsparent太多，你可以考虑暂停每批项目，直到至少一些线程完成，但我认为这种管理是不平凡的。

For more sophisticated approach you can use an actor based approach, I have not used this librarymyself but I think it could help in that case.

对于更复杂的方法，您可以使用基于演员的方法，我自己没有使用过这个库，但我认为在这种情况下它会有所帮助。

回答by keithhackbarth

Here's a hacky way to do it:

这是一个hacky的方法：

try:
    requests.get("http://127.0.0.1:8000/test/",timeout=0.0000000001)
except requests.exceptions.ReadTimeout: 
    pass

回答by Nemolovich

Elegant solution from Andrew Gorcester. In addition, without using futures, it is possible to use the callbackand error_callbackattributes (see doc) in order to perform asynchronous processing:

来自Andrew Gorcester 的优雅解决方案。此外，在不使用期货的情况下，可以使用callback和error_callback属性（参见 doc）来执行异步处理：

def on_success(r: Response):
    if r.status_code == 200:
        print(f'Post succeed: {r}')
    else:
        print(f'Post failed: {r}')

def on_error(ex: Exception):
    print(f'Post requests failed: {ex}')

pool.apply_async(requests.post, args=['http://server.host'], kwargs={'json': {'key':'value'},
                        callback=on_success, error_callback=on_error))