Python 我如何在 asyncio 中使用请求？

How could I use requests in asyncio?

提问by flyer

I want to do parallel http request tasks in asyncio, but I find that python-requestswould block the event loop of asyncio. I've found aiohttpbut it couldn't provide the service of http request using a http proxy.

我想要做的并行http请求的任务asyncio，但我发现，python-requests会阻止事件循环asyncio。我找到了aiohttp，但它无法使用 http 代理提供 http 请求服务。

So I want to know if there's a way to do asynchronous http requests with the help of asyncio.

所以我想知道是否有办法在asyncio.

采纳答案by christian

To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executorto run a function in another thread and yield from it to get the result. For example:

要将请求（或任何其他阻塞库）与 asyncio 一起使用，您可以使用BaseEventLoop.run_in_executor在另一个线程中运行一个函数并从中产生结果以获得结果。例如：

import asyncio
import requests

@asyncio.coroutine
def main():
    loop = asyncio.get_event_loop()
    future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
    future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
    response1 = yield from future1
    response2 = yield from future2
    print(response1.text)
    print(response2.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

This will get both responses in parallel.

这将同时获得两个响应。

With python 3.5 you can use the new await/asyncsyntax:

使用 python 3.5，您可以使用新的await/async语法：

import asyncio
import requests

async def main():
    loop = asyncio.get_event_loop()
    future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
    future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
    response1 = await future1
    response2 = await future2
    print(response1.text)
    print(response2.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

See PEP0492for more.

有关更多信息，请参阅PEP0492。

回答by Lukasa

Requests does not currently support asyncioand there are no plans to provide such support. It's likely that you could implement a custom "Transport Adapter" (as discussed here) that knows how to use asyncio.

Requests 目前不支持asyncio，也没有计划提供此类支持。这可能是因为你可以实现一个自定义的“传输适配器”（如讨论这里），它知道如何使用asyncio。

If I find myself with some time it's something I might actually look into, but I can't promise anything.

如果我发现自己有一些时间，我可能会真正研究一下，但我不能保证任何事情。

回答by mindmaster

aiohttpcan be used with HTTP proxy already:

aiohttp已经可以与 HTTP 代理一起使用了：

import asyncio
import aiohttp


@asyncio.coroutine
def do_request():
    proxy_url = 'http://localhost:8118'  # your proxy address
    response = yield from aiohttp.request(
        'GET', 'http://google.com',
        proxy=proxy_url,
    )
    return response

loop = asyncio.get_event_loop()
loop.run_until_complete(do_request())

回答by Ilya Rusin

There is a good case of async/await loops and threading in an article by Pimin Konstantin Kefaloukos Easy parallel HTTP requests with Python and asyncio:

在 Pimin Konstantin Kefaloukos 的一篇文章中，有一个很好的 async/await 循环和线程案例， 使用 Python 和 asyncio 轻松并行 HTTP 请求：

To minimize the total completion time, we could increase the size of the thread pool to match the number of requests we have to make. Luckily, this is easy to do as we will see next. The code listing below is an example of how to make twenty asynchronous HTTP requests with a thread pool of twenty worker threads:

为了最小化总完成时间，我们可以增加线程池的大小以匹配我们必须发出的请求数量。幸运的是，这很容易做到，我们将在接下来看到。下面的代码清单是如何使用 20 个工作线程的线程池发出 20 个异步 HTTP 请求的示例：

# Example 3: asynchronous requests with larger thread pool
import asyncio
import concurrent.futures
import requests

async def main():

    with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:

        loop = asyncio.get_event_loop()
        futures = [
            loop.run_in_executor(
                executor, 
                requests.get, 
                'http://example.org/'
            )
            for i in range(20)
        ]
        for response in await asyncio.gather(*futures):
            pass


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

回答by ospider

The answers above are still using the old Python 3.4 style coroutines. Here is what you would write if you got Python 3.5+.

上面的答案仍然使用旧的 Python 3.4 样式协程。如果你有 Python 3.5+，这就是你会写的东西。

aiohttpsupportshttp proxy now

aiohttp现在支持http代理

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    urls = [
            'http://python.org',
            'https://google.com',
            'http://yifei.me'
        ]
    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in urls:
            tasks.append(fetch(session, url))
        htmls = await asyncio.gather(*tasks)
        for html in htmls:
            print(html[:100])

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())