我可以更改 Python 的“请求”模块的连接池大小吗？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/18466079/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Can I change the connection pool size for Python's "requests" module?
提问by Skip Huffman
(edit: Perhaps I am wrong in what this error means. Is this indicating that the connection pool at my CLIENT is full? or a connection pool at the SERVER is full and this is the error my client is being given?)
（编辑：也许我在这个错误的含义上错了。这是否表明我的客户端的连接池已满？或者服务器上的连接池已满，这是我的客户端出现的错误？）
I am attempting to make a large number of httprequests concurrently using the python threadingand requestsmodule. I am seeing this error in logs:
我正在尝试http使用 pythonthreading和requests模块同时发出大量请求。我在日志中看到此错误：
WARNING:requests.packages.urllib3.connectionpool:HttpConnectionPool is full, discarding connection:
What can I do to increase the size of the connection pool for requests?
我可以做些什么来增加请求的连接池的大小？
回答by Jahaja
This should do the trick:
这应该可以解决问题：
import requests
sess = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=100, pool_maxsize=100)
sess.mount('http://', adapter)
resp = sess.get("/mypage")
回答by Michael_Scharf
Note:Use this solution only if you cannot control the construction of the connection pool (as described in @Jahaja's answer).
注意：仅当您无法控制连接池的构造时才使用此解决方案（如@Jahaja 的回答中所述）。
The problem is that the urllib3creates the pools on demand. It calls the constructor of the urllib3.connectionpool.HTTPConnectionPoolclass without parameters. The classes are registered in urllib3 .poolmanager.pool_classes_by_scheme. The trick is to replace the classes with your classes that have different default parameters:
问题是urllib3按需创建池。它urllib3.connectionpool.HTTPConnectionPool不带参数调用类的构造函数。类注册在urllib3 .poolmanager.pool_classes_by_scheme. 诀窍是用具有不同默认参数的类替换类：
def patch_http_connection_pool(**constructor_kwargs):
"""
This allows to override the default parameters of the
HTTPConnectionPool constructor.
For example, to increase the poolsize to fix problems
with "HttpConnectionPool is full, discarding connection"
call this function with maxsize=16 (or whatever size
you want to give to the connection pool)
"""
from urllib3 import connectionpool, poolmanager
class MyHTTPConnectionPool(connectionpool.HTTPConnectionPool):
def __init__(self, *args,**kwargs):
kwargs.update(constructor_kwargs)
super(MyHTTPConnectionPool, self).__init__(*args,**kwargs)
poolmanager.pool_classes_by_scheme['http'] = MyHTTPConnectionPool
Then you can call to set new default parameters. Make sure this is called before any connection is made.
然后你可以调用来设置新的默认参数。确保在建立任何连接之前调用它。
patch_http_connection_pool(maxsize=16)
If you use https connections you can create a similar function:
如果您使用 https 连接，您可以创建一个类似的功能：
def patch_https_connection_pool(**constructor_kwargs):
"""
This allows to override the default parameters of the
HTTPConnectionPool constructor.
For example, to increase the poolsize to fix problems
with "HttpSConnectionPool is full, discarding connection"
call this function with maxsize=16 (or whatever size
you want to give to the connection pool)
"""
from urllib3 import connectionpool, poolmanager
class MyHTTPSConnectionPool(connectionpool.HTTPSConnectionPool):
def __init__(self, *args,**kwargs):
kwargs.update(constructor_kwargs)
super(MyHTTPSConnectionPool, self).__init__(*args,**kwargs)
poolmanager.pool_classes_by_scheme['https'] = MyHTTPSConnectionPool
