如何在python请求库中实现重试机制？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/23267409/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to implement retry mechanism into python requests library?
提问by sorin
I would like to add a retry mechanism to python request library, so scripts that are using it will retry for non fatal errors.
我想向 python 请求库添加重试机制，因此使用它的脚本将重试非致命错误。
At this moment I do consider three kind of errors to be recoverable:
目前我确实认为可以恢复三种错误：
- HTTP return codes 502, 503, 504
- host not found (less important now)
- request timeout
- HTTP 返回码 502、503、504
- 找不到主机（现在不太重要）
- 请求超时
At the first stage I do want to retry specified 5xx requests every minute.
在第一阶段，我确实想每分钟重试指定的 5xx 请求。
I want to be able to add this functionality transparently, without having to manually implement recovery for each HTTP call made from inside these scripts or libraries that are using python-requests.
我希望能够透明地添加此功能，而不必为从这些使用 python 请求的脚本或库内部进行的每个 HTTP 调用手动实现恢复。
采纳答案by sorin
I was able to obtain the desired level of reliability by extending requests.Sessionclass.
我能够通过扩展requests.Session类获得所需的可靠性水平。
Here is the code https://bitbucket.org/bspeakmon/jira-python/src/a7fca855394402f58507ca4056de87ccdbd6a213/jira/resilientsession.py?at=master
EDITThat code was:
编辑该代码是：
from requests import Session
from requests.exceptions import ConnectionError
import logging
import time
class ResilientSession(Session):
"""
This class is supposed to retry requests that do return temporary errors.
At this moment it supports: 502, 503, 504
"""
def __recoverable(self, error, url, request, counter=1):
if hasattr(error,'status_code'):
if error.status_code in [502, 503, 504]:
error = "HTTP %s" % error.status_code
else:
return False
DELAY = 10 * counter
logging.warn("Got recoverable error [%s] from %s %s, retry #%s in %ss" % (error, request, url, counter, DELAY))
time.sleep(DELAY)
return True
def get(self, url, **kwargs):
counter = 0
while True:
counter += 1
try:
r = super(ResilientSession, self).get(url, **kwargs)
except ConnectionError as e:
r = e.message
if self.__recoverable(r, url, 'GET', counter):
continue
return r
def post(self, url, **kwargs):
counter = 0
while True:
counter += 1
try:
r = super(ResilientSession, self).post(url, **kwargs)
except ConnectionError as e:
r = e.message
if self.__recoverable(r, url, 'POST', counter):
continue
return r
def delete(self, url, **kwargs):
counter = 0
while True:
counter += 1
try:
r = super(ResilientSession, self).delete(url, **kwargs)
except ConnectionError as e:
r = e.message
if self.__recoverable(r, url, 'DELETE', counter):
continue
return r
def put(self, url, **kwargs):
counter = 0
while True:
counter += 1
try:
r = super(ResilientSession, self).put(url, **kwargs)
except ConnectionError as e:
r = e.message
if self.__recoverable(r, url, 'PUT', counter):
continue
return r
def head(self, url, **kwargs):
counter = 0
while True:
counter += 1
try:
r = super(ResilientSession, self).head(url, **kwargs)
except ConnectionError as e:
r = e.message
if self.__recoverable(r, url, 'HEAD', counter):
continue
return r
def patch(self, url, **kwargs):
counter = 0
while True:
counter += 1
try:
r = super(ResilientSession, self).patch(url, **kwargs)
except ConnectionError as e:
r = e.message
if self.__recoverable(r, url, 'PATCH', counter):
continue
return r
def options(self, url, **kwargs):
counter = 0
while True:
counter += 1
try:
r = super(ResilientSession, self).options(url, **kwargs)
except ConnectionError as e:
r = e.message
if self.__recoverable(r, url, 'OPTIONS', counter):
continue
return r
回答by Mikel Emaldi Manrique
This is a snippet of code I used to retry for the petitions made with urllib2. Maybe you could use it for your purposes:
这是我用来重试使用 urllib2 提交的请求的代码片段。也许您可以将它用于您的目的：
retries = 1
success = False
while not success:
try:
response = urllib2.urlopen(request)
success = True
except Exception as e:
wait = retries * 30;
print 'Error! Waiting %s secs and re-trying...' % wait
sys.stdout.flush()
time.sleep(wait)
retries += 1
The waiting time grows incrementally to avoid be banned from server.
等待时间逐渐增加以避免被服务器禁止。
回答by datashaman
This snippet of code will make all HTTP requests from the same session retry for a total of 5 times, sleeping between retries with an increasing backoff of 0s, 2s, 4s, 8s, 16s (the first retry is done immediately). It will retry on basic connectivity issues (including DNS lookup failures), and HTTP status codes of 502, 503 and 504.
这段代码将使来自同一会话的所有 HTTP 请求重试总共 5 次，在重试之间休眠，并逐渐增加 0s、2s、4s、8s、16s（第一次重试立即完成）。它将重试基本连接问题（包括 DNS 查找失败）以及 HTTP 状态代码 502、503 和 504。
import logging
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
logging.basicConfig(level=logging.DEBUG)
s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])
s.mount('http://', HTTPAdapter(max_retries=retries))
s.get("http://httpstat.us/503")
See Retry classfor details.
有关详细信息，请参阅重试类。
回答by Daniil Mashkin
Possible solution using retrying package
使用重试包的可能解决方案
from retrying import retry
import requests
def retry_if_connection_error(exception):
""" Specify an exception you need. or just True"""
#return True
return isinstance(exception, ConnectionError)
# if exception retry with 2 second wait
@retry(retry_on_exception=retry_if_connection_error, wait_fixed=2000)
def safe_request(url, **kwargs):
return requests.get(url, **kwargs)
response = safe_request('test.com')
回答by sarjit07
Its mostly the logic in Java. Can try looking at it. Working fine.
它主要是Java中的逻辑。可以试试看。工作正常。
public int callAPI() {
return 1; //some method to be retried
}
public int retrylogic() throws InterruptedException, IOException{
int retry = 0;
int status = -1;
boolean delay = false;
do {
if (delay) {
Thread.sleep(2000);
}
try {
status = callAPI();
}
catch (Exception e) {
System.out.println("Error occured");
status = -1;
}
finally {
switch (status) {
case 200:
System.out.println(" **OK**");
return status;
default:
System.out.println(" **unknown response code**.");
break;
}
retry++;
System.out.println("Failed retry " + retry + "/" + 3);
delay = true;
}
}while (retry < 3);
return status;
}
