Python 使用 boto3 时 S3 连接超时
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/41263304/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
S3 Connection timeout when using boto3
提问by Python Novice
I am using boto3 to operate with S3. If my application is unable to reach S3 due to a network issue, the connection will hang until eventually it times out. I would like to set a lower connection timeout. I came across this PRfor botocore that allows setting a timeout:
我正在使用 boto3 来操作 S3。如果我的应用程序由于网络问题而无法访问 S3，则连接将挂起，直到最终超时。我想设置一个较低的连接超时。我遇到了这个允许设置超时的 botocore公关：
$ sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP
from botocore.client import Config
import boto3
config = Config(connect_timeout=5, read_timeout=5)
s3 = boto3.client('s3', config=config)
s3.head_bucket(Bucket='my-s3-bucket')
This throws a ConnectTimeout, but it still takes too long to error out:
这会引发 ConnectTimeout，但仍然需要很长时间才能出错：
ConnectTimeout: HTTPSConnectionPool(host='my-s3-bucket.s3.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x2ad5dd0>, 'Connection to my-s3-bucket.s3.amazonaws.com timed out. (connect timeout=5)'))
Tweaking both the connect and read timeouts doesn't impact how quickly the connection responds.
调整连接和读取超时不会影响连接响应的速度。
回答by llude
You are probably getting bitten by boto3's default behaviour of retrying connections multiple times and exponentially backing off in between. I had good results with the following:
您可能会被 boto3 的默认行为所困扰，即多次重试连接并在其间以指数方式后退。我在以下方面取得了不错的成绩：
from botocore.client import Config
import boto3
config = Config(connect_timeout=5, retries={'max_attempts': 0})
s3 = boto3.client('s3', config=config)
回答by EM Bee
Did you ever get this resolved? My suspicion is that you need the credentials for your boto connection.
你有没有解决过这个问题？我怀疑您需要 boto 连接的凭据。
Here is how I connect to boto3:
这是我连接到 boto3 的方式：
import boto3
from botocore.exceptions import ClientError
import re
from io import BytesIO
import gzip
import datetime
import dateutil.parser as dparser
from datetime import datetime
import tarfile
import requests
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
## Needed glue stuff
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
##
## currently this will run for everything that is in the staging directory of omniture
# set needed parms
myProfileName = 'MyDataLake'
dhiBucket = 'data-lake'
#create boto3 session
try:
session = boto3.Session(aws_access_key_id='aaaaaaaaaaaa', aws_secret_access_key='abcdefghijklmnopqrstuvwxyz', region_name='us-east-1', aws_session_token=None, region_name=None, botocore_session=None)
s3 = session.resource('s3') # establish connection to s3
except Exception as conne:
print ("Unable to connect: " + str(conne))
errtxt = requests.post("https://errorcapturesite", data= {'message':'Unable to connect to : ' + myProfileName, 'notify':True,'color':'red'})
print(errtxt.text)
exit()
