Python 使用 boto3 从 S3 存储桶读取文件内容
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/36205481/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Read file content from S3 bucket with boto3
提问by mar tin
I read the filenames in my S3 bucket by doing
我通过执行读取 S3 存储桶中的文件名
objs = boto3.client.list_objects(Bucket='my_bucket')
while 'Contents' in objs.keys():
objs_contents = objs['Contents']
for i in range(len(objs_contents)):
filename = objs_contents[i]['Key']
Now, I need to get the actual content of the file, similarly to a open(filename).readlines(). What is the best way?
现在，我需要获取文件的实际内容，类似于open(filename).readlines(). 什么是最好的方法？
回答by Jordon Phillips
boto3 offers a resource model that makes tasks like iterating through objects easier. Unfortunately, StreamingBody doesn't provide readlineor readlines.
boto3 提供了一个资源模型，使诸如遍历对象之类的任务更容易。不幸的是，StreamingBody 不提供readline或readlines。
s3 = boto3.resource('s3')
bucket = s3.Bucket('test-bucket')
# Iterates through all the objects, doing the pagination for you. Each obj
# is an ObjectSummary, so it doesn't contain the body. You'll need to call
# get to get the whole body.
for obj in bucket.objects.all():
key = obj.key
body = obj.get()['Body'].read()
回答by caffreyd
You might also consider the smart_openmodule, which supports iterators:
您还可以考虑smart_open支持迭代器的模块：
from smart_open import smart_open
# stream lines from an S3 object
for line in smart_open('s3://mybucket/mykey.txt', 'rb'):
print(line.decode('utf8'))
and context managers:
和上下文管理器：
with smart_open('s3://mybucket/mykey.txt', 'rb') as s3_source:
for line in s3_source:
print(line.decode('utf8'))
s3_source.seek(0) # seek to the beginning
b1000 = s3_source.read(1000) # read 1000 bytes
Find smart_openat https://pypi.org/project/smart_open/
smart_open在https://pypi.org/project/smart_open/找到
回答by Martin Thoma
When you want to read a file with a different configuration than the default one, feel free to use either mpu.aws.s3_read(s3path)directly or the copy-pasted code:
当你想读取一个与默认配置不同的文件时，可以mpu.aws.s3_read(s3path)直接使用或复制粘贴代码：
def s3_read(source, profile_name=None):
"""
Read a file from an S3 source.
Parameters
----------
source : str
Path starting with s3://, e.g. 's3://bucket-name/key/foo.bar'
profile_name : str, optional
AWS profile
Returns
-------
content : bytes
botocore.exceptions.NoCredentialsError
Botocore is not able to find your credentials. Either specify
profile_name or add the environment variables AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.
See https://boto3.readthedocs.io/en/latest/guide/configuration.html
"""
session = boto3.Session(profile_name=profile_name)
s3 = session.client('s3')
bucket_name, key = mpu.aws._s3_path_split(source)
s3_object = s3.get_object(Bucket=bucket_name, Key=key)
body = s3_object['Body']
return body.read()
回答by reubano
If you already know the filename, you can use the boto3builtin download_fileobj
如果您已经知道filename，则可以使用boto3内置download_fileobj
import boto3
from io import BytesIO
session = boto3.Session()
s3_client = session.client("s3")
f = BytesIO()
s3_client.download_fileobj(bucket_name, filename, f)
f.seek(0)
print(f.getvalue())
