bash 从 Amazon S3 存储桶下载文件的脚本
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/30876123/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
script to download file from Amazon S3 bucket
提问by DuffManLight
Trying to write script to download file from Amazon S3 bucket.
尝试编写脚本以从 Amazon S3 存储桶下载文件。
Having trouble with the example on the cURL site. The script below produces:
cURL 站点上的示例有问题。下面的脚本产生：
The request signature we calculated does not match the signature you provided. Check your key and signing method.
我们计算的请求签名与您提供的签名不匹配。检查您的密钥和签名方法。
Appreciate any help.
感谢任何帮助。
#!/bin/sh
file="filename.php"
bucket="my-bucket"
resource="/${bucket}/${file}"
contentType="text/html"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key='ABCABCABCABCABCABCAB'
s3Secret='xyzxyzyxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzx'
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} - binary | base64`
curl -v -H "Host:lssngen-updates-east.s3.amazonaws.com" \
-H "Date:${dateValue}" \
-H "Content-Type:${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
回答by Julio Faerman
Avoid signing the request yourself, a lot can go wrong or be hard to do. For example, you should check that the date is set to GMT or use x-amz-date headers.
避免自己签署请求，很多事情可能会出错或很难做到。例如，您应该检查日期是否设置为 GMT 或使用 x-amz-date 标头。
Another approach is to use the AWS Command Line Interfaceand so use $ aws s3 cpor $ aws s3 sync.
另一种方法是使用AWS 命令行界面，因此使用 $ aws s3 cp或$ aws s3 sync。
回答by ytdm
I write this bash script to download file from s3 (I download compressed file, you can change contentType to download other types of file)
我写这个 bash 脚本来从 s3 下载文件（我下载的是压缩文件，你可以更改 contentType 来下载其他类型的文件）
#!/bin/sh
outputFile="Your_PATH"
amzFile="AMAZON_FILE_PATH"
bucket="YOUR_BUCKET"
resource="/${bucket}/${amzFile}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="YOUR_S3_KEY"
s3Secret="YOUR_S3SECRET"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${amzFile} -o $outputFile
回答by TheStoryCoder
As of August 2019 I found this to work. Has added region, and format of URL has changed.
截至 2019 年 8 月，我发现此方法有效。增加了区域，URL 的格式也发生了变化。
#!/bin/sh
outputFile="/PATH/TO/LOCALLY/SAVED/FILE"
amzFile="BUCKETPATH/TO/FILE"
region="YOUR-REGION"
bucket="SOME-BUCKET"
resource="/${bucket}/${amzFile}"
contentType="binary/octet-stream"
dateValue=`TZ=GMT date -R`
# You can leave our "TZ=GMT" if your system is already GMT (but don't have to)
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: s3-${region}.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://s3-${region}.amazonaws.com/${bucket}/${amzFile} -o $outputFile
回答by Kunal
#!/bin/sh
# This works for cross region
outputFile="/PATH/TO/FILE"
awsFile="BUCKETPATH/TO/FILE"
bucket="SOME-BUCKET"
resource="/${bucket}/${awsFile}"
contentType="application/x-compressed-tar"
# Change the content type as desired
dateValue=`TZ=GMT date -R`
#Use dateValue=`date -R` if your TZ is already GMT
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -n ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${awsFile} -o $outputFile
回答by Adrian Bartyczak
Here's a fully outlined easy-to-follow AWS signature creator and GET request tester. You can find the original file here.
这是一个完整概述且易于遵循的 AWS 签名创建者和 GET 请求测试器。您可以在此处找到原始文件。
readonly AWS_ACCESS_KEY_ID='<your_access_key_id>'
readonly AWS_SECRET_ACCESS_KEY='<your_secret_access_key>'
readonly AWS_SERVICE='s3'
readonly AWS_REGION='us-east-1'
readonly AWS_S3_BUCKET_NAME='<your_bucket_name>'
readonly AWS_SERVICE_ENDPOINT_URL="\
${AWS_S3_BUCKET_NAME}.${AWS_SERVICE}.amazonaws.com"
# Create an SHA-256 hash in hexadecimal.
# Usage:
# hash_sha256 <string>
function hash_sha256 {
printf "" | openssl dgst -sha256 | sed 's/^.* //'
}
# Create an SHA-256 hmac in hexadecimal format.
# Usage:
# hmac_sha256 <key> <data>
function hmac_sha256 {
key=""
data=""
printf "${data}" | openssl dgst -sha256 -mac HMAC -macopt "${key}" | \
sed 's/^.* //'
}
readonly CURRENT_DATE_DAY="$(date -u '+%Y%m%d')"
readonly CURRENT_DATE_TIME="$(date -u '+%H%M%S')"
readonly CURRENT_DATE_ISO8601="${CURRENT_DATE_DAY}T${CURRENT_DATE_TIME}Z"
readonly HTTP_REQUEST_METHOD='GET'
readonly HTTP_REQUEST_PAYLOAD=''
readonly HTTP_REQUEST_PAYLOAD_HASH="$(printf "${HTTP_REQUEST_PAYLOAD}" | \
openssl dgst -sha256 | sed 's/^.* //')"
readonly HTTP_CANONICAL_REQUEST_URI='/video_clips/0940.m3u8'
readonly HTTP_CANONICAL_REQUEST_QUERY_STRING=''
readonly HTTP_REQUEST_CONTENT_TYPE='application/x-www-form-urlencoded'
readonly HTTP_CANONICAL_REQUEST_HEADERS="\
content-type:${HTTP_REQUEST_CONTENT_TYPE}
host:${AWS_SERVICE_ENDPOINT_URL}
x-amz-content-sha256:${HTTP_REQUEST_PAYLOAD_HASH}
x-amz-date:${CURRENT_DATE_ISO8601}"
# Note: The signed headers must match the canonical request headers.
readonly HTTP_REQUEST_SIGNED_HEADERS="\
content-type;host;x-amz-content-sha256;x-amz-date"
readonly HTTP_CANONICAL_REQUEST="\
${HTTP_REQUEST_METHOD}
${HTTP_CANONICAL_REQUEST_URI}
${HTTP_CANONICAL_REQUEST_QUERY_STRING}
${HTTP_CANONICAL_REQUEST_HEADERS}\n
${HTTP_REQUEST_SIGNED_HEADERS}
${HTTP_REQUEST_PAYLOAD_HASH}"
# Create the signature.
# Usage:
# create_signature
function create_signature {
stringToSign="AWS4-HMAC-SHA256
${CURRENT_DATE_ISO8601}
${CURRENT_DATE_DAY}/${AWS_REGION}/${AWS_SERVICE}/aws4_request
$(hash_sha256 "${HTTP_CANONICAL_REQUEST}")"
dateKey=$(hmac_sha256 key:"AWS4${AWS_SECRET_ACCESS_KEY}" \
"${CURRENT_DATE_DAY}")
regionKey=$(hmac_sha256 hexkey:"${dateKey}" "${AWS_REGION}")
serviceKey=$(hmac_sha256 hexkey:"${regionKey}" "${AWS_SERVICE}")
signingKey=$(hmac_sha256 hexkey:"${serviceKey}" "aws4_request")
printf "${stringToSign}" | openssl dgst -sha256 -mac HMAC -macopt \
hexkey:"${signingKey}" | awk '{print }'
}
readonly SIGNATURE="$(create_signature)"
readonly HTTP_REQUEST_AUTHORIZATION_HEADER="\
AWS4-HMAC-SHA256 Credential=${AWS_ACCESS_KEY_ID}/${CURRENT_DATE_DAY}/\
${AWS_REGION}/${AWS_SERVICE}/aws4_request, \
SignedHeaders=${HTTP_REQUEST_SIGNED_HEADERS};x-amz-date, Signature=${SIGNATURE}"
curl -X "${HTTP_REQUEST_METHOD}" -v \
"https://${AWS_SERVICE_ENDPOINT_URL}${HTTP_CANONICAL_REQUEST_URI}" \
-H "Authorization: ${HTTP_REQUEST_AUTHORIZATION_HEADER}" \
-H "content-type: ${HTTP_REQUEST_CONTENT_TYPE}" \
-H "x-amz-content-sha256: ${HTTP_REQUEST_PAYLOAD_HASH}" \
-H "x-amz-date: ${CURRENT_DATE_ISO8601}"
