使用 bash 从 S3 下载私有文件
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/27658147/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Download private file from S3 using bash
提问by UKatz
I am trying to get the following bash script to work (copied from http://curl.haxx.se/mail/archive-2014-10/0006.html#replies):
我正在尝试使用以下 bash 脚本（从http://curl.haxx.se/mail/archive-2014-10/0006.html#replies复制）：
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET
${contentType}
${dateValue}
${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
I am getting a SignatureDoesNotMatch error no matter what I do.
无论我做什么，我都会收到 SignatureDoesNotMatch 错误。
Any ideas on how to fix this will be greatly appreciated.
任何有关如何解决此问题的想法将不胜感激。
采纳答案by UKatz
After way too much time spent on this I finally got it to work:
在花了太多时间之后，我终于让它工作了：
This line:
这一行：
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
is missing an 'e':
缺少一个“e”：
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
In other words, characters weren't being escaped before the string was signed.
换句话说，在对字符串进行签名之前，字符不会被转义。
As an aside, I also learned that for get requests, the content type is meaningless.
顺便说一句，我还了解到对于 get 请求，内容类型毫无意义。
回答by jpillora
Using various answers in this thread, I converted it into a handy s3getbash function:
使用此线程中的各种答案，我将其转换为一个方便的s3getbash 函数：
#!/bin/bash
#usage - s3get writes the specified object to stdout
# s3get <bucket/key> [region]
#set these in your environment/profile (NOT HERE)
AWS_ACCESS_KEY=""
AWS_SECRET_KEY=""
#example usage
s3get my-bucket/a/path/to/my/file > /tmp/file
function s3get {
#helper functions
function fail { echo "" > /dev/stderr; exit 1; }
#dependency check
if ! hash openssl 2>/dev/null; then fail "openssl not installed"; fi
if ! hash curl 2>/dev/null; then fail "curl not installed"; fi
#params
path=""
bucket=$(cut -d '/' -f 1 <<< "$path")
key=$(cut -d '/' -f 2- <<< "$path")
region="${2:-us-west-1}"
#load creds
access="$AWS_ACCESS_KEY"
secret="$AWS_SECRET_KEY"
#validate
if [[ "$bucket" = "" ]]; then fail "missing bucket (arg 1)"; fi;
if [[ "$key" = "" ]]; then fail "missing key (arg 1)"; fi;
if [[ "$region" = "" ]]; then fail "missing region (arg 2)"; fi;
if [[ "$access" = "" ]]; then fail "missing AWS_ACCESS_KEY (env var)"; fi;
if [[ "$secret" = "" ]]; then fail "missing AWS_SECRET_KEY (env var)"; fi;
#compute signature
contentType="text/html; charset=UTF-8"
date="`date -u +'%a, %d %b %Y %H:%M:%S GMT'`"
resource="/${bucket}/${key}"
string="GET\n\n${contentType}\n\nx-amz-date:${date}\n${resource}"
signature=`echo -en $string | openssl sha1 -hmac "${secret}" -binary | base64`
#get!
curl -H "x-amz-date: ${date}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${access}:${signature}" \
"https://s3-${region}.amazonaws.com${resource}"
}
Tested on OSX and Ubuntu. Saved in this Github gist.
在 OSX 和 Ubuntu 上测试。保存在这个Github gist 中。
回答by mvaneijk
The TS asked for a working SHA-1 version of the script. However, SHA-1 is outdated and Amazon has datacenters that only accept SHA-256 encryption, hereby the download script that can be used for all S3 datacenters: It also follows HTTP 307 redirects.
TS 要求提供脚本的有效 SHA-1 版本。然而，SHA-1 已经过时，亚马逊的数据中心只接受 SHA-256 加密，特此下载脚本可用于所有 S3 数据中心：它还遵循 HTTP 307 重定向。
#!/bin/sh
#USAGE:
# download-aws.sh <bucket> <region> <source-file> <dest-file>
set -e
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
file=
bucket=
host="${bucket}.s3.amazonaws.com"
resource="/${file}"
contentType="text/plain"
dateValue="`date +'%Y%m%d'`"
X_amz_date="`date +'%Y%m%dT%H%M%SZ'`"
X_amz_algorithm="AWS4-HMAC-SHA256"
awsRegion=
awsService="s3"
X_amz_credential="$s3Key%2F$dateValue%2F$awsRegion%2F$awsService%2Faws4_request"
X_amz_credential_auth="$s3Key/$dateValue/$awsRegion/$awsService/aws4_request"
signedHeaders="host;x-amz-algorithm;x-amz-content-sha256;x-amz-credential;x-amz-date"
contentHash="e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
HMAC_SHA256_asckey () {
var=`/bin/echo -en | openssl sha256 -hmac -binary | xxd -p -c256`
echo $var
}
HMAC_SHA256 () {
var=`/bin/echo -en | openssl dgst -sha256 -mac HMAC -macopt hexkey: -binary | xxd -p -c256`
echo $var
}
REQUEST () {
canonicalRequest="GET\n$resource\n\n"\
"host:\n"\
"x-amz-algorithm:$X_amz_algorithm""\n"\
"x-amz-content-sha256:$contentHash""\n"\
"x-amz-credential:$X_amz_credential""\n"\
"x-amz-date:$X_amz_date""\n\n"\
"$signedHeaders\n"\
"$contentHash"
#echo $canonicalRequest
canonicalHash=`/bin/echo -en "$canonicalRequest" | openssl sha256 -binary | xxd -p -c256`
stringToSign="$X_amz_algorithm\n$X_amz_date\n$dateValue/$awsRegion/s3/aws4_request\n$canonicalHash"
#echo $stringToSign
s1=`HMAC_SHA256_asckey "AWS4""$s3Secret" $dateValue`
s2=`HMAC_SHA256 "$s1" "$awsRegion"`
s3=`HMAC_SHA256 "$s2" "$awsService"`
signingKey=`HMAC_SHA256 "$s3" "aws4_request"`
signature=`/bin/echo -en $stringToSign | openssl dgst -sha256 -mac HMAC -macopt hexkey:$signingKey -binary | xxd -p -c256`
#echo signature
authorization="$X_amz_algorithm Credential=$X_amz_credential_auth,SignedHeaders=$signedHeaders,Signature=$signature"
result=$(curl --silent -H "Host: " -H "X-Amz-Algorithm: $X_amz_algorithm" -H "X-Amz-Content-Sha256: $contentHash" -H "X-Amz-Credential: $X_amz_credential" -H "X-Amz-Date: $X_amz_date" -H "Authorization: $authorization" https:///${file} -o "" --write-out "%{http_code}")
if [ $result -eq 307 ]; then
redirecthost=`cat | sed -n 's:.*<Endpoint>\(.*\)</Endpoint>.*::p'`
REQUEST "$redirecthost" ""
fi
}
REQUEST "$host" ""
Tested on Ubuntu
在 Ubuntu 上测试
If someone knows a solution to remove the HMAC-ASCII step, you're welcome to reply. I got this only working in this way.
如果有人知道删除HMAC-ASCII步骤的解决方案，欢迎您回复。我只能以这种方式工作。
回答by Shubham Bhartiya
bucket=your-bucket-name
contentType="text/plain"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxx
s3Secret=xxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
file1=file-name
resource1="/${bucket}/${file1}"
curl -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" "https://s3-us-west-2.amazonaws.com/${resource1}" -o "file-name-to-save-the-output"
I was getting errors in the actual answer. This works for me. This will get the file as what it is and not as string.
我在实际答案中遇到错误。这对我有用。这将使文件原样而不是字符串。
回答by Dave Side
It required minor adjustment, but the following lines works well
它需要稍作调整，但以下几行效果很好
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" https://${bucket}.s3.amazonaws.com/${file}
