使用 bash、curl 访问 Azure blob 存储

Accessing Azure blob storage using bash, curl

提问by yozhik

I am attempting to use the Azure blob storage service from a bash script using the REST API. I know it is possible to accomplish this using various other tools or languages, however I'd like to do it as a bash script.

我正在尝试使用 REST API 从 bash 脚本使用 Azure blob 存储服务。我知道可以使用各种其他工具或语言来完成此操作，但是我想将其作为 bash 脚本来完成。

The script below is an attempt to list the blobs in an Azure storage container.

下面的脚本尝试列出 Azure 存储容器中的 blob。

This script results in an authentication error. The signing string and headers look correct based on the REST API (reference) documentation. I suspect the problem may be in juggling the various parts of the signing process.

此脚本会导致身份验证错误。根据 REST API（参考）文档，签名字符串和标头看起来是正确的。我怀疑问题可能在于处理签名过程的各个部分。

Has anyone successfully used bash and curl to access cloud storage resources like Azure or other providers?

有没有人成功使用 bash 和 curl 访问 Azure 或其他提供商等云存储资源？

#!/bin/bash

# List the blobs in an Azure storage container.

echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"

storage_account=""
container_name=""
access_key=""

blob_store_url="blob.core.windows.net"
authorization="SharedKey"

request_method="GET"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2011-08-18"

# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"

# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"

string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"

# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"

# Create the HMAC signature for the Authorization header
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" | sed 's/^.*= //' | base64 -w0)

authorization_header="Authorization: $authorization $storage_account:$signature"

curl \
  -H "$x_ms_date_h" \
  -H "$x_ms_version_h" \
  -H "$authorization_header" \
  "https://${storage_account}.${blob_store_url}/${container_name}?restype=container&comp=list"

Update- The storage service error and the corresponding signing string that the script generated.

更新- 存储服务错误和脚本生成的相应签名字符串。

Following is what the storage service returns for the AuthenticationFailederror.

以下是存储服务针对AuthenticationFailed错误返回的内容。

<?xml version="1.0" encoding="utf-8"?>
<Error>
  <Code>AuthenticationFailed</Code>
  <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:27e6337e-52f3-4e85-98c7-2fabaacd9ebc
Time:2013-11-21T22:10:11.7029042Z</Message>
  <AuthenticationErrorDetail>The MAC signature found in the HTTP request
'OGYxYjk1MTFkYmNkMCgzN2YzODQwNzcyNiIyYTQxZDg0OWFjNGJiZDlmNWY5YzM1ZWQzMWViMGFjYTAyZDY4NAo='
is not the same as any computed signature. Server used following string to sign:
'GET

x-ms-date:Thu, 21 Nov 2013 22:10:11 GMT
x-ms-version:2011-08-18
/storage_account_name/storage_container
comp:list
restype:container'
  </AuthenticationErrorDetail>
</Error>

Next is the string_to_signthat the script generates.

接下来是string_to_sign脚本生成的。

GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Nov 2013 22:10:11 GMT\nx-ms-version:2011-08-18\n/storage_account_name/storage_container\ncomp:list\nrestype:container

采纳答案by Zak Kristjanson

I was able to get it working. There were two things wrong with this code, the first, as Patrick Park noted, was replacing the echo -nwith printf. The second was replacing the sedmagic with the -binaryoption on openssl.

我能够让它工作。这段代码有两个问题，第一个，正如帕特里克帕克指出的那样，echo -n用printf. 第二个是用opensslsed上的-binary选项替换魔法。

Compare the original:

对比原文：

signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | sed 's/^.*= //' | base64 -w0)

with the fixed:

与固定：

signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)

The echo change is needed because echo -nwill not convert the \ninto actual newlines.

需要回声更改，因为echo -n不会将其转换\n为实际的换行符。

The -binarychange is needed because even though you are stripping off the bad part, openssl was still outputting the signature in ascii-encoded-hex, not in binary. So after it was passed to base64, the result was the b64 encoded version of the hex representation, instead of the raw value.

将-binary需要改变，因为即使你剥离坏的部分，OpenSSL的仍然在ASCII编码，十六进制输出的签名，而不是二进制。因此，在传递给 之后base64，结果是十六进制表示的 b64 编码版本，而不是原始值。

回答by Neil Mackenzie

Use Fiddler(or an equivalent on your platform) to intercept the call to Windows Azure Storage. On failure, this will show you the string that the Storage Service used to authenticate the call and you can compare this with the one you used.

使用Fiddler（或您平台上的等效程序）拦截对 Windows Azure 存储的调用。失败时，这将显示存储服务用于验证调用的字符串，您可以将其与您使用的字符串进行比较。

回答by Gaurav Mantri-AIS

Looking at the REST API documentation and your code above, I believe there's an issue with the way you're constructing canonicalized_resourcestring. You're missing the query parameters in that string. Your canonicalized_resourcestring should be:

查看上面的 REST API 文档和您的代码，我相信您构造canonicalized_resource字符串的方式存在问题。您缺少该字符串中的查询参数。你的canonicalized_resource字符串应该是：

canonicalized_resource="/${storage_account}/${container_name}\ncomp:list\nrestype:container"

回答by KiteRunner

It looks like openssl dgst does not generate proper HMAC for you. I wrote a simple program in C that does the following:

看起来 openssl dgst 没有为您生成正确的 HMAC。我用 C 编写了一个简单的程序，它执行以下操作：

1. Takes base64-encoded key from the command line and decodes it into binary.
2. Reads string to sign from standard input.
3. Uses libcrypto HMAC() routine to generate the signature.
4. base64-encodes the signature and prints the result to standard output.

1. 从命令行获取 base64 编码的密钥并将其解码为二进制。
2. 从标准输入读取要签名的字符串。
3. 使用 libcrypto HMAC() 例程生成签名。
4. 对签名进行 base64 编码并将结果打印到标准输出。

I then replaced openssl dgst pipeline in your script with the call to my program and it did the trick.

然后我用对我的程序的调用替换了脚本中的 openssl dgst 管道，它就成功了。

Please note that the output you are getting from Azure is XML-wrapped and base-64 encoded, so you'll need to come up with some sort of parsing/conversion code for it.

请注意，您从 Azure 获得的输出是 XML 包装和 base-64 编码的，因此您需要为它提供某种解析/转换代码。

回答by Patrick Park

use printf instead of echo (it works for me)

使用 printf 而不是 echo （它对我有用）

for example:

例如：

SIGNATURE=printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt hexkey:$HEXKEY -binary | base64 -w0

签名=printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt hexkey:$HEXKEY -binary | base64 -w0