Download private file from S3 using bash

Question

I am trying to get the following bash script to work (copied from http://curl.haxx.se/mail/archive-2014-10/0006.html#replies):

#!/bin/sh 
file=path/to/file          


        

Answer 1

The TS asked for a working SHA-1 version of the script. However, SHA-1 is outdated and Amazon has datacenters that only accept SHA-256 encryption, hereby the download script that can be used for all S3 datacenters: It also follows HTTP 307 redirects.

#!/bin/sh

#USAGE:
# download-aws.sh <bucket> <region> <source-file> <dest-file>

set -e

s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

file=$3
bucket=$1
host="${bucket}.s3.amazonaws.com"
resource="/${file}"
contentType="text/plain"
dateValue="`date +'%Y%m%d'`"
X_amz_date="`date +'%Y%m%dT%H%M%SZ'`"
X_amz_algorithm="AWS4-HMAC-SHA256"
awsRegion=$2
awsService="s3"
X_amz_credential="$s3Key%2F$dateValue%2F$awsRegion%2F$awsService%2Faws4_request"
X_amz_credential_auth="$s3Key/$dateValue/$awsRegion/$awsService/aws4_request"

signedHeaders="host;x-amz-algorithm;x-amz-content-sha256;x-amz-credential;x-amz-date"
contentHash="e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

HMAC_SHA256_asckey () {
        var=`/bin/echo -en $2 | openssl sha256 -hmac $1 -binary | xxd -p -c256`
        echo $var
}
HMAC_SHA256 () {
        var=`/bin/echo -en $2 | openssl dgst -sha256 -mac HMAC -macopt hexkey:$1 -binary | xxd -p -c256`
        echo $var
}
REQUEST () {
        canonicalRequest="GET\n$resource\n\n"\
"host:$1\n"\
"x-amz-algorithm:$X_amz_algorithm""\n"\
"x-amz-content-sha256:$contentHash""\n"\
"x-amz-credential:$X_amz_credential""\n"\
"x-amz-date:$X_amz_date""\n\n"\
"$signedHeaders\n"\
"$contentHash"
        #echo $canonicalRequest
        canonicalHash=`/bin/echo -en "$canonicalRequest" | openssl sha256 -binary | xxd -p -c256`
        stringToSign="$X_amz_algorithm\n$X_amz_date\n$dateValue/$awsRegion/s3/aws4_request\n$canonicalHash"
        #echo $stringToSign


        s1=`HMAC_SHA256_asckey "AWS4""$s3Secret" $dateValue`
        s2=`HMAC_SHA256 "$s1" "$awsRegion"`
        s3=`HMAC_SHA256 "$s2" "$awsService"`
        signingKey=`HMAC_SHA256 "$s3" "aws4_request"`
        signature=`/bin/echo -en $stringToSign | openssl dgst -sha256 -mac HMAC -macopt hexkey:$signingKey -binary | xxd -p -c256`
        #echo signature

        authorization="$X_amz_algorithm Credential=$X_amz_credential_auth,SignedHeaders=$signedHeaders,Signature=$signature"
        result=$(curl --silent -H "Host: $1" -H "X-Amz-Algorithm: $X_amz_algorithm" -H "X-Amz-Content-Sha256: $contentHash" -H "X-Amz-Credential: $X_amz_credential" -H "X-Amz-Date: $X_amz_date" -H "Authorization: $authorization" https://${1}/${file} -o "$2" --write-out "%{http_code}")
        if [ $result -eq 307 ]; then
                redirecthost=`cat $2 | sed -n 's:.*<Endpoint>\(.*\)</Endpoint>.*:\1:p'`
                REQUEST "$redirecthost" "$2"
        fi
}
REQUEST "$host" "$4"

Tested on Ubuntu

If someone knows a solution to remove the HMAC-ASCII step, you're welcome to reply. I got this only working in this way.

Answer 2

It required minor adjustment, but the following lines works well

#!/bin/sh 
file=path/to/file 
bucket=your-bucket 
resource="/${bucket}/${file}" 
contentType="application/x-compressed-tar" 
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`" 
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}" 
s3Key=xxxxxxxxxxxxxxxxxxxx 
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64` 
curl -H "Host: ${bucket}.s3.amazonaws.com" -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" https://${bucket}.s3.amazonaws.com/${file}

Answer 3

Using various answers in this thread, I converted it into a handy s3get bash function:

#!/bin/bash

#usage - s3get writes the specified object to stdout
#  s3get <bucket/key> [region]

#set these in your environment/profile (NOT HERE)
AWS_ACCESS_KEY="" 
AWS_SECRET_KEY=""

#example usage
s3get my-bucket/a/path/to/my/file > /tmp/file

function s3get {
    #helper functions
    function fail { echo "$1" > /dev/stderr; exit 1; }
    #dependency check
    if ! hash openssl 2>/dev/null; then fail "openssl not installed"; fi
    if ! hash curl 2>/dev/null; then fail "curl not installed"; fi
    #params
    path="${1}"
    bucket=$(cut -d '/' -f 1 <<< "$path")
    key=$(cut -d '/' -f 2- <<< "$path")
    region="${2:-us-west-1}"
    #load creds
    access="$AWS_ACCESS_KEY"
    secret="$AWS_SECRET_KEY"
    #validate
    if [[ "$bucket" = "" ]]; then fail "missing bucket (arg 1)"; fi;
    if [[ "$key" = ""    ]]; then fail "missing key (arg 1)"; fi;
    if [[ "$region" = "" ]]; then fail "missing region (arg 2)"; fi;
    if [[ "$access" = "" ]]; then fail "missing AWS_ACCESS_KEY (env var)"; fi;
    if [[ "$secret" = "" ]]; then fail "missing AWS_SECRET_KEY (env var)"; fi;
    #compute signature
    contentType="text/html; charset=UTF-8" 
    date="`date -u +'%a, %d %b %Y %H:%M:%S GMT'`"
    resource="/${bucket}/${key}"
    string="GET\n\n${contentType}\n\nx-amz-date:${date}\n${resource}"
    signature=`echo -en $string | openssl sha1 -hmac "${secret}" -binary | base64` 
    #get!
    curl -H "x-amz-date: ${date}" \
        -H "Content-Type: ${contentType}" \
        -H "Authorization: AWS ${access}:${signature}" \
        "https://s3-${region}.amazonaws.com${resource}"
}

Tested on OSX and Ubuntu. Saved in this Github gist.

Answer 4

bucket=your-bucket-name
contentType="text/plain" 
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`" 
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxx 
s3Secret=xxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64` 

file1=file-name
resource1="/${bucket}/${file1}" 
curl -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" "https://s3-us-west-2.amazonaws.com/${resource1}" -o "file-name-to-save-the-output"

I was getting errors in the actual answer. This works for me. This will get the file as what it is and not as string.

Answer 5

After way too much time spent on this I finally got it to work:

This line:

signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`

is missing an 'e':

signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`

In other words, characters weren't being escaped before the string was signed.

As an aside, I also learned that for get requests, the content type is meaningless.