How to restore folders (or entire buckets) to Amazon S3 from Glacier?

Question

I changed the lifecycle for a bunch of my buckets on Amazon S3 so their storage class was set to Glacier. I did this using the online AWS Console. I now need those files a

Answer 1

A variation on Dustin's answer to use AWS CLI, but to use recursion and pipe to sh to skip errors (like if some objects have already requested restore...)

BUCKET=my-bucket
BPATH=/path/in/bucket
DAYS=1
aws s3 ls s3://$BUCKET$BPATH --recursive | awk '{print $4}' | xargs -L 1 \
 echo aws s3api restore-object --restore-request Days=$DAYS \
 --bucket $BUCKET --key | sh

The xargs echo bit generates a list of "aws s3api restore-object" commands and by piping that to sh, you can continue on error.

NOTE: Ubuntu 14.04 aws-cli package is old. In order to use --recursive you'll need to install via github.

POSTSCRIPT: Glacier restores can get unexpectedly pricey really quickly. Depending on your use case, you may find the Infrequent Access tier to be more appropriate. AWS have a nice explanation of the different tiers.

Answer 2

I've been through this mill today and came up with the following based on the answers above and having also tried s3cmd. s3cmd doesn't work for mixed buckets (Glacier and Standard). This will do what you need in two steps - first create a glacier file list and then ping the s3 cli requests off (even if they have already occurred). It will also keep a track of which have been requested already so you can restart the script as necessary. Watch out for the TAB (\t) in the cut command quoted below:

#/bin/sh

bucket="$1"
glacier_file_list="glacier-restore-me-please.txt"
glacier_file_done="glacier-requested-restore-already.txt"

if [ "X${bucket}" = "X" ]
then
  echo "Please supply bucket name as first argument"
  exit 1
fi

aws s3api list-objects-v2 --bucket ${bucket} --query "Contents[?StorageClass=='GLACIER']" --output text |cut -d '\t' -f 2 > ${glacier_file_list}

if $? -ne 0
then
  echo "Failed to fetch list of objects from bucket ${bucket}"
  exit 1
fi

echo "Got list of glacier files from bucket ${bucket}"

while read x
do
  echo "Begin restoring $x"
  aws s3api restore-object --restore-request Days=7 --bucket ${bucket} --key "$x"

  if [ $? -ne 0 ]
  then
    echo "Failed to restore \"$x\""
  else
    echo "Done requested restore of \"$x\""
  fi

  # Log those done
  #
  echo "$x" >> ${glacier_file_done}

done < ${glacier_file_list}

Answer 3

Here is my version of the aws cli interface and how to restore data from glacier. I modified some of the above examples to work when the key of the files to be restored contain spaces.

# Parameters
BUCKET="my-bucket" # the bucket you want to restore, no s3:// no slashes
BPATH="path/in/bucket/" # the objects prefix you wish to restore (mind the `/`) 
DAYS=1 # For how many days you wish to restore the data.

# Restore the objects
aws s3 ls s3://{BUCKET}/${BPATH} --recursive | \
awk '{out=""; for(i=4;i<=NF;i++){out=out" "$i}; print out}'| \
xargs -I {} aws s3api restore-object --restore-request Days={DAYS} \
--bucket {BUCKET} --key "{}"

Answer 4

If you're using the AWS CLI tool (it's nice, you should), you can do it like this:

aws s3 ls s3://<bucket_name> | awk '{print $4}' | xargs -L 1 aws s3api restore-object --restore-request Days=<days> --bucket <bucket_name> --key

Replace <bucket_name> with the bucket name you want.

Replace <days> with the number of days you want to restore the object for.