Downloading the latest file in an S3 bucket using AWS CLI?

问题

I have an S3 bucket that contains database backups. I am creating a script that I would like to download the latest backup (and eventually restore it somewhere else), but I'm not sure how to go about only grabbing the most recent file from a bucket.

Is it possible to copy only the most recent file from a s3 bucket to a local directory using AWS CLI tools?

回答1:

This is a approach you can take.

You can list all the objects in the bucket with aws s3 ls $BUCKET --recursive:

$ aws s3 ls $BUCKET --recursive
2015-05-05 15:36:17          4 an_object.txt
2015-06-08 14:14:44   16322599 some/other/object
2015-04-29 12:09:29      32768 yet-another-object.sh

They're sorted alphabetically by key, but that first column is the last modified time. A quick sort will reorder them by date:

$ aws s3 ls $BUCKET --recursive | sort
2015-04-29 12:09:29      32768 yet-another-object.sh
2015-05-05 15:36:17          4 an_object.txt
2015-06-08 14:14:44   16322599 some/other/object

tail -n 1 selects the last row, and awk '{print $4}' extracts the fourth column (the name of the object).

$ aws s3 ls $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}'
some/other/object

Last but not least, drop that into aws s3 cp to download the object:

$ KEY=`aws s3 ls $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}'`
$ aws s3 cp s3://$BUCKET/$KEY ./latest-object

回答2:

And here is a bash script create based on @error2007s's answer. This script requires your aws profile and bucket name as variables, and downloads the latest object to your ~/Downloads folder:

#!/bin/sh
PROFILE=your_profile
BUCKET=your_bucket

OBJECT="$(aws s3 ls --profile $PROFILE $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}')"
aws s3 cp s3://$BUCKET/$OBJECT ~/Downloads/$OBJECT --profile $PROFILE

回答3:

The Above solutions are Bash for if one want to do the same thing in Powershell for downloading in windows using the follwing script :

$s3location = 's3://bucket/'
$filename=C:\Progra~1\Amazon\AWSCLI\aws.exe s3 ls s3://bucket/PROD_FULL/ -- 
recursive | sort |select -last 3 
$Dpath='I:\Data_S3'

foreach($files in $filename)
{
#$files.ToString()
$testpath1 = Split-Path $path -leaf
$testpath1
$path=$s3location+$files

C:\Progra~1\Amazon\AWSCLI\aws.exe s3 cp $path $Dpath
echo(" ***Files Downloaded ***")

}

来源：https://stackoverflow.com/questions/38384879/downloading-the-latest-file-in-an-s3-bucket-using-aws-cli

标签

amazon-web-services

amazon-s3

aws-cli