looking for s3cmd download command for a certain date

问题

I am trying to figure out on what the s3cmd command would be to download files from bucket by date, so for example i have a bucket named "test" and in that bucket there are different files from different dates. I am trying to get the files that were uploaded yesterday. what would the command be?

回答1:

There is no single command that will allow you to do that. You have to write a script some thing like this. Or use a SDK that allows you to do this. Below script is a sample script that will get S3 files from last 30 days.

#!/bin/bash
# Usage: ./getOld "bucketname" "30 days"
s3cmd ls s3://$1 | while read -r line;  do

createDate=`echo $line|awk {'print $1" "$2'}`
createDate=`date -d"$createDate" +%s`
olderThan=`date -d"-$2" +%s`
if [[ $createDate -lt $olderThan ]]
  then 
    fileName=`echo $line|awk {'print $4'}`
    echo $fileName
    if [[ $fileName != "" ]]
      then
        s3cmd get "$fileName"
    fi
fi
done;

回答2:

I like s3cmd but to work with single line command, I prefer the JSon output of aws cli and jq JSon processor

The command will look like

aws s3api list-objects --bucket "yourbucket" |\
jq '.Contents[] | select(.LastModified | startswith("yourdate")).Key' --raw-output |\
xargs -I {} aws s3 cp s3://yourbucket/{} .

basically what the script does

list all object from a given bucket
(the interesting part) jq will parse the Contents array and select element where the LastModified value start with your pattern (you will need to change), get the Key of the s3 object and add --raw-output so it strips the quote from the value
pass the result to an aws copy command to download the file from s3

if you want to automate a bit further you can get yesterday from the command line

for mac os

$ export YESTERDAY=`date -v-1w +%F`
$ aws s3api list-objects --bucket "ariba-install" |\
jq '.Contents[] | select(.LastModified | startswith('\"$YESTERDAY\"')).Key' --raw-output |\
xargs -I {} aws s3 cp s3://ariba-install/{} .

for linux os (or other flavor of bash that I am not familiar)

$ export YESTERDAY=`date -d "1 day ago" '+%Y-%m-%d' `
$ aws s3api list-objects --bucket "ariba-install" |\
jq '.Contents[] | select(.LastModified | startswith('\"$YESTERDAY\"')).Key' --raw-output |\
xargs -I {} aws s3 cp s3://ariba-install/{} .

Now you get the idea if you want to change the YESTERDAY variable to have different kind of date

来源：https://stackoverflow.com/questions/39479862/looking-for-s3cmd-download-command-for-a-certain-date

标签

amazon-web-services

amazon-s3

aws-cli

s3cmd