Selective file download in AWS S3 CLI

问题

I have files in S3 bucket, I was trying to download files based on date like 08th aug, 09th Aug, how can I download selective date file?. I used following code but it still downloads entire bucket list

aws s3 cp s3://bucketname/ folder/file --profile pname --exclude \"*\" --recursive --include \"" + "2015-08-09" + "*\"

I am not sure, how to achieve this.

回答1:

This command will copy all files starting with 2015-08-15:

aws s3 cp s3://BUCKET/ folder --exclude "*" --include "2015-08-15*" --recursive

If your goal is to synchronize a set of files without copying them twice, use the sync command:

aws s3 sync s3://BUCKET/ folder

That will copy all files that have been added or modified since the previous sync.

In fact, this is the equivalent of the above cp command:

aws s3 sync s3://BUCKET/ folder --exclude "*" --include "2015-08-15*"

References:

• AWS CLI s3 sync command documentation
• AWS CLI s3 cp command documentation

回答2:

In case your bucket size is large in the upwards of 10 to 20 gigs, this was true in my own personal use case, you can achieve the same goal by using sync in multiple terminal windows.

All the terminal sessions can use the same token, in case you need to generate a token for prod environment.

 $ aws s3 sync s3://bucket-name/sub-name/another-name folder-name-in-pwd/
 --exclude "*" --include "name_date1*"  --profile UR_AC_SomeName

and another terminal window (same pwd)

$ aws s3 sync s3://bucket-name/sub-name/another-name folder-name-in-pwd/ 
    --exclude "*" --include "name_date2*"  --profile UR_AC_SomeName

and another two for "name_date3*" and "name_date4*"

Additionally, you can also do multiple excludes in the same sync command as in:

$ aws s3 sync s3://bucket-name/sub-name/another-name my-local-path/ 
--exclude="*.log/*" --exclude=img --exclude=".error" --exclude=tmp 
--exclude="*.cache"

来源：https://stackoverflow.com/questions/31942341/selective-file-download-in-aws-s3-cli