Downloading an entire S3 bucket?

Question

I noticed that there doesn\'t seem to be an option to download an entire S3 bucket from the AWS Management Console.

Is there an easy way to grab everything in one of

Answer 1

AWS CLI

See the "AWS CLI Command Reference" for more information.

AWS recently released their Command Line Tools, which work much like boto and can be installed using

sudo easy_install awscli

or

sudo pip install awscli

Once installed, you can then simply run:

aws s3 sync s3://<source_bucket> <local_destination>

For example:

aws s3 sync s3://mybucket .

will download all the objects in mybucket to the current directory.

And will output:

download: s3://mybucket/test.txt to test.txt
download: s3://mybucket/test2.txt to test2.txt

This will download all of your files using a one-way sync. It will not delete any existing files in your current directory unless you specify --delete, and it won't change or delete any files on S3.

You can also do S3 bucket to S3 bucket, or local to S3 bucket sync.

Check out the documentation and other examples.

Whereas the above example is how to download a full bucket, you can also download a folder recursively by performing

aws s3 cp s3://BUCKETNAME/PATH/TO/FOLDER LocalFolderName --recursive

This will instruct the CLI to download all files and folder keys recursively within the PATH/TO/FOLDER directory within the BUCKETNAME bucket.

Answer 2

You can use s3cmd to download your bucket:

s3cmd --configure
s3cmd sync s3://bucketnamehere/folder /destination/folder

---

There is another tool you can use called rclone. This is a code sample in the Rclone documentation:

rclone sync /home/local/directory remote:bucket

Answer 3

You can do this with https://github.com/minio/mc :

mc cp -r https://s3-us-west-2.amazonaws.com/bucketName/ localdir

mc also supports sessions, resumable downloads, uploads and many more. mc supports Linux, OS X and Windows operating systems. Written in Golang and released under Apache Version 2.0.

Answer 4

If you use Firefox with S3Fox, that DOES let you select all files (shift-select first and last) and rightclick and download all... I've done it with 500+ files w/o problem

Answer 5

When in Windows, my preferred GUI tool for this is Cloudberry Explorer for S3., http://www.cloudberrylab.com/free-amazon-s3-explorer-cloudfront-IAM.aspx. Has a fairly polished file explorer, ftp-like interface.

Answer 6

To add another GUI option, we use WinSCP's S3 functionality. It's very easy to connect, only requiring your access key and secret key in the UI. You can then browse and download whatever files you require from any accessible buckets, including recursive downloads of nested folders.

Since it can be a challenge to clear new software through security and WinSCP is fairly prevalent, it can be really beneficial to just use it rather than try to install a more specialized utility.