Downloading an entire S3 bucket?

Question

I noticed that there doesn\'t seem to be an option to download an entire S3 bucket from the AWS Management Console.

Is there an easy way to grab everything in one of

Answer 1

You've many options to do that, but the best one is using the AWS CLI.

Here's a walk-through:

1. Download and install AWS CLI in your machine:

   • Install the AWS CLI using the MSI Installer (Windows).
   • Install the AWS CLI using the Bundled Installer for Linux, OS X, or Unix.

2. Configure AWS CLI:

   Make sure you input valid access and secret keys, which you received when you created the account.

3. Sync the S3 bucket using:

   aws s3 sync s3://yourbucket /local/path
   

   In the above command, replace the following fields:

   • yourbucket >> your S3 bucket that you want to download.
   • /local/path >> path in your local system where you want to download all the files.

Answer 2

Try this command:

aws s3 sync yourBucketnameDirectory yourLocalDirectory

For example, if your bucket name is myBucket and local directory is c:\local, then:

aws s3 sync s3://myBucket c:\local

For more informations about awscli check this aws cli installation

Answer 3

aws sync is the perfect solution. It does not do a two way.. it is a one way from source to destination. Also, if you have lots of items in bucket it will be a good idea to create s3 endpoint first so that download happens faster (because download does not happen via internet but via intranet) and no charges

Answer 4

You may simple get it with s3cmd command:

s3cmd get --recursive --continue s3://test-bucket local-directory/

Answer 5

You can use this AWS cli command to download entire S3 bucket content to local folder

aws s3 sync s3://your-bucket-name "Local Folder Path"

If you see error like this

fatal error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

--no-verify-ssl (boolean)

By default, the AWS CLI uses SSL when communicating with AWS services. For each SSL connection, the AWS CLI will verify SSL certificates. This option overrides the default behavior of verifying SSL certificates. reference

Use this tag with command --no-verify-ssl

aws s3 sync s3://your-bucket-name "Local Folder Path" --no-verify-ssl

Answer 6

I've done a bit of development for S3 and I have not found a simple way to download a whole bucket.

If you want to code in Java the jets3t lib is easy to use to create a list of buckets and iterate over that list to download them.

First, get a public private key set from the AWS management consule so you can create an S3service object:

AWSCredentials awsCredentials = new AWSCredentials(YourAccessKey, YourAwsSecretKey);
s3Service = new RestS3Service(awsCredentials);

Then, get an array of your buckets objects:

S3Object[] objects = s3Service.listObjects(YourBucketNameString);

Finally, iterate over that array to download the objects one at a time with:

S3Object obj = s3Service.getObject(bucket, fileName);
            file = obj.getDataInputStream();

I put the connection code in a threadsafe singleton. The necessary try/catch syntax has been omitted for obvious reasons.

If you'd rather code in Python you could use Boto instead.

After looking around BucketExplorer, "Downloading the whole bucket" may do what you want.