Downloading an entire S3 bucket?

Question

I noticed that there doesn\'t seem to be an option to download an entire S3 bucket from the AWS Management Console.

Is there an easy way to grab everything in one of

Answer 1

To download using AWS S3 CLI:

aws s3 cp s3://WholeBucket LocalFolder --recursive
aws s3 cp s3://Bucket/Folder LocalFolder --recursive

To download using code, use the AWS SDK.

To download using GUI, use Cyberduck.

Answer 2

If you only want to download the bucket from AWS, first install the AWS CLI in your machine. In terminal change the directory to where you want to download the files and run this command.

aws s3 sync s3://bucket-name .

If you also want to sync the both local and s3 directories (in case you added some files in local folder), run this command:

aws s3 sync . s3://bucket-name

Answer 3

If the bucket is quite big there is a command called s4cmd which makes parallel connections and improves the download time:

To install it on Debian like

apt install s4cmd

If you have pip:

pip install s4cmd

It will read the ~/.s3cfg file if present (if not install s3cmd and run s3cmd --configure) or you can specify --access-key=ACCESS_KEY --secret-key=SECRET_KEY on the command.

The cli is similar to s3cmd. In your case a sync is recommended as you can cancel the download and start it again without having to re-downloaded the files.

s4cmd [--access-key=ACCESS_KEY --secret-key=SECRET_KEY] sync s3://<your-bucket> /some/local/dir

Be careful if you download a lot of data (>1TB) this may impact your bill, calculate first which will be the cost

Answer 4

AWS sdk API will only best option for upload entire folder and repo to s3 and download entire bucket of s3 to locally.

For uploading whole folder to s3

aws s3 sync . s3://BucketName

for download whole s3 bucket locally

aws s3 sync s3://BucketName . 

you can also assign path As like BucketName/Path for particular folder in s3 to download

Answer 5

Here is some stuff to download all buckets, list them, list their contents.

    //connection string
    private static void dBConnection() {
    app.setAwsCredentials(CONST.getAccessKey(), CONST.getSecretKey());
    conn = new AmazonS3Client(app.getAwsCredentials());
    app.setListOfBuckets(conn.listBuckets());
    System.out.println(CONST.getConnectionSuccessfullMessage());
    }

    private static void downloadBucket() {

    do {
        for (S3ObjectSummary objectSummary : app.getS3Object().getObjectSummaries()) {
            app.setBucketKey(objectSummary.getKey());
            app.setBucketName(objectSummary.getBucketName());
            if(objectSummary.getKey().contains(CONST.getDesiredKey())){
                //DOWNLOAD
                try 
                {
                    s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
                    s3Client.getObject(
                            new GetObjectRequest(app.getBucketName(),app.getBucketKey()),
                            new File(app.getDownloadedBucket())
                            );
                } catch (IOException e) {
                    e.printStackTrace();
                }

                do
                {
                     if(app.getBackUpExist() == true){
                        System.out.println("Converting back up file");
                        app.setCurrentPacsId(objectSummary.getKey());
                        passIn = app.getDataBaseFile();
                        CONVERT= new DataConversion(passIn);
                        System.out.println(CONST.getFileDownloadedMessage());
                    }
                }
                while(app.getObjectExist()==true);

                if(app.getObjectExist()== false)
                {
                    app.setNoObjectFound(true);
                }
            }
        }
        app.setS3Object(conn.listNextBatchOfObjects(app.getS3Object()));
    } 
    while (app.getS3Object().isTruncated());
}

/----------------------------Extension Methods-------------------------------------/

//Unzip bucket after download 
public static void unzipBucket() throws IOException {
    unzip = new UnZipBuckets();
    unzip.unZipIt(app.getDownloadedBucket());
    System.out.println(CONST.getFileUnzippedMessage());
}

//list all S3 buckets
public static void listAllBuckets(){
    for (Bucket bucket : app.getListOfBuckets()) {
        String bucketName = bucket.getName();
        System.out.println(bucketName + "\t" + StringUtils.fromDate(bucket.getCreationDate()));
    }
}

//Get the contents from the auto back up bucket
public static void listAllBucketContents(){     
    do {
        for (S3ObjectSummary objectSummary : app.getS3Object().getObjectSummaries()) {
            if(objectSummary.getKey().contains(CONST.getDesiredKey())){
                System.out.println(objectSummary.getKey() + "\t" + objectSummary.getSize() + "\t" + StringUtils.fromDate(objectSummary.getLastModified()));
                app.setBackUpCount(app.getBackUpCount() + 1);   
            }
        }
        app.setS3Object(conn.listNextBatchOfObjects(app.getS3Object()));
    } 
    while (app.getS3Object().isTruncated());
    System.out.println("There are a total of : " + app.getBackUpCount() + " buckets.");
}

}

Answer 6

As Neel Bhaat has explained in this blog, there are many different tools that can be used for this purpose. Some are AWS provided, where most are third party tools. All these tools require you to save your AWS account key and secret in the tool itself. Be very cautious when using third party tools, as the credentials you save in might cost you, your entire worth and drop you dead.

Therefore, I always recommend using the AWS CLI for this purpose. You can simply install this from this link. Next, run the following command and save your key, secret values in AWS CLI.

aws configure

And use the following command to sync your AWS S3 Bucket to your local machine. (The local machine should have AWS CLI installed)

aws s3 sync <source> <destination>

Examples:

1) For AWS S3 to Local Storage

aws s3 sync <S3Uri> <LocalPath>

2) From Local Storage to AWS S3

aws s3 sync <LocalPath> <S3Uri>

3) From AWS s3 bucket to another bucket

aws s3 sync <S3Uri> <S3Uri>