How do I delete a versioned bucket in AWS S3 using the CLI?

Question

I have tried both s3cmd:

$ s3cmd -r -f -v del s3://my-versioned-bucket/

And the AWS CLI:

$ aws s3 rm s3://my-v         


        

Answer 1

This works for me. Maybe running later versions of something and above > 1000 items. been running a couple of million files now. However its still not finished after half a day and no means to validate in AWS GUI =/

# Set bucket name to clearout
BUCKET = 'bucket-to-clear'

import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET)

max_len         = 1000      # max 1000 items at one req
chunk_counter   = 0         # just to keep track
keys            = []        # collect to delete

# clear files
def clearout():
    global bucket
    global chunk_counter
    global keys
    result = bucket.delete_objects(Delete=dict(Objects=keys))

    if result["ResponseMetadata"]["HTTPStatusCode"] != 200:
        print("Issue with response")
        print(result)

    chunk_counter += 1
    keys = []
    print(". {n} chunks so far".format(n=chunk_counter))
    return

# start
for key in bucket.object_versions.all():
    item = {'Key': key.object_key, 'VersionId': key.id}
    keys.append(item)
    if len(keys) >= max_len:
        clearout()

# make sure last files are cleared as well
if len(keys) > 0:
    clearout()

print("")
print("Done, {n} items deleted".format(n=chunk_counter*max_len))
#bucket.delete() #as per usual uncomment if you're sure!

Answer 2

I found the other answers either incomplete or requiring external dependencies to be installed (like boto), so here is one that is inspired by those but goes a little deeper.

As documented in Working with Delete Markers, before a versioned bucket can be removed, all its versions must be completely deleted, which is a 2-step process:

1. "delete" all version objects in the bucket, which marks them as deleted but does not actually delete them
2. complete the deletion by deleting all the deletion marker objects

Here is the pure CLI solution that worked for me (inspired by the other answers):

#!/usr/bin/env bash

bucket_name=...

del_s3_bucket_obj()
{
    local bucket_name=$1
    local obj_type=$2
    local query="{Objects: $obj_type[].{Key:Key,VersionId:VersionId}}"
    local s3_objects=$(aws s3api list-object-versions --bucket ${bucket_name} --output=json --query="$query")
    if ! (echo $s3_objects | grep -q '"Objects": null'); then
        aws s3api delete-objects --bucket "${bucket_name}" --delete "$s3_objects"
    fi
}

del_s3_bucket_obj ${bucket_name} 'Versions'
del_s3_bucket_obj ${bucket_name} 'DeleteMarkers'

Once this is done, the following will work:

aws s3 rb "s3://${bucket_name}"

Not sure how it will fare with 1000+ objects though, if anyone can report that would be awesome.

Answer 3

Here is a one liner you can just cut and paste into the command line to delete all versions and delete markers (it requires aws tools, replace yourbucket-name-backup with your bucket name)

echo '#!/bin/bash' > deleteBucketScript.sh \
&& aws --output text s3api list-object-versions --bucket $BUCKET_TO_PERGE \
| grep -E "^VERSIONS" |\
awk '{print "aws s3api delete-object --bucket $BUCKET_TO_PERGE --key "$4" --version-id "$8";"}' >> \
deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh; echo '#!/bin/bash' > \
deleteBucketScript.sh && aws --output text s3api list-object-versions --bucket $BUCKET_TO_PERGE \
| grep -E "^DELETEMARKERS" | grep -v "null" \
| awk '{print "aws s3api delete-object --bucket $BUCKET_TO_PERGE --key "$3" --version-id "$5";"}' >> \
deleteBucketScript.sh && . deleteBucketScript.sh; rm -f deleteBucketScript.sh;

then you could use:

aws s3 rb s3://bucket-name --force

Answer 4

You can do this from the AWS Console using Lifecycle Rules.

Open the bucket in question. Click the Management tab at the top. Make sure the Lifecycle Sub Tab is selected. Click + Add lifecycle rule

On Step 1 (Name and scope) enter a rule name (e.g. removeall) Click Next to Step 2 (Transitions) Leave this as is and click Next.

You are now on the 3. Expiration step. Check the checkboxes for both Current Version and Previous Versions. Click the checkbox for "Expire current version of object" and enter the number 1 for "After _____ days from object creation Click the checkbox for "Permanently delete previous versions" and enter the number 1 for "After _____ days from becoming a previous version"

click the checkbox for "Clean up incomplete multipart uploads" and enter the number 1 for "After ____ days from start of upload" Click Next Review what you just did.
Click Save

Come back in a day and see how it is doing.

Answer 5

For those using multiple profiles via ~/.aws/config

import boto3

PROFILE = "my_profile"
BUCKET = "my_bucket"

session = boto3.Session(profile_name = PROFILE)
s3 = session.resource('s3')
bucket = s3.Bucket(BUCKET)
bucket.object_versions.delete()

Answer 6

By far the easiest method I've found is to use this CLI tool, s3wipe. It's provided as a docker container so you can use it like so:

$ docker run -it --rm slmingol/s3wipe --help
usage: s3wipe [-h] --path PATH [--id ID] [--key KEY] [--dryrun] [--quiet]
              [--batchsize BATCHSIZE] [--maxqueue MAXQUEUE]
              [--maxthreads MAXTHREADS] [--delbucket] [--region REGION]

Recursively delete all keys in an S3 path

optional arguments:
  -h, --help               show this help message and exit
  --path PATH              S3 path to delete (e.g. s3://bucket/path)
  --id ID                  Your AWS access key ID
  --key KEY                Your AWS secret access key
  --dryrun                 Don't delete. Print what we would have deleted
  --quiet                  Suprress all non-error output
  --batchsize BATCHSIZE    # of keys to batch delete (default 100)
  --maxqueue MAXQUEUE      Max size of deletion queue (default 10k)
  --maxthreads MAXTHREADS  Max number of threads (default 100)
  --delbucket              If S3 path is a bucket path, delete the bucket also
  --region REGION          Region of target S3 bucket. Default vaue `us-
                           east-1`

Example

Here's an example where I'm deleting all the versioned objects in a bucket and then deleting the bucket:

$ docker run -it --rm slmingol/s3wipe \
   --id $(aws configure get default.aws_access_key_id) \
   --key $(aws configure get default.aws_secret_access_key) \
   --path s3://bw-tf-backends-aws-example-logs \
   --delbucket
[2019-02-20@03:39:16] INFO: Deleting from bucket: bw-tf-backends-aws-example-logs, path: None
[2019-02-20@03:39:16] INFO: Getting subdirs to feed to list threads
[2019-02-20@03:39:18] INFO: Done deleting keys
[2019-02-20@03:39:18] INFO: Bucket is empty.  Attempting to remove bucket

How it works

There's a bit to unpack here but the above is doing the following:

• docker run -it --rm mikelorant/s3wipe - runs s3wipe container interactively and deletes it after each execution
• --id & --key - passing our access key and access id in
• aws configure get default.aws_access_key_id - retrieves our key id
• aws configure get default.aws_secret_access_key - retrieves our key secret
• --path s3://bw-tf-backends-aws-example-logs - bucket that we want to delete
• --delbucket - deletes bucket once emptied

References

• https://github.com/slmingol/s3wipe
• Is there a way to export an AWS CLI Profile to Environment Variables?
• https://cloud.docker.com/u/slmingol/repository/docker/slmingol/s3wipe