How do I delete a versioned bucket in AWS S3 using the CLI?

Question

I have tried both s3cmd:

$ s3cmd -r -f -v del s3://my-versioned-bucket/

And the AWS CLI:

$ aws s3 rm s3://my-v         


        

Answer 1

You can delete all the objects in the versioned s3 bucket. But I don't know how to delete specific objects.

$ aws s3api delete-objects \
      --bucket <value> \
      --delete "$(aws s3api list-object-versions \
      --bucket <value> | \
      jq '{Objects: [.Versions[] | {Key:.Key, VersionId : .VersionId}], Quiet: false}')"

Alternatively without jq:

$ aws s3api delete-objects \
    --bucket ${bucket_name} \
    --delete "$(aws s3api list-object-versions \
    --bucket "${bucket_name}" \
    --output=json \
    --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

Answer 2

If you have to delete/empty large S3 buckets, it becomes quite inefficient (and expensive) to delete every single object and version. It's often more convenient to let AWS expire all objects and versions.

aws s3api put-bucket-lifecycle-configuration \
  --lifecycle-configuration '{"Rules":[{
      "ID":"empty-bucket",
      "Status":"Enabled",
      "Prefix":"",
      "Expiration":{"Days":1},
      "NoncurrentVersionExpiration":{"NoncurrentDays":1}
    }]}' \
  --bucket YOUR-BUCKET

Then you just have to wait 1 day and the bucket can be deleted with:

aws s3api delete-bucket --bucket YOUR-BUCKET

Answer 3

This two bash lines are enough for me to enable the bucket deletion !

1: Delete objects aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --query='{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

2: Delete markers aws s3api delete-objects --bucket ${buckettoempty} --delete "$(aws s3api list-object-versions --bucket ${buckettoempty} --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"

Answer 4

One way to do it is iterate through the versions and delete them. A bit tricky on the CLI, but as you mentioned Java, that would be more straightforward:

AmazonS3Client s3 = new AmazonS3Client();
String bucketName = "deleteversions-"+UUID.randomUUID();

//Creates Bucket
s3.createBucket(bucketName);

//Enable Versioning
BucketVersioningConfiguration configuration = new BucketVersioningConfiguration(ENABLED);
s3.setBucketVersioningConfiguration(new SetBucketVersioningConfigurationRequest(bucketName, configuration ));

//Puts versions
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("some-bytes".getBytes()), null);
s3.putObject(bucketName, "some-key",new ByteArrayInputStream("other-bytes".getBytes()), null);

//Removes all versions
for ( S3VersionSummary version : S3Versions.inBucket(s3, bucketName) ) {
    String key = version.getKey();
    String versionId = version.getVersionId();          
    s3.deleteVersion(bucketName, key, versionId);
}

//Removes the bucket
s3.deleteBucket(bucketName);
System.out.println("Done!");

You can also batch delete calls for efficiency if needed.

Answer 5

I ran into issues with Abe's solution as the list_buckets generator is used to create a massive list called all_keys and I spent an hour without it ever completing. This tweak seems to work better for me, I had close to a million objects in my bucket and counting!

import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket("your-bucket-name-here")

chunk_counter = 0 #this is simply a nice to have
keys = []
for key in bucket.list_versions():
    keys.append(key)
    if len(keys) > 1000:
        bucket.delete_keys(keys)
        chunk_counter += 1
        keys = []
        print("Another 1000 done.... {n} chunks so far".format(n=chunk_counter))

#bucket.delete() #as per usual uncomment if you're sure!

Hopefully this helps anyone else encountering this S3 nightmare!

Answer 6

This bash script found here: https://gist.github.com/weavenet/f40b09847ac17dd99d16

worked as is for me.

I saved script as: delete_all_versions.sh and then simply ran:

./delete_all_versions.sh my_foobar_bucket

and that worked without a flaw.

Did not need python or boto or anything.