howto abort all incomplete multipart uploads for a bucket

Question

Sometimes multipart uploads hang or don\'t complete for some reason. In that case you are stuck with orphaned parts that are tricky to remove. You can list them with:

Answer 1

You can set up lifecycle rules to automatically purge those after some amount of time. Here's a blog post demonstrating how to do it in the console:

https://aws.amazon.com/blogs/aws/s3-lifecycle-management-update-support-for-multipart-uploads-and-delete-markers/

To do this in boto3:

import boto3


s3 = boto3.client('s3')
try:
    lifecycle = s3.get_bucket_lifecycle(Bucket='bucket')
except ClientError:
    lifecycle = {'Rules': []}
lifecycle['Rules'].append({
    'ID': 'PruneAbandonedMultipartUploads',
    'Status': 'Enabled',
    'Prefix': '',
    'AbortIncompleteMultipartUpload': {
        'DaysAfterInitiation': 7
    }
})
s3.put_bucket_lifecycle(Bucket='bucket', LifecycleConfiguration=lifecycle)

Adding that configuration in the cli would be much the same:

$ aws s3api get-bucket-lifecycle --bucket bucket > lifecycle.json
# Edit the lifecycle, adding the same configuration as in the boto3 sample
$ aws s3api put-bucket-lifecycle --bucket bucket --lifecycle-configuration file://lifecycle.json

If you have no lifecycle policy on your bucket, get-bucket-lifecycle will raise a ClientError. A robust implementation would make sure the right error is returned.

A policy only with that configuration would look like so:

{
    "Rules": [
        {
            "ID": "PruneAbandonedMultipartUpload",
            "Status": "Enabled",
            "AbortIncompleteMultipartUpload": {
                "DaysAfterInitiation": 7
            }
        }
    ]
}