Replacing bytes of an uploaded file in Amazon S3

Question

I understand that in order to upload a file to Amazon S3 using Multipart, the instructions are here: http://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html

How do I go about replacing the bytes (say, between the range 4-1523) of an uploaded file? Do I need to make use of Multipart Upload to achieve this? or do I fire a REST call with the range specified in the HTTP header?

Appreciate any advice.

Answer 1

Objects in S3 are immutable.

If it's a small object, you'll need to upload the entire object again.

If it's an object over 5MB in size, then there is a workaround that allows you to "patch" a file, using a modified approach to the multipart upload API.

Background:

As you know, a multipart upload allows you to upload a file in "parts," with minimum part size 5MB and maximum part count 10,000.

However a multipart "upload" doesn't mean you have to "upload" all the data again, if some or all of it already exists in S3, and you can address it.

PUT part/copy allows you to "upload" the individual parts by specifying octet ranges in an existing object. Or more than one object.

Since uploads are atomic, the "existing object" can be the object you're in the process of overwriting, since it remains unharmed and in place until you actually complete the multipart upload.

But there appears to be nothing stopping you from using the copy capability to provide the data for the parts you want to leave the same, avoiding the actual upload then using a normal PUT part request to upload the parts that you want to have different content.

So, while not a byte-range patch with granularity to the level of 1 octet, this could be useful for emulating an in-place modification of a large file. Examples of valid "parts" would be replacing a minimum 5 MB chunk, on a 5MB boundary, for files smaller than 50GB, or replacing a mimimum 500MB chunk on 500MB boundary for objects up to 5TB, with minimum part sizes varying between those to extremes, because of the requirement that a multipart upload have no more than 10,000 parts. The catch is that a part must start at an appropriate offset, and you need to replace the whole part.