Locking with S3

Question

What\'s the recommended way to implement a simple locking mechanism to be used in conjunction with S3?

Example of what I want to do:

• acquire lock by ob

Answer 1

i dont think that you can do this using S3 only, using simpleDB's consistency enhancements as james said is a good way that works

you can look for some examples here : Amazon SimpleDB Consistency Enhancements

another approach that might be fine is using the versioning feature of S3
so basically, store an object id/version id pair in simpleDB as the most "valid" version
and assure that all GET requests will retrieve that version
after a successful PUT of a modified object, update the version id in the DB

this way you could also use the ability to retrieve previous versions of an object for restoring if needed.

Answer 2

Ok, I spent some time this morning playing with boto and I think I have a solution that works using SimpleDB. You need the latest boto release so that conditional puts and consistent reads are supported.

Example code here: http://pastebin.com/3XzhPqfY

Please post comments/suggestions. I believe this code should be fairly safe -- my test in main() tries it with 10 threads.

One thing I haven't addressed is that S3 reads are not consistent (right?), so in theory a thread may be operating on an old copy of the S3 value. It looks like there may be a workaround for that as described here:

http://www.shlomoswidler.com/2009/12/read-after-write-consistency-in-amazon.html