Locking a file in Python

Question

I need to lock a file for writing in Python. It will be accessed from multiple Python processes at once. I have found some solutions online, but most fail for my purposes as

Answer 1

Locking is platform and device specific, but generally, you have a few options:

1. Use flock(), or equivalent (if your os supports it). This is advisory locking, unless you check for the lock, its ignored.
2. Use a lock-copy-move-unlock methodology, where you copy the file, write the new data, then move it (move, not copy - move is an atomic operation in Linux -- check your OS), and you check for the existence of the lock file.
3. Use a directory as a "lock". This is necessary if you're writing to NFS, since NFS doesn't support flock().
4. There's also the possibility of using shared memory between the processes, but I've never tried that; it's very OS-specific.

For all these methods, you'll have to use a spin-lock (retry-after-failure) technique for acquiring and testing the lock. This does leave a small window for mis-synchronization, but its generally small enough to not be an major issue.

If you're looking for a solution that is cross platform, then you're better off logging to another system via some other mechanism (the next best thing is the NFS technique above).

Note that sqlite is subject to the same constraints over NFS that normal files are, so you can't write to an sqlite database on a network share and get synchronization for free.