How to prevent a race condition when multiple processes attempt to write to and then read from a file at the same time

Question

I have the following code (simplified for clarity):

import os
import errno
import imp


lib_dir = os.path.expanduser(\'~/.brian/cython_extensions\')
module_n         


        

Answer 1

The way to do this is to take an exclusive lock each time you open it. The writer holds the lock while writing data, while the reader blocks until the writer releases the lock with the fdclose call. This will of course fail if the file has been partially written and the writing process exits abnormally, so a suitable error to delete the file should be displayed if the module can't be loaded:

import os
import fcntl as F

def load_module():
    pyx_file = os.path.join(lib_dir, module_name + '.pyx')

    try:
        # Try and create/open the file only if it doesn't exist.
        fd = os.open(pyx_file, os.O_CREAT | os.O_EXCL | os.O_WRONLY):

        # Lock the file exclusively to notify other processes we're writing still.
        F.flock(fd, F.LOCK_EX)
        with os.fdopen(fd, 'w') as f:
            f.write(code)

    except OSError as e:
        # If the error wasn't EEXIST we should raise it.
        if e.errno != errno.EEXIST:
            raise

    # The file existed, so let's open it for reading and then try and
    # lock it. This will block on the LOCK_EX above if it's held by
    # the writing process.
    with file(pyx_file, "r") as f:
        F.flock(f, F.LOCK_EX)

    return imp.load_dynamic(module_name, module_path)

module = load_module()