How can I lock a file while writing to it asynchronously

Question

I\'ve got two node threads running, one watching a directory for file consumption and another that is responsible for writing files to given directories.

Typically t

Answer 1

Writing a lock-state system is actually pretty simple. I can't find where I did this, but the idea is to:

1. create lock files whenever you acquire a lock,
2. delete them when releasing a lock,
3. delete them after a timeout has occurred,
4. throw whenever requesting a lock for a file whose lock file already exists.

A lock file is simply an empty file in a single directory. Each lock file gets its name from the hash of the full path of the file it represents. I used MD5 (which is relatively slow), but any hashing algo should be fine as long as you are confident there will be no collisions for path strings.

This isn't 100% thread-safe, since (unless I've missed something stupid) you can't atomically check if a file exists and create it in Node, but in my use case, I was holding locks for 10 seconds or more, so microsecond race conditions didn't seem that much of a threat. If you are holding and releasing thousands of locks per second on the same files, then this race condition might apply to you.

These will be advisory locks only, clearly, so it is up to you to ensure your code requests locks and catches the expected exceptions.

Answer 2

I'd use proper-lockfile for this. You can specify an amount of retries or use a retry config object to use an exponential backoff strategy. That way you can handle situations where two processes need to modify the same file at the same time.

Here's a simple example with some retry options:

const lockfile = require('proper-lockfile');
const Promise = require('bluebird');
const fs = require('fs-extra');
const crypto = require('crypto'); // random buffer contents

const retryOptions = {
    retries: {
        retries: 5,
        factor: 3,
        minTimeout: 1 * 1000,
        maxTimeout: 60 * 1000,
        randomize: true,
    }
};

let file;
let cleanup;
Promise.try(() => {
    file = '/var/tmp/file.txt';
    return fs.ensureFile(file); // fs-extra creates file if needed
}).then(() => {
    return lockfile.lock(file, retryOptions);
}).then(release => {
    cleanup = release;

    let buffer = crypto.randomBytes(4);
    let stream = fs.createWriteStream(file, {flags: 'a', encoding: 'binary'});
    stream.write(buffer);
    stream.end();

    return new Promise(function (resolve, reject) {
        stream.on('finish', () => resolve());
        stream.on('error', (err) => reject(err));
    });
}).then(() => {
    console.log('Finished!');
}).catch((err) => {
    console.error(err);
}).finally(() => {
    cleanup && cleanup();
});