Non-blocking file access with Twisted

Question

I\'m trying to figure out if there is a defacto pattern for file access using twisted. Lots of examples I\'ve looked at (twisted.python.log, twisted.persisted.dirdbm, twiste

Answer 1

The fdesc module might be useful for asynchronously talking to a socket or pipe, but when given an fd that refers to an ordinary filesystem file, it does blocking io (and via a rather odd interface at that). For disk io, fdesc is effectively snake oil; don't use it.

As of May 2017, the only reasonable way to get async disk io in twisted is by wrapping synchronous io calls in a deferToThread.

Answer 2

I'm not sure what you want to achieve. When you do logging, then Python will make sure (by the global interpreter log) that log messages from several threads go into the file one after the other.

If you're concerned about blocking IO, then the OS adds default buffers for your files (usually 4KB), and you can pass a buffer size in the open() call.

If you're concerned about something else, then please clarify your question.

Answer 3

I think you are looking for the fdesc module. For more information on non-blocking I/O in Python, you can also watch this video.

Answer 4

There is an open ticket for this in Twisted - #3983.

Answer 5

After plenty of searching, trial, and error, I finally figured how to use fdesc.

from __future__ import print_function

from twisted.internet.task import react
from twisted.internet import stdio, protocol
from twisted.internet.defer import Deferred
from twisted.internet.fdesc import readFromFD, setNonBlocking


class FileReader(protocol.Protocol):
    def __init__(self, filename):
        self.f = open(filename, 'rb')

    def dataReceived(self, data):
        self.transport.write(data)

    def connectionMade(self):
        fd = self.f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, self.dataReceived)

    def connectionLost(self, reason):
        self.f.close()

def main(reactor, filename):
    stdio.StandardIO(FileReader(filename))

[Edit: I also just figured out a simpler way that doesn't require the use of a protocol]

def getFile(filename):
    with open(filename) as f:
        d = Deferred()
        fd = f.fileno()
        setNonBlocking(fd)
        readFromFD(fd, d.callback)
        return d


def main(reactor, filename):
    d = getFile(filename)
    return d.addCallback(print)

Run either like so:

react(main, ['/path/to/file'])