Efficient FIFO queue for arbitrarily sized chunks of bytes in Python

Question

How do I implement a FIFO buffer to which I can efficiently add arbitrarily sized chunks of bytes to the head and from which I can efficiently pop arbitrarily sized chunks of by

Answer 1

Can you assume anything about the expected read/write amounts?

Chunking the data into, for example, 1024 byte fragments and using deque[1] might then work better; you could just read N full chunks, then one last chunk to split and put the remainder back on the start of the queue.

1) collections.deque

class collections.deque([iterable[, maxlen]])

Returns a new deque object initialized left-to-right (using append()) with data from iterable. If iterable is not specified, the new deque is empty.

Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short for “double-ended queue”). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction. ...