Add a big buffer to a pipe between two commands

Question

Given a bash command line of the form

commandA | commandB

I want to add a buffer of size ~1MB that sits between commandA and <

Answer 1

alternatively you could use a named pipe and run them in parallel:

mkfifo myfifo
commandB < myfifo &
commandA > myfifo
rm myfifo

Answer 2

BUFFER is called buffer. (man 1 buffer, maybe after apt-get install buffer)

Answer 3

The program buffer uses shared memory. This might be a problem, because in case of an error, memory may leak, because shared memory can outlive the program, which allocated the memory.

An alternative may be GNU dd:

commandA |
dd status=none iflag=fullblock bs=1M |
commandB

It is important to use the fullblock option. Otherwise dd may cause data loss, when reading from a pipe.

Parameters of dd explained

• status=none

  Set the level of information to print to stderr; 'none' suppresses everything but error messages

• iflag=fullblock

  accumulate full blocks of input

• bs=1M

  read and write up to one Mega bytes at a time (default: 512 bytes);

Answer 4

You can use

• buffer (mentioned)
• mbuffer (works on solaris too, possibly other UNIXes)

E.g.

    process1 | mbuffer -m 1024M | process2

to use a 1G buffer

Answer 5

There is another tool, pv - pipe viewer:

process1 | pv -pterbTCB 1G | process2

• B specifies the buffer size, here 1 Gigibyte
• C disables splice, which is required for B
• T shows the buffer level
• pterb are the default display switches needed due to the presence of T

pv might be available on systems where mbuffer/buffer is not in the official repositories (such as arch linux).