xargs: losing output when redirecting stdout to a file in parallel mode

Question

I am using GNU xargs (version 4.2.2) in parallel mode and I seem to be reliably losing output when redirecting to a file. When redirecting to a pipe, it appears to work corr

Answer 1

I know this question is about xargs, but if you keep on having issues with it, then perhaps GNU Parallel may be of help. Your xargs invocation would translate to:

$ < /tmp/nums parallel -j20 -N100 echo > /tmp/out; wc /tmp/out
26  2550 11643 /tmp/out

Answer 2

Your problem is due to the output from different processes being mixed. It is shown here:

parallel perl -e '\$a=\"1{}\"x10000000\;print\ \$a,\"\\n\"' '>' {} ::: a b c d e f
ls -l a b c d e f
parallel -kP4 -n1 grep 1 > out.par ::: a b c d e f
echo a b c d e f | xargs -P4 -n1 grep 1 > out.xargs-unbuf
echo a b c d e f | xargs -P4 -n1 grep --line-buffered 1 > out.xargs-linebuf
echo a b c d e f | xargs -n1 grep 1 > out.xargs-serial
ls -l out*
md5sum out*

The solution is to buffer the output from each job - either in memory or in tmpfiles (like GNU Parallel does).