Bash while read loop extremely slow compared to cat, why?

Question

A simple test script here:

while read LINE; do
        LINECOUNT=$(($LINECOUNT+1))
        if [[ $(($LINECOUNT % 1000)) -eq 0 ]]; then echo $LINECOUNT; fi
do         


        

Answer 1

It's because the bash script is interpreted and not really optimised for speed in this case. You're usually better off using one of the external tools such as:

awk 'NR%1000==0{print}' inputFile

which matches your "print every 1000 lines" sample.

If you wanted to (for each line) output the line count in characters followed by the line itself, and pipe it through another process, you could also do that:

awk '{print length($0)" "$0}' inputFile | someOtherProcess

Tools like awk, sed, grep, cut and the more powerful perl are far more suited to these tasks than an interpreted shell script.