How to use while read line with tail -n

我的梦境 提交于 2020-04-11 07:15:08

问题


Problem: I have a CSV dump file - with excess of 250,000 lines. When I use while read - it takes a while (no pun intended). I would like to go back to the last 10,000 lines to do what I need to do instead of the 250,000 lines.

Code Snippet: My current code is this:

IFS=","
while read line
do

    awk_var=`echo "$line" | awk -F" " '{print $0}'`

    var_array=($awk_var)

    read -a var_array <<< "${awk_var}"

    echo "${var_array[1]}"


done </some_directory/directory/file_in_question.csv

Question: How can I use tail -n10000 with while read line when reading the file_in_question.csv with a bash script?


回答1:


Replace:

done </some_directory/directory/file_in_question.csv

with:

done < <(tail -n10000 /some_directory/directory/file_in_question.csv)

The <(...) construct is called process substitution. It creates a file-like object that bash can read from. Thus, this replaces reading from some_directory/directory/file_in_question.csv directly with reading from tail -n10000 /some_directory/directory/file_in_question.csv.

Using process substitution like this allows you to keep your while loop in the main shell, not a subshell. Because of this, variables that you create in the while loop will retain their value after the loop exits.

Speeding up the original code

The code as shown prints the second column of a CSV file. If that is all that the code is supposed to do, then it can be replaced with:

awk -F, '{print $2}' /some_directory/directory/file_in_question.csv



回答2:


Something like:

IFS=","
tail /var/log/httpd/error_log | while read foo bar
do
    echo $foo
done

I recommend you do the splitting in bash with read, instead of calling awk inefficiently there. Obviously rewriting the whole thing as an awk script will be faster than shell, but awk is harder less common language.




回答3:


Or this one.

while : 
do read l || { sleep 1 ; continue; }
   echo "==> $l"
done < /var/log/httpd/error_log


来源:https://stackoverflow.com/questions/33535558/how-to-use-while-read-line-with-tail-n

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!