Awk argument too long when merging csv files

问题

I have more than 10000 csv files in a folder and I'm trying to merge them by line using awk but if I run this command:
printf '%s\n' *.csv | xargs cat | awk 'FNR==1 && NR!=1{next;}{print}' *.csv > master.csv
I get the following errors:
/usr/bin/awk: Argument list too long and printf: write error: Broken pipe

回答1:

With the printf and xargs parts, you are sending the contents of the csv files into awk, but you also provide the filenames to awk. Pick one or the other: I'd suggest:

{ printf '%s\n' *.csv | xargs awk 'FNR==1 && NR!=1{next;}{print}'; } > master.csv

回答2:

If your file names don't contain newlines then you could do:

printf '%s\n' *.csv | awk 'NR==FNR{ARGV[ARGC++]=$0; next} !c++ || FNR>1' -

or if they can contain newlines then:

printf '%s\0' *.csv | awk -v RS='\0' 'NR==FNR{ARGV[ARGC++]=$0; next} !c++ || FNR>1' RS='\0' - RS='\n'

i.e. have awk read the list of CSV file names as input rather than the shell passing it to awk as arguments. That would work even if you had millions of CSV files.

For example, given this input:

$ head -n +50 file*.csv
==> file1.csv <==
Number
1
2

==> file2.csv <==
Number
10
11
12

the above would produce this output:

$ printf '%s\n' *.csv | awk 'NR==FNR{ARGV[ARGC++]=$0; next} !c++ || FNR>1' -
Number
1
2
10
11
12

来源：https://stackoverflow.com/questions/63672255/awk-argument-too-long-when-merging-csv-files