How to use uniq -cd in bash scripting and extract only the count and not the line?

问题

I have a .sh file that takes a log file and extracts data and makes reports. I would like to calculate what percentage of the total lines does an error pop-up (top talkers).

So far I have this:

awk '// {print $4, substr($0, index($0,$9))}' | sort \
                        | uniq -cd | sort -nr | head -n20 > $filename-sr1.tmp

This outputs two columns, the count followed by the line.

How can I take just the count to make the calculations. Eg. count / total_lines = 0.000000...

回答1:

First I looked to get some similar output to you

cat text.txt | sort | uniq -cd | sort -nr | head -n20 > output.txt

output.txt now looks like this:

      5 red
      3 orange
      3 blue
      2 green

Hopefully that is similar to the output you have?

To get the percentages, count the lines in the original file, and then loop though each line in the output file and use cut to snip out each word/phrase count and bc to do the sums:

total_lines=$(wc -l < text.txt)
while read -r line; do 
    count=$(echo $line | cut -f1 -d " "); 
    percent=$(echo "scale=4; ($count/$total_lines)*100" | bc); 
    echo "$percent% -- $line"; 
done < output.txt;

Result looks like:

38.4600% -- 5 red
23.0700% -- 3 orange
23.0700% -- 3 blue
15.3800% -- 2 green

回答2:

Here's using just awk, though the output order will be arbitrary, so you may want to pipe it to a sort -n

$ cat file
foo
foo
bar
foo
quux
quux
$ awk '{a[$0]++} END{for (i in a) if (a[i]>1) printf "%5.2f%%\t%s\n", 100*a[i]/NR, i}' file
 33.3%  quux
 50.0%  foo

And adapting your current awk:

awk '{a[$4" "substr($0, index($0,$9))]++} END{for (i in a) if (a[i]>1) printf "%5.2f%%\t%s\n", 100*a[i]/NR, i}'
# or possibly
awk '{s=$4; for(i=9;i<=NF;++i) s=s" "$i; a[s]++} END{for (i in a) if (a[i]>1) printf "%5.2f%%\t%s\n", 100*a[i]/NR, i}'

来源：https://stackoverflow.com/questions/11959867/how-to-use-uniq-cd-in-bash-scripting-and-extract-only-the-count-and-not-the-lin

标签

bash

uniq