Why does “uniq” count identical words as different?

问题

I want to calculate the frequency of the words from a file, where the words are one by line. The file is really big, so this might be the problem (it counts 300k lines in this example).

I do this command:

cat .temp_occ | uniq -c | sort -k1,1nr -k2 > distribution.txt

and the problem is that it gives me a little bug: it considers the same words as different.

For example, the first entries are:

306 continua 
278 apertura 
211 eventi 
189 murah 
182 giochi 
167 giochi

with giochi repeated twice as you can see.

At the bottom of the file it becomes even worse and it looks like this:

  1 win 
  1 win 
  1 win 
  1 win 
  1 win 
  1 win 
  1 win 
  1 win 
  1 win 
  1 winchester 
  1 wind 
  1 wind

for all the words.

What am I doing wrong?

回答1:

Try to sort first:

cat .temp_occ | sort| uniq -c | sort -k1,1nr -k2 > distribution.txt

回答2:

Or use "sort -u" which also eliminates duplicates. See here.

回答3:

The size of the file has nothing to do with what you're seeing. From the man page of uniq(1):

Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use 'sort -u' without 'uniq'. Also, comparisons honor the rules specified by 'LC_COLLATE'.`

So running uniq on

a
b
a

will return:

a
b
a

回答4:

Is it possible that some of the words have whitespace characters after them? If so you should remove them using something like this:

cat .temp_occ | tr -d ' ' | uniq -c | sort -k1,1nr -k2 > distribution.txt

来源：https://stackoverflow.com/questions/11860452/why-does-uniq-count-identical-words-as-different

标签

Linux

bash

shell

uniq