How to delete duplicate lines in a file…AWK, SED, UNIQ not working on my file

Question

I find many ways to do this, AWK, SED, UNIQ, but none of them are working on my file.

I want to delete duplicate lines. Here is an example of part of my file:

KTBX
KFSO
KCLK
KTBX
KFSO
KCLK
PAJZ
PAJZ

NOTE: I had to manually add line feeds when I cut and pasted from the file...for some reason it was putting all the variables on one line. Makes me think that my 44,000 line text file actually has only "1" line? Is there a way to modify it so I can delete dups?

Answer 1

You can see all non-printed characters with this command:

od -c oldfile

If all your records are on one line, you can use sed to replace a whitespace (space, tab, newline) with a linebreak:

sed -e 's/\s\+/\n/g' oldfile > oldfile.1

Once you have multiple lines, this awk one-liner:

awk '!x[$0]++' oldfile.1 > newfile

my outfile:

KTBX
KFSO
KCLK
PAJZ

Answer 2

Perl One-Liner:

perl -nle 'unless($hash{$_}++){print $_}' file