How to remove duplicates from a file and write to the same file?

Question

I know my title is not much self-explanatory but let me try to explain it here.

I have a file name test.txt which has some duplicate lines. Now, what I want to do is remove those duplicate lines and at the same time update test.txt with the new content.

test.txt

AAAA
BBBB
AAAA
CCCC

I know I can use sort -u test.txt to remove the duplicates but to update the file with new content how do I redirect it's output to the same file. The below command doesn't work.

sort -u test.txt > test.txt

So, why the above command is not working and whats the correct way?

Also is there any other way like

sort_and_update_file test.txt

which sorts and automatically updates my file without any need of redirection.

Answer 1

This might work for you:

sort -u -o test.txt test.txt

Answer 2

Redirection in the shell will not work as you are trying to read and write from the same file at the same time. Actually the file is opened for writing (> file.txt) before the sort is even executed

@potong's answer works because the sort program itself probably stores all lines in memory, I would not rely on it because it does not explicitly specifies in the manpage that it CAN be the same as the input file (though it will likely work). Unless documented to work "in place" I would not do it (@perreal's answer would work, or you can store intermediate results in shell memory)

Answer 3

Use Sponge for Reading/Writing to Same File

You can use the sponge utility from moreutils to soak up standard output before writing the file. This prevents you from having to shuffle files around, and approximates an in-place edit. For example:

sort -u test.txt | sponge test.txt

Sample Output

Using your corpus, this results in the expected output.

$ cat test.txt 
AAAA
BBBB
CCCC

Answer 4

this is not as inefficient as it looks:

sort -u test.txt > test.txt.tmp && mv test.txt.tmp test.txt 

Answer 5

You can use vim for editing file in-place:

$ ex -s +'%!sort' -cxa test.txt

Multiple files:

$ ex -s +'bufdo!%!sort' -cxa *.*