remove duplicate lines with similar prefix

Question

I need to remove similar lines in a file which has duplicate prefix and keep the unique ones.

From this,

abc/def/ghi/
abc/def/ghi/jkl/one/
abc/def/gh         


        

Answer 1

Answer in case reordering the output is allowed.

sort -r file | awk 'a!~"^"$0{a=$0;print}'

1. sort -r file : sort lines in revers this way longer lines with the same pattern will be placed before shorter line of the same pattern

2. awk 'a!~"^"$0{a=$0;print}' : parse sorted output where a holds the previous line and $0 holds the current line

   • a!~"^"$0 checks for each line if current line is not a substring at the beginning of the previous line.
   • if $0 is not a substring (ie. not similar prefix), we print it and save new string in a (to be compared with next line)

The first line $0 is not in a because no value was assigned to a (first line is always printed)