`uniq` without sorting an immense text file?

Question

I have a stupidly large text file (i.e. 40 gigabytes as of today) that I would like to filter for unique lines without sorting the file.

The file ha

Answer 1

Maybe not the answer you've been looking for but here goes: use a bloom filter. https://en.wikipedia.org/wiki/Bloom_filter This sort of problem is one of the main reasons they exist.