Matching a string in a Large text file?

Question

I have a list of strings containing about 7 million items in a text file of size 152MB. I was wondering what could be best way to implement the a function that takes a single

Answer 1

Are you going to have to match against this text file several times? If so, I'd create a HashSet. Otherwise, just read it line by line (I'm assuming there's one string per line) and see whether it matches.

152MB of ASCII will end up as over 300MB of Unicode data in memory - but in modern machines have plenty of memory, so keeping the whole lot in a HashSet will make repeated lookups very fast indeed.

The absolute simplest way to do this is probably to use File.ReadAllLines, although that will create an array which will then be discarded - not great for memory usage, but probably not too bad:

HashSet strings = new HashSet(File.ReadAllLines("data.txt"));
...

if (strings.Contains(stringToCheck))
{
    ...
}