Read large txt file multithreaded?

Question

I have large txt file with 100000 lines. I need to start n-count of threads and give every thread unique line from this file.

What is the best way to do this? I thin

Answer 1

As @dtb mentioned above, the fastest way to read a file and then process the individual lines in a file is to: 1) do a File.ReadAllLines() into an array 2) Use a Parallel.For loop to iterate over the array.

You can read more performance benchmarks here.

The basic gist of the code you would have to write is:

string[] AllLines = File.ReadAllLines(fileName);
Parallel.For(0, AllLines.Length, x =>
{
    DoStuff(AllLines[x]);
    //whatever you need to do
});

With the introduction of bigger array sizes in .Net4, as long as you have plenty of memory, this shouldn't be an issue.