Is there a better way to determine the number of lines in a large txt file(1-2 GB)? [duplicate]

Answer 1

You already have the appropriate solution but you can simplify all your code to:

var lineCount = File.ReadLines(@"C:\MyHugeFile.txt").Count();

Benchmarks

I am not sure how dreamlax achieved his benchmark results but here is something so that anyone can reproduce on their machine; you can just copy-paste into LINQPad.

First let us prepare our input file:

var filePath = @"c:\MyHugeFile.txt";

for (int counter = 0; counter < 5; counter++)
{
    var lines = new string[30000000];

    for (int i = 0; i < lines.Length; i++)
    {
        lines[i] = $"This is a line with a value of: {i}";
    }

    File.AppendAllLines(filePath, lines);
}

This should produce a 150 million lines file which is roughly 6 GB.

Now let us run each method:

void Main()
{
    var filePath = @"c:\MyHugeFile.txt";
    // Make sure you clear windows cache!
    UsingFileStream(filePath);

    // Make sure you clear windows cache!
    UsingStreamReaderLinq(filePath);

    // Make sure you clear windows cache!
    UsingStreamReader(filePath);
}

private void UsingFileStream(string path)
{
    var sw = Stopwatch.StartNew();
    using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read))
    {
        long lineCount = 0;
        byte[] buffer = new byte[1024 * 1024];
        int bytesRead;

        do
        {
            bytesRead = fs.Read(buffer, 0, buffer.Length);
            for (int i = 0; i < bytesRead; i++)
                if (buffer[i] == '\n')
                    lineCount++;
        }
        while (bytesRead > 0);       
        Console.WriteLine("[FileStream] - Read: {0:n0} in {1}", lineCount, sw.Elapsed);
    }
}

private void UsingStreamReaderLinq(string path)
{
    var sw = Stopwatch.StartNew();
    var lineCount = File.ReadLines(path).Count();
    Console.WriteLine("[StreamReader+LINQ] - Read: {0:n0} in {1}", lineCount, sw.Elapsed);
}

private void UsingStreamReader(string path)
{
    var sw = Stopwatch.StartNew();
    long lineCount = 0;
    string line;
    using (var file = new StreamReader(path))
    {
        while ((line = file.ReadLine()) != null) { lineCount++; }
        Console.WriteLine("[StreamReader] - Read: {0:n0} in {1}", lineCount, sw.Elapsed);
    }
}

Which results in:

[FileStream] - Read: 150,000,000 in 00:00:37.3397443

[StreamReader+LINQ] - Read: 150,000,000 in 00:00:33.8842190

[StreamReader] - Read: 150,000,000 in 00:00:34.2102178

Update

Running with optimization ON results in:

[FileStream] - Read: 150,000,000 in 00:00:18.1636374

[StreamReader+LINQ] - Read: 150,000,000 in 00:00:33.3173354

[StreamReader] - Read: 150,000,000 in 00:00:32.3530890

Answer 2

I'm just thinking out loud here, but chances are performance is I/O bound and not CPU bound. In any case, I'm wondering if interpreting the file as text may be slowing things down as it will have to convert between the file's encoding and string's native encoding. If you know the encoding is ASCII or compatible with ASCII, you might be able to get away with just counting the number of times a byte with the value 10 appears (which is the character code for a linefeed).

What if you had the following:

FileStream fs = new FileStream("path.txt", FileMode.Open, FileAccess.Read, FileShare.None, 1024 * 1024);

long lineCount = 0;
byte[] buffer = new byte[1024 * 1024];
int bytesRead;

do
{
    bytesRead = fs.Read(buffer, 0, buffer.Length);
    for (int i = 0; i < bytesRead; i++)
        if (buffer[i] == '\n')
            lineCount++;
}
while (bytesRead > 0);

My benchmark results for 1.5GB text file, timed 10 times, averaged:

• StreamReader approach, 4.69 seconds
• File.ReadLines().Count() approach, 4.54 seconds
• FileStream approach, 1.46 seconds