How best to read a File into List

Question

I am using a list to limit the file size since the target is limited in disk and ram. This is what I am doing now but is there a more efficient way?

readonly         


        

Answer 1

[Edit]

If you are doing this to trim the beginning of a log file, you can avoid loading the entire file by doing something like this:

// count the number of lines in the file
int count = 0;
using (var sr = new StreamReader("file.txt"))
{
    while (sr.ReadLine() != null) 
        count++;
}

// skip first (LOG_MAX - count) lines
count = LOG_MAX - count;
using (var sr = new StreamReader("file.txt"))
using (var sw = new StreamWriter("output.txt"))
{
    // skip several lines
    while (count > 0 && sr.ReadLine() != null) 
        count--;

    // continue copying
    string line = "";
    while ((line = sr.ReadLine()) != null)
        sw.WriteLine(line);
}

---

First of all, since File.ReadAllLines loads the entire file into a string array (string[]), copying to a list is redundant.

Second, you must understand that a List is implemented using a dynamic array under the hood. This means that CLR will need to allocate and copy several arrays until it can accommodate the entire file. Since the file is already on disk, you might consider trading speed for memory and working on disk data directly, or processing it in smaller chunks.

1. If you need to load it entirely in memory, at least try to leave in an array:

    string[] lines = File.ReadAllLines("file.txt");
   

2. If it really needs to be a List, load lines one by one:

    List lines = new List();
    using (var sr = new StreamReader("file.txt"))
    {
         while (sr.Peek() >= 0)
             lines.Add(sr.ReadLine());
    }
   

   Note: List has a constructor which accepts a capacity parameter. If you know the number of lines in advance, you can prevent multiple allocations by preallocating the array in advance:

    List lines = new List(NUMBER_OF_LINES);
   

3. Even better, avoid storing the entire file in memory and process it "on the fly":

    using (var sr = new StreamReader("file.txt"))
    {
         string line;
         while ((line = sr.ReadLine()) != null) 
         {
             // process the file line by line
         }
    }