How to efficiently write a large text file in C#?

Question

I am creating a method in C# which generates a text file for a Google Product Feed. The feed will contain upwards of 30,000 records and the text file currently weighs in at

Answer 1

Just move the using statement so it encompasses the whole of your code, and write directly to the file. I see no point in keeping it all in memory first.

Answer 2

File I/O operations are generally well optimized in modern operating systems. You shouldn't try to assemble the entire string for the file in memory ... just write it out piece by piece. The FileStream will take care of buffering and other performance considerations.

You can make this change easily by moving:

using (StreamWriter outfile = new StreamWriter(filePath)) {

to the top of the function, and getting rid of the StringBuilder writing directly to the file instead.

There are several reasons why you should avoid building up large strings in memory:

1. It can actually perform worse, because the StringBuilder has to increase its capacity as you write to it, resulting in reallocation and copying of memory.
2. It may require more memory than you can physically allocate - which may result in the use of virtual memory (the swap file) which is much slower than RAM.
3. For truly large files (> 2Gb) you will run out of address space (on 32-bit platforms) and will fail to ever complete.
4. To write the StringBuilder contents to a file you have to use ToString() which effectively doubles the memory consumption of the process since both copies must be in memory for a period of time. This operation may also fail if your address space is sufficiently fragmented, such that a single contiguous block of memory cannot be allocated.

Answer 3

Write one string at a time using StreamWriter.Write rather than caching everything in a StringBuilder.