Reading large file in chunks c#

Question

I want to read very large file (4GBish) chunk by chunk.

I am currently trying to use a StreamReader and the Read() read method. The syntax is:

sr.Read(char[] buffer, int index, int count)

Because the index is an int it will overflow in my case. What should i use instead?

Answer 1

The index is the starting index of buffer not the index of file pointer, usually it would be zero. On each Read call you will read characters equal to the count parameter of Read method. You would not read all the file at once rather read in chunks and use that chunk.

The index of buffer at which to begin writing, reference.

char[] c = null;
while (sr.Peek() >= 0) 
{
    c = new char[1024];
    sr.Read(c, 0, c.Length);
    //The output will look odd, because 
    //only five characters are read at a time.
    Console.WriteLine(c);
}

The above example will ready 1024 bytes and will write to console. You can use these bytes, for instance sending these bytes to other application using TCP connection.

When using the Read method, it is more efficient to use a buffer that is the same size as the internal buffer of the stream, where the internal buffer is set to your desired block size, and to always read less than the block size. If the size of the internal buffer was unspecified when the stream was constructed, its default size is 4 kilobytes (4096 bytes), MSDN.

Answer 2

You could try the simpler version of Read which doesn't chunk the stream but instead reads it character by character. You would have to implement the chunking your self, but it would give you more control allowing you to use a Long instead.

http://msdn.microsoft.com/en-us/library/ath1fht8(v=vs.110).aspx