Writing to file using StreamWriter much slower than file copy over slow network

Question

I have a program that attempts to write quite a large amount of text to a file on a remote server overseas, which has a slow network connection.

Using the following code, where outputFileContent is a StringBuilder:

using (var outfile = new StreamWriter(myRemoteFilePath))
{
    outfile.Write(outputFileContent.ToString());
}

it is taking a seriously long time to run (several minutes), whereas if I first write to a local file and then copy it across to the remote location, it is much quicker (20-30 secs):

string tempFilePath = Path.GetTempFileName();
using (var outfile = new StreamWriter(tempFilePath))
{
    outfile.Write(outputFileContent.ToString());
}

System.IO.File.Copy(tempFilePath, myRemoteFilePath, true)

Any idea why this is happening? My only guess is that it is something to do with buffering across the network, or perhaps because the stream writer doesn't know how big it needs to be ahead of time.

Answer 1

If you create your StreamWriter using default buffer sizes then the underlying SMB protocol will be issuing write requests in chunks no larger than 4096 bytes, which means a large number of round-trips over the network. You could increase your StreamWriter's buffer size up to a maximum of 64k in order to reduce the number of round trips:

using (var outfile = new StreamWriter(myRemoteFilePath, false, Encoding.ASCII, 0x10000))

Increasing the buffer size beyond 64k will not help in any circumstance, as the underlying SMB protocol does not support buffer lengths beyond 64k. Note that a direct file copy still uses the SMB protocol, so from a network traffic perspective there's little difference between the operations except for the buffer sizes.

Answer 2

I'm not a security expert, but in my opinion, this is most related to permission issue.

Everytime you write somethin to a disk on remote server the permission is going to be checked against the user that execution that action. The permission to write, naturally.

In case instead of the copy, that control is executed only ones. So it's much faster.

In fact, to prove this you can try to copy several files over the network and after copy ZIP files (with the same amount of memory in zipped state) over network. In second case it will be much faster, as permission control is executed only once, for single ZIP file.

Hope this helps.