What is the best memory buffer size to allocate to download a file from Internet?

Question

What is the best memory buffer size to allocate to download a file from Internet? Some of the samples said that it should be 1K. Well, I need to know in general why is it? A

Answer 1

For what it's worth, I tested reading a 1484 KB text file using progressive powers of two (sizes of 2,4,8,16...). I printed out to the console window the number of milliseconds required to read each one. Much past 8192 it didn't seem like much of a difference. Here are the results on my Windows 7 64 bit machine.

2^1 = 2 :264.0151
2^2 = 4 :193.011
2^3 = 8 :175.01
2^4 = 16 :153.0088
2^5 = 32 :139.0079
2^6 = 64 :134.0077
2^7 = 128 :132.0075
2^8 = 256 :130.0075
2^9 = 512 :133.0076
2^10 = 1024 :133.0076
2^11 = 2048 :90.0051
2^12 = 4096 :69.0039
2^13 = 8192 :60.0035
2^14 = 16384 :56.0032
2^15 = 32768 :53.003
2^16 = 65536 :53.003
2^17 = 131072 :52.003
2^18 = 262144 :53.003
2^19 = 524288 :54.0031
2^20 = 1048576 :55.0031
2^21 = 2097152 :54.0031
2^22 = 4194304 :54.0031
2^23 = 8388608 :54.003
2^24 = 16777216 :55.0032

Answer 2

I have problem with remote machine closing connection when used 64K buffer when downloading from iis.

I solved the problem raising the buffer to 2M

Answer 3

2k, 4k or 8k are good choices. It is not important how much is the page size, the change in speed would be really marginal and unpredictable.

First of all, C# memory can be moved, C# uses a compacting generational garbage collector. There is not any kind of information on where data will be allocated.

Second, arrays in C# can be formed by non-contiguous area of memory! Arrays are stored contiguously in virtual memory but contiguous virtual memory doesn't mean contiguous physical memory.

Third, array data structure in C# occupies some bytes more than the content itself (it stores array size and other informations). If you allocate page size amount of bytes, using the array will switch page almost always!

I would think that optimizing code using page size can be an non-optimization.

Usually C# arrays performs very well but if you really need precise allocation of data you need to use pinned arrays or Marshal allocation, but this will slow down the garbage collector.

Using marshal allocation and unsafe code can be a little faster but really it don't worth the effort.

I would say it is better to just use your arrays without thinking too much about the page size. Use 2k, 4k or 8k buffers.

Answer 4

Use at least 4KB. It's the normal page size for Windows (i.e. the granularity at which Windows itself manages memory), which means that the .Net memory allocator doesn't need to break down a 4KB page into 1KB allocations.

Of course, using a 64KB block will be faster, but only marginally so.

Answer 5

In some cases you can find out (or know, or hack around in a debugger and hence find out albeit in a non-change-resistant way) the size of a buffer used by the stream(s) you are writing to or reading from. In this case it will give a slight advantage if you match that size, or failing that, for one buffer to be a whole multiple of the other.

Otherwise 4096 unless you've a reason otherwise (wanting a small buffer to give rapid UI feedback for example), for the reasons already given.