C# chunked array

Question

I need to allocate very large arrays of simple structs (1 GB RAM). After a few allocations/deallocations the memory becomes fragmented and an OutOfMemory exception is thrown.

This is under 32 bit. I'd rather not use 64 bit due to the performance penalty I get - the same application runs 30% slower in 64 bit mode.

Are you aware of some implementations of IList compatible arrays which allocate memory in chunks and not all at once? That would avoid my memory fragmentation problem.

Answer 1

Josh Williams presented a BigArray<T> class on his blog using a chunked array:

BigArray<T>, getting around the 2GB array size limit

You will find more useful information in this related question:

C# huge size 2-dim arrays

A simple ad-hoc fix might be to enable the 3GB switch for your application. Doing so allows your application to use more than the 2GB per-process limit of 32-bit Windows. However, be aware that the maximum object size that the CLR allows is still 2GB. The switch can be enabled using a post-built action for your main executable:

call "$(DevEnvDir)..\tools\vsvars32.bat"
editbin.exe /LARGEADDRESSAWARE "$(TargetPath)"

Answer 2

When instantiating an array, .Net tries to find a contiguous part of memory for your array. Since total memory limit for a 32-bit app is 2Gb, you can see that it is going to be tough to find such block after several allocations.

1. You can try using something like a LinkedList<T>, to avoid the need for contiguous allocation, or restructure your code to make these chunks smaller (although you will not be completely safe this won't happen with a 500Mb array also).

2. On the other hand, one solution would be to instantiate this large buffer only once, at the start of your app, and then implement an algorithm which would reuse this same space during you app's lifetime.

3. If you can use IEnumerable instead of IList to pass your data to rest of your program, you would be able to collapse this list using the SelectMany LINQ method.

4. And at the end, you can simply implement the IList interface in a custom class, and use several smaller arrays under the hood.

Answer 3

Would LinkedList work for you? http://msdn.microsoft.com/en-us/library/he2s3bh7.aspx