Microsoft Visual C# 2008 Reducing number of loaded dlls

Question

How can I reduce the number of loaded dlls When debugging in Visual C# 2008 Express Edition?

When running a visual C# project in the debugger I get an OutOfMemoryE

Answer 1

3rd update: You can reduce the number of loaded DLLs significantly by disabling the Visual Studio hosting process (project properties, debug). Doing so will still allow you to debug the application, but it will get rid of a lot of DLLs and a number of helper threads as well.

On a small test project the number of loaded DLLs went from 69 to 34 when I disabled the hosting process. I also got rid of 10+ threads. All in all a significant reduction in memory usage which should also help reduce heap fragmentation.

Additional info on the hosting process: http://msdn.microsoft.com/en-us/library/ms242202.aspx

---

The reason you can load the second array in a new application is that each process gets a full 2 GB virtual address space. I.e. the OS will swap pages to allow each process to address the total amount of memory. When you try to allocate both arrays in one process the runtime must be able to allocate two contiguous chunks of the desired size. What are you storing in the array? If you store objects, you need additional space for each of the objects.

Remember an application doesn't actually request physical memory. Instead each application is given an address space from which they can allocate virtual memory. The OS then maps the virtual memory to physical memory. It is a rather complex process (Russinovich spends 100+ pages on how Windows handle memory in his Windows Internal book). For more details on how Windows does this please see http://blogs.technet.com/markrussinovich/archive/2008/11/17/3155406.aspx

Update: I've been pondering this question for a while and it does sound a bit odd. When you run the application through Visual Studio, you may see additional modules loaded depending on your configuration. On my setup I get a number of different DLLs loaded during debug due to profilers and TypeMock (which essentially does its magic via the profiler hooks).

Depending on the size and load address of these they may prevent the runtime from allocating contiguous memory. Having said that, I am still a bit surprised that you get an OOM after allocating just two of those big arrays as their combined size is less than 1 GB.

You can look at the loaded DLLs using the listdlls tools from SysInternals. It will show you load addresses and size. Alternatively, you can use WinDbg. The lm command shows loaded modules. If you want size as well, you need to specify the v option for verbose output. WinDbg will also allow you to examine the .NET heaps, which may help you to pinpoint why memory cannot be allocated.

2nd Update: If you're on Windows XP, you can try to rebase some of the loaded DLLs to free up more contiguous space. Vista and Windows 7 uses ASLR, so I am not sure you'll benefit from rebasing on those platforms.

Answer 2

This isn't an answer per se, but perhaps an alternative might work.

If the problem is indeed that you have fragmented memory, then perhaps one workaround would be to just use those holes, instead of trying to find a hole big enough for everything consecutively.

Here's a very simple BigArray class that doesn't add too much overhead (some overhead is introduced, especially in the constructor, in order to initialize the buckets).

The statistics for the array is:

• Main executes in 404ms
• static Program-constructor doesn't show up

The statistics for the class is:

• Main took 473ms
• static Program-constructor takes 837ms (initializing the buckets)

The class allocates a bunch of 8192-element arrays (13 bit indexes), which on 64-bit for reference types will fall below the LOB limit. If you're only going to use this for Int32, you can probably up this to 14 and probably even make it nongeneric, although I doubt it will improve performance much.

In the other direction, if you're afraid you're going to have a lot of holes smaller than the 8192-element arrays (64KB on 64-bit or 32KB on 32-bit), you can just reduce the bit-size for the bucket indexes through its constant. This will add more overhead to the constructor, and add more memory-overhead, since the outmost array will be bigger, but the performance should not be affected.

Here's the code:

using System;
using NUnit.Framework;

namespace ConsoleApplication5
{
    class Program
    {
        // static int[] a = new int[100 * 1024 * 1024];
        static BigArray<int> a = new BigArray<int>(100 * 1024 * 1024);

        static void Main(string[] args)
        {
            int l = a.Length;
            for (int index = 0; index < l; index++)
                a[index] = index;
            for (int index = 0; index < l; index++)
                if (a[index] != index)
                    throw new InvalidOperationException();
        }
    }

    [TestFixture]
    public class BigArrayTests
    {
        [Test]
        public void Constructor_ZeroLength_ThrowsArgumentOutOfRangeException()
        {
            Assert.Throws<ArgumentOutOfRangeException>(() =>
            {
                new BigArray<int>(0);
            });
        }

        [Test]
        public void Constructor_NegativeLength_ThrowsArgumentOutOfRangeException()
        {
            Assert.Throws<ArgumentOutOfRangeException>(() =>
            {
                new BigArray<int>(-1);
            });
        }

        [Test]
        public void Indexer_SetsAndRetrievesCorrectValues()
        {
            BigArray<int> array = new BigArray<int>(10001);
            for (int index = 0; index < array.Length; index++)
                array[index] = index;
            for (int index = 0; index < array.Length; index++)
                Assert.That(array[index], Is.EqualTo(index));
        }

        private const int PRIME_ARRAY_SIZE = 10007;

        [Test]
        public void Indexer_RetrieveElementJustPastEnd_ThrowsIndexOutOfRangeException()
        {
            BigArray<int> array = new BigArray<int>(PRIME_ARRAY_SIZE);
            Assert.Throws<IndexOutOfRangeException>(() =>
            {
                array[PRIME_ARRAY_SIZE] = 0;
            });
        }

        [Test]
        public void Indexer_RetrieveElementJustBeforeStart_ThrowsIndexOutOfRangeException()
        {
            BigArray<int> array = new BigArray<int>(PRIME_ARRAY_SIZE);
            Assert.Throws<IndexOutOfRangeException>(() =>
            {
                array[-1] = 0;
            });
        }

        [Test]
        public void Constructor_BoundarySizes_ProducesCorrectlySizedArrays()
        {
            for (int index = 1; index < 16384; index++)
            {
                BigArray<int> arr = new BigArray<int>(index);
                Assert.That(arr.Length, Is.EqualTo(index));

                arr[index - 1] = 42;
                Assert.That(arr[index - 1], Is.EqualTo(42));
                Assert.Throws<IndexOutOfRangeException>(() =>
                {
                    arr[index] = 42;
                });
            }
        }
    }

    public class BigArray<T>
    {
        const int BUCKET_INDEX_BITS = 13;
        const int BUCKET_SIZE = 1 << BUCKET_INDEX_BITS;
        const int BUCKET_INDEX_MASK = BUCKET_SIZE - 1;

        private readonly T[][] _Buckets;
        private readonly int _Length;

        public BigArray(int length)
        {
            if (length < 1)
                throw new ArgumentOutOfRangeException("length");

            _Length = length;
            int bucketCount = length >> BUCKET_INDEX_BITS;
            bool lastBucketIsFull = true;
            if ((length & BUCKET_INDEX_MASK) != 0)
            {
                bucketCount++;
                lastBucketIsFull = false;
            }

            _Buckets = new T[bucketCount][];
            for (int index = 0; index < bucketCount; index++)
            {
                if (index < bucketCount - 1 || lastBucketIsFull)
                    _Buckets[index] = new T[BUCKET_SIZE];
                else
                    _Buckets[index] = new T[(length & BUCKET_INDEX_MASK)];
            }
        }

        public int Length
        {
            get
            {
                return _Length;
            }
        }

        public T this[int index]
        {
            get
            {
                return _Buckets[index >> BUCKET_INDEX_BITS][index & BUCKET_INDEX_MASK];
            }

            set
            {
                _Buckets[index >> BUCKET_INDEX_BITS][index & BUCKET_INDEX_MASK] = value;
            }
        }
    }
}

Answer 3

You are battling virtual memory address space fragmentation. A process on the 32-bit version of Windows has 2 gigabytes of memory available. That memory is shared by code as well as data. Chunks of code are the CLR and the JIT compiler as well as the ngen-ed framework assemblies. Chunks of data are the various heaps used by .NET, including the loader heap (static variables) and the garbage collected heaps. These chunks are located at various addresses in the memory map. The free memory is available for you to allocate your arrays.

Problem is, a large array requires a contiguous chunk of memory. The "holes" in the address space, between chunks of code and data, are not large enough to allow you to allocate such large arrays. The first hole is typically between 450 and 550 Megabytes, that's why your first array allocation succeeded. The next available hole is a lot smaller. Too small to fit another big array, you'll get OOM even though you've got an easy gigabyte of free memory left.

You can look at the virtual memory layout of your process with the SysInternals' VMMap utility. Okay for diagnostics, but it isn't going to solve your problem. There's only one real fix, moving to a 64-bit version of Windows. Perhaps better: rethink your algorithm so it doesn't require such large arrays.

Answer 4

I had a similar issue once and what I ended up doing was using a list instead of an array. When creating the lists I set the capacity to the required sizes and I defined both lists BEFORE I tried adding values to them. I'm not sure if you can use lists instead of arrays but it might be something to consider. In the end I had to run the executable on a 64 bit OS, because when I added the items to the list the overall memory usage went above 2GB, but at least I wa able to run and debug locally with a reduced set of data.

Answer 5

A question: Are all elements of your array occupied? If many of them contain some default value then maybe you could reduce memory consumption using an implementation of a sparse array that only allocates memory for the non-default values. Just a thought.

Answer 6

Each 32bit process has a 2GB address space (unless you ask the user to add /3GB in boot options), so if you can accept some performance drop-off, you can start a new process to get 2GB more in address space - well, a little less than that. The new process would be still fragmented with all the CLR dlls plus all the Win32 DLLs they use, so you can get rid of all address space fragmentation caused by CLR dlls by writing the new process in a native language e.g. C++. You can even move some of your calculation to the new process so you get more address space in your main app and less chatty with your main process.

You can communicate between your processes using any of the interprocess communication methods. You can find many IPC samples in the All-In-One Code Framework.