I keep a large cache in a dictionary with value IEnumerable. I remove items from the dictionary periodically and add
Remember that a Heap can get fragmented as @Gabe mentioned. Even though you might have free memory it might not have a chunk large enough to allocate the dictionary to when it performs it's resize.
Perhaps you could use the caching block from the patterns and practices library MSDN Library Link that will assist you in implementing a good cache. Maybe you could choose an algorithm that doesn't dynamically allocate memory, with a fixed number of entries?
Also note that if there isn't any memory that you can use then it's a problem with the size of your cache, not the garbage collector.