I keep a large cache in a dictionary with value IEnumerable. I remove items from the dictionary periodically and add
It's quite possible that the GC is coming to your rescue for a lot of the time, but that you're just going beyond its capabilities sometimes.
Just to be absolutely clear, is this a Dictionary or a Dictionary? If it's the latter, then perhaps you're holding onto references somewhere else?
How large is your cache getting? Do you have monitoring to keep track of it? What makes you think it's the dictionary which is causing the problem? If you have control over how much you cache, have you tried reducing the size?