Instead of removing just do the inverse (create a new dictionary from the old one containing only the elements you are interested in) and let the garbage collector take care of the old dictionary:
var newDictionary = oldDictionary.Where(x => x.Value != foo);