I find myself very intrigued by the existence of a ConcurrentBag
Bags are useful for storing objects wh
Since ordering doesn't matter a ConcurrentBag could be using a hashtable behind the scenes to allow for fast retrieval of data. But unlike a Hashset a bag accepts duplicates. Maybe each item could be paired with a Count property which is set to 1 when an item is added. If you add the same item for a second time, you could just increment the Count property of this item.
Then, to remove an item which has a count greater than one, you could just decrease the Count for this item. If the count was one, you would remove the Item-Count pair from the hashtable.