How Can I determine the number of items in ThreadPool Queue

Question

I am using the ThreadPool to queue 1000's of workitems

While(reading in data for processing)
{
    args = some data that has been read;
    ThreadPool.QueueUserWorkItem(new WaitCallback(threadRunner), args);
}

This is working very well, however, as the main thread queues the requests faster than they are processed memory is slowly eaten up.

I would like to do something akin to the following to throttle the queueing as the queue grows

Thread.Sleep(numberOfItemsCurrentlyQueued);

This would result in longer waits as the queue grows.

Is there any way to discover how many items are in the queue?

Answer 1

I don't think there is a built-in way, but you can introduce a [static?] counter that would increase/decrease; for that you would have to create your own method that would wrap ThreadPool.QueueUserWorkItem() and take care of the counter.

By the way, just in case you are running .NET 4.0, you should use TaskFactory.StartNew instead of ThreadPool.QueueUserWorkItem() - it's said to have better memory/thread management.

Answer 2

A more manageable abstraction for Producer/Consumer queue is BlockingCollection<T>. The example code there shows how to use Tasks to seed and drain the queue. The queue count is readily available via the Count property.

If you can, avoid using Sleep to delay production of more items. Have the producer wait on an Event or similar when queue gets too large, and have consumer(s) signal the Event when the queue backlog reaches a threshold where you are comfortable allowing more items to be produced. Always try to make things event-driven - Sleep is a bit of a guess.