Limit Threads count

Question

I have a List with items that I want to download. I use a for Loop to iterate the list.

For each item in this List I start a new Thread that references the item. My

Answer 1

The best way to handle this is to only create maxDownloads number of threads. Put all of your work items into a queue and let the threads compete with each other to figure out which one processes each work item.

var queue = new ConcurrentQueue(downloadList);
for (int i = 0; i < Math.Min(maxDownloads, queue.Count))
{
  var thread = new Thread(
    () =>
    {
      while (true)
      {
        downloadItem item = null;
        if (queue.TryDequeue(out item))
        {
          // Process the next work item.
          DownloadItem(item);
        }
        else
        {
          // No more work items are left.
          break;
        }
      }
    });
    thread.IsBackground = true;
    thread.Start();
}

You could also use a semaphore to throttle the number of threads processing work items. This is especially useful when the actual number of threads is unknown as would be the case if you were using the ThreadPool.

var semaphore = new Semaphore(maxDownloads, maxDownloads);
for (int i = 0; i < downloadList.Count; i++)
{
  downloadItem item = downloadList[i];
  ThreadPool.QueueUserWorkItem(
    (state) =>
    {
      semaphore.WaitOne();
      try
      {
        DownloadItem(item);
      }
      finally
      {
        semaphore.Release();
      }
    });
}

I am not particularly fond of either approach. The problem with the first is that a nonfixed amount of threads are created. It is generally advised to avoid creating threads in a for loop as that tends to not scale well. The problem with the second is that the semaphore will block some of the ThreadPool threads. That is not advised either because you are effecitvely claiming one of threads and then doing nothing on it. That might effect the performance of other unrelated tasks that happen to be sharing the ThreadPool. I think in this case either of the two options will be fine since crafting a more scalable pattern is more work than it is worth.