Processing a very large list in batches using System.Threading.Semaphore

问题

I have written the following code in .NET 2.0

foreach(Item i in millonItemsIterator) {
    ItemThreadManager m = new ItemThreadManager(i);
    System.Threading.Thread t = new System.Threading.Thread(m.Process);
    t.Start()
}
SendCompletionEmail();


public class ItemThreadManager {
    private static Semaphore s = new Semaphore(20, 20);
    private Item item = null;
    public ItemThreadManager(Item i) {
        item = i;
    } 

    public void Process() {
        s.WaitOne();
        // do something with item.
        s.Release();
    }
}

My intention behind using the Semaphore approach was that I process the items in the millionItems list in batches of 20.

But I don't want the email to be sent until all million items have been processed. The above code is sending email before the processing is complete.

My understanding is that above code will only process 20 items at a time. and will block if all 20 threads are in use.... but I think for the last 20 .... it will not wait for them to complete but straight go for sending email. So just for the last 20 how do I implement a Join?

Is my understanding correct?

回答1:

To make the process run sequentially with the call to SendEmail you can either just use a lock or use Tasks and chain the calls. Here's a link on How-to http://msdn.microsoft.com/en-us/library/dd537612.aspx

In light of the new info that it's .net 2.0 and you don't have Parallel Tasks. Use a BackgroundWorker to run the batch processing and then hook to it's RunWorkerCompleted event to know when the work is done and the email can be sent. Documentation here -> http://msdn.microsoft.com/en-US/library/system.componentmodel.backgroundworker_events(v=vs.80).aspx

来源：https://stackoverflow.com/questions/15111178/processing-a-very-large-list-in-batches-using-system-threading-semaphore

标签

.net-2.0