How to properly parallelise job heavily relying on I/O

Question

I\'m building a console application that have to process a bunch of data.

Basically, the application grabs references from a DB. For each reference, parse the conten

Answer 1

Just a suggestion, but have you looked into the Consumer / Producer pattern ? A certain number of threads would read your files on disk and feed the content to a queue. Then another set of threads, known as the consumers, would "consume" the queue as its filled. http://zone.ni.com/devzone/cda/tut/p/id/3023