Library or tool to download multiple files in parallel [closed]

Answer 1

There are lots of options but it will be hard to find one which fits all your needs.

In your case, try this approach:

1. Create a queue.
2. Put URLs to download into this queue (or "config objects" which contain the URL and other data like the user name, the destination file, etc).
3. Create a pool of threads
4. Each thread should try to fetch a URL (or a config object) from the queue and process it.

Use another thread to collect the results (i.e. another queue). When the number of result objects == number of puts in the first queue, then you're finished.

Make sure that all communication goes via the queue or the "config object". Avoid accessing data structures which are shared between threads. This should save you 99% of the problems.