Java multithreaded file downloading performance

Question

Having recently worked on a project which required some more IO interaction than I\'m used to, I felt like I wanted to look past the regular libraries (Commons IO, in partic

Answer 1

Presumably the Apache HTTP client will be doing some buffering, with a smaller buffer. It will need a buffer to read the HTTP header reasonably, and probably handling chunked encoding.

Answer 2

Set a very large socket receive buffer. But really your performance will be limited by the network bandwidth, not CPU bandwidth. All you're doing really is allocating 1/3 of the network bandwidth to each downloader. I'd be surprised if you get much benefit.

Answer 3

To answer my own questions:

1. The increased CPU usage was due to a while() {} loop that was waiting for the threads to finish. As it turns out, awaitTermination is a much better alternative to wait for an Executor to finish :)
2. (And 3 and 4) This seems to be the nature of the beast; in the end I achieved what I wanted to do by using careful synchronization of the different threads that each download a chunk of data (well, in particular the writes of these chunks back to disk).

Answer 4

My immediate thought for best performance on Windows would be to use IO completions ports. What I don't know is (a) whether there are similar concepts in other OSes, and (b) whether there are any suitable Java wrappers? If portability isn't important to you, though, it may be possible to roll your own wrapper with JNI.