Downloading Multiple Files Parallelly or Asynchronously in Java

问题

Here I am trying to download multiple files one after another:

Environment - Java 1.6

public List<Attachment> download(List<Attachment> attachments)
{
  for(Attachment attachment : attachments) {
    attachment.setDownStatus("Failed");
    String destLocation = "C:\Users\attachments";
    try {
        String attUrl = attachment.getUrl();
        String fileName = attachment.getFileName();            
        URL url = new URL(attUrl);
        File fileLocation = new File(destLoc, fileName);
        FileUtils.copyURLToFile(url, fileLocation);
        if(fileLocation.exists()) {
           attachment.setDownStatus("Completed");
         }
       } catch(Exception e) {
          attachment.setDownStatus("Failed");
       } finally {
          attachment.setDestLocation(destLocation);
       }
   }
  return attachments;
}

I am downloading the file from provided URL (http://cdn.octafinance.com/wp-content/uploads/2015/07/google-hummingbird.jpg).

FileUtils.copyURLToFile(url, fileLocation);

The above code does its downloading job perfectly, without any issues.

My Problem:
If the list of attachments are more it will take more time, so I would like to make it an asynchronous or parallel process instead of downloading sequentially.

回答1:

Actually, after carefully looking, Boris' code is faulty and will indeed not set some stuff sometimes. Here's a better version that fixes that:

public List<Attachment> download(List<Attachment> attachments) {
  ExecutorService executorService = Executors.newCachedThreadPool();
  List<Future<Attachment>> futures = new ArrayList<Future<Attachment>>();
  for (final Attachment attachment : attachments) {
    futures.add(executorService.submit(new Callable<Attachment>() {
      @Override
      public Attachment call() throws Exception {
        return doDownload(attachment);
      }
    }));
  }
  for (Future<Attachment> future: futures) {
    try {
      future.get();
    } catch (Exception ex) {
      // Do something
    }
  }
  return attachments;
}

private Attachment doDownload(Attachment attachment) throws Exception {
  attachment.setDownStatus("Failed");
  attachment.setDestLocation("C:\\Users\\attachments");
  String attUrl = attachment.getUrl();
  String fileName = attachment.getFileName();
  URL url = new URL(attUrl);
  File fileLocation = new File(attachment.getDestLocation(), fileName);
  FileUtils.copyURLToFile(url, fileLocation);
  if (fileLocation.exists()) {
    attachment.setDownStatus("Completed");
  }
  return attachment;
}

However, this is absolutely not optimal given your structure of Attachment and how you use it. I did not fix that: I only answered the question as it was asked.

回答2:

Use Java 8 Streams in combination with ForkJoinPool

public List<Attachment> download(List<Attachment> attachments) throws InterruptedException, ExecutionException {

    ForkJoinPool forkJoinPool = new ForkJoinPool(attachments.size());

    return forkJoinPool.submit(() -> processAttachments(attachments)).get();
}

private List<Attachment> processAttachments(List<Attachment> attachments) {
    return attachments.stream().parallel().map(attachment -> processSingleAttachment(attachment)).collect(Collectors.toList());
}

private Attachment processSingleAttachment(Attachment attachment){
     //business logic to download single attachment
    .
    .
}

回答3:

public List<Attachment> download(List<Attachment> attachments)
{
  ExecutorService executorService = Executors.newCachedThreadPool();
  for(final Attachment attachment : attachments){
    executorService.submit(new Runnable() {

        @Override
        public void run() {
          try{
            String attUrl = attachment.getUrl();
            String fileName = attachment.getFileName();
            String destLocation = "C:\Users\attachments";
            URL url = new URL(attUrl);
            String fileLocation = new File(destLoc, fileName);
            FileUtils.copyURLToFile(url, fileLocation);
            if(fileLocation.exists()) {
              attachment.setDownStatus("Completed");
            }
          }
          catch(Exception e){
            attachment.setDownStatus("Failed");
          }
        }
    });
 }
 executorService.shutdown();
 return attachments;
}

来源：https://stackoverflow.com/questions/33075208/downloading-multiple-files-parallelly-or-asynchronously-in-java