A good way to bulk download images over http with Java

Question

We have a web application that needs to import 10-20 images from a partner site via http. If I have a list of strings that represent the urls I want to download does anybody have a suggestion for how to download them as fast as possible?

I could just put them in a for loop but if there is a simple way to parallelize this it would be probably be good for the end user. I would like to avoid using straight Java threads, although the executor framework might be a good idea.

Any ideas?

Answer 1

The Executor framework is EXACTLY what you want. Specifically the ExecutorCompletionService. Using this, you'll be able to submit requests quickly and in any order. Then you'll retrieve them exactly as they completed (as opposed to submission order).

Answer 2

Here is my take using Resty. (DISCLAIMER: I'm the author of Resty) Downloads all URLs given on the command line and prints out the file names.

package us.monoid.web.parallel;

import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import us.monoid.web.Resty;

public class Downloader {

    public static void main(String[] args) throws InterruptedException, ExecutionException {
        ExecutorService pool = Executors.newFixedThreadPool(10);
        List<Callable<File>> tasks = new ArrayList<Callable<File>>(args.length);
        for (final String url : args) {
            tasks.add(new Callable<File>() {
                public File call() throws Exception {
                    return new Resty().bytes(url).save(File.createTempFile("img", ".png"));
                }               
            });
        }
        List<Future<File>> results = pool.invokeAll(tasks);
        for (Future<File> ff : results) {
            System.out.println(ff.get());
        }
    }

}

Answer 3

Using Resty Library images can be downloaded with custom names as follows

try {
        ExecutorService pool = Executors.newFixedThreadPool(Names.size());
         List<Callable<File>> tasks = new ArrayList<Callable<File>>(Names.size());

         for (final String url : Urls) {

             tasks.add(new Callable<File>() {
               public File call() throws Exception {

                     File f=new File(directory+iimage);

                     return new Resty().bytes(url).save(f);

                 }

             });
             i++;
         }
         i=0;
         List<Future<File>> results = pool.invokeAll(tasks);
         for (Future<File> ff : results) {
             System.out.println(ff.get());
         }
    } catch (ExecutionException e) {
        // TODO Auto-generated catch block
        fails++;
        e.printStackTrace();

    }