What happens if an application calls more than 10 asynchronous URL Fetch on Google App Engine?

Question

Reading the Google App Engine documentation on asynchronous URL Fetch:

The app can have up to 10 simultaneous asynchronous URL Fetch calls

Answer 1

This is an old question, but I believe the accepted answer is incorrect or outdated and may confuse people. It's been a couple of months that I actually tested this, but in my experience Swizec is quite right that GAE will not queue but rather fail most asynchronous URL fetches exceeding the limit of around 10 simultaneous ones per request.

See https://developers.google.com/appengine/docs/python/urlfetch/#Python_Making_requests and https://groups.google.com/forum/#!topic/google-appengine/EoYTmnDvg8U for a description of the limit.

David Underhill has come up with a URL Fetch Manager for Python, which queues asynchronous URL fetches that exceed the limit in application code.

I have implemented something similar for Java, which synchronously blocks (due to the lack of a callback function or ListenableFutures) additional requests:

/**
 * A URLFetchService wrapper that ensures that only 10 simultaneous asynchronous fetch requests are scheduled. If the
 * limit is reached, the fetchAsync operations will block until another request completes.
 */
public class BlockingURLFetchService implements URLFetchService {
    private final static int MAX_SIMULTANEOUS_ASYNC_REQUESTS = 10;

    private final URLFetchService urlFetchService = URLFetchServiceFactory.getURLFetchService();
    private final Queue> activeFetches = new LinkedList<>();

    @Override
    public HTTPResponse fetch(URL url) throws IOException {
        return urlFetchService.fetch(url);
    }

    @Override
    public HTTPResponse fetch(HTTPRequest request) throws IOException {
        return urlFetchService.fetch(request);
    }

    @Override
    public Future fetchAsync(URL url) {
        block();

        Future future = urlFetchService.fetchAsync(url);
        activeFetches.add(future);
        return future;
    }

    @Override
    public Future fetchAsync(HTTPRequest request) {
        block();

        Future future = urlFetchService.fetchAsync(request);
        activeFetches.add(future);
        return future;
    }

    private void block() {
        while (activeFetches.size() >= MAX_SIMULTANEOUS_ASYNC_REQUESTS) {
            // Max. simultaneous async requests reached; wait for one to complete
            Iterator> it = activeFetches.iterator();
            while (it.hasNext()) {
                if (it.next().isDone()) {
                    it.remove();
                    break;
                }
            }
        }
    }
}