Volley - download directly to file (no in memory byte array)

Question

I'm using Volley as my network stack in a project I'm working on in Android. Part of my requirements is to download potentially very large files and save them on the file system.

Ive been looking at the implementation of volley, and it seems that the only way volley works is it downloads an entire file into a potentially massive byte array and then defers handling of this byte array to some callback handler.

Since these files can be very large, I'm worried about an out of memory error during the download process.

Is there a way to tell volley to process all bytes from an http input stream directly into a file output stream? Or would this require me to implement my own network object?

I couldn't find any material about this online, so any suggestions would be appreciated.

Answer 1

Okay, so I've come up with a solution which involves editing Volley itself. Here's a walk through:

Network response can't hold a byte array anymore. It needs to hold an input stream. Doing this immediately breaks all request implementations, since they rely on NetworkResponse holding a public byte array member. The least invasive way I found to deal with this is to add a "toByteArray" method inside NetworkResponse, and then do a little refactoring, making any reference to a byte array use this method, rather than the removed byte array member. This means that the transition of the input stream to a byte array happens during the response parsing. I'm not entirely sure what the long term effects of this are, and so some unit testing / community input would be a huge help here. Here's the code:

public class NetworkResponse {
    /**
     * Creates a new network response.
     * @param statusCode the HTTP status code
     * @param data Response body
     * @param headers Headers returned with this response, or null for none
     * @param notModified True if the server returned a 304 and the data was already in cache
     */
    public NetworkResponse(int statusCode, inputStream data, Map<String, String> headers,
            boolean notModified, ByteArrayPool byteArrayPool, int contentLength) {
        this.statusCode = statusCode;
        this.data = data;
        this.headers = headers;
        this.notModified = notModified;
        this.byteArrayPool = byteArrayPool;
        this.contentLength = contentLength;
    }

    public NetworkResponse(byte[] data) {
        this(HttpStatus.SC_OK, data, Collections.<String, String>emptyMap(), false);
    }

    public NetworkResponse(byte[] data, Map<String, String> headers) {
        this(HttpStatus.SC_OK, data, headers, false);
    }

    /** The HTTP status code. */
    public final int statusCode;

    /** Raw data from this response. */
    public final InputStream inputStream;

    /** Response headers. */
    public final Map<String, String> headers;

    /** True if the server returned a 304 (Not Modified). */
    public final boolean notModified;

    public final ByteArrayPool byteArrayPool;
    public final int contentLength;

    // method taken from BasicNetwork with a few small alterations.
    public byte[] toByteArray() throws IOException, ServerError {
        PoolingByteArrayOutputStream bytes =
                new PoolingByteArrayOutputStream(byteArrayPool, contentLength);
        byte[] buffer = null;
        try {

            if (inputStream == null) {
                throw new ServerError();
            }
            buffer = byteArrayPool.getBuf(1024);
            int count;
            while ((count = inputStream.read(buffer)) != -1) {
                bytes.write(buffer, 0, count);
            }
            return bytes.toByteArray();
        } finally {
            try {
                // Close the InputStream and release the resources by "consuming the content".
                // Not sure what to do about the entity "consumeContent()"... ideas?
                inputStream.close();
            } catch (IOException e) {
                // This can happen if there was an exception above that left the entity in
                // an invalid state.
                VolleyLog.v("Error occured when calling consumingContent");
            }
            byteArrayPool.returnBuf(buffer);
            bytes.close();
        }
    }

}

Then to prepare the NetworkResponse, we need to edit the BasicNetwork to create the NetworkResponse correctly (inside BasicNetwork.performRequest):

int contentLength = 0;
if (httpResponse.getEntity() != null)
{
     responseContents = httpResponse.getEntity().getContent(); // responseContents is now an InputStream
     contentLength = httpResponse.getEntity().getContentLength();
}

...

return new NetworkResponse(statusCode, responseContents, responseHeaders, false, mPool, contentLength);

That's it. Once the data inside network response is an input stream, I can build my own requests which can parse it directly into a file output stream which only hold a small in-memory buffer.

From a few initial tests, this seems to be working alright without harming other components, however a change like this probably requires some more intensive testing & peer reviewing, so I'm going to leave this answer not marked as correct until more people weigh in, or I see it's robust enough to rely on.

Please feel free to comment on this answer and/or post answers yourselves. This feels like a serious flaw in Volley's design, and if you see flaws with this design, or can think of better designs yourselves, I think it would benefit everyone.