Can't grab progress on http POST file upload (Android)

Question

I am developing an Android app which enables the user to upload a file to services like Twitpic and others. The POST upload is done without any external libraries and works just fine. My only problem is, that I can't grab any progress because all the uploading is done when I receive the response, not while writing the bytes into the outputstream. Here is what I do:

URL url = new URL(urlString); 
HttpURLConnection conn = (HttpURLConnection) url.openConnection(); 
conn.setDoInput(true); 
conn.setDoOutput(true); 
conn.setUseCaches(false); 
conn.setRequestMethod("POST"); 
conn.setRequestProperty("Connection", "Keep-Alive"); 
conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary); 
DataOutputStream dos = new DataOutputStream(conn.getOutputStream());

Then I write the form data to dos, which is not so important here, now. After that I write the file data itself (reading from "in", which is the InputStream of the data I want to send):

while ((bytesAvailable = in.available()) > 0)
{ 
 bufferSize = Math.min(bytesAvailable, maxBufferSize);
 byte[] buffer = new byte[bufferSize];
 bytesRead = in.read(buffer, 0, bufferSize);
 dos.write(buffer, 0, bytesRead);
} 

After that, I send the multipart form data to indicate the end of the file. Then, I close the streams:

in.close(); 
dos.flush(); 
dos.close(); 

This all works perfectly fine, no problem so far. My problem is, however, that the whole process up to this point takes about one or two seconds no matter how large the file is. The upload itself seems to happen when I read the response:

DataInputStream inStream = new DataInputStream(conn.getInputStream());

This takes several seconds or minutes, depending on how large the file is and how fast the internet connection. My questions now are: 1) Why doesn't the uplaod happen when I write the bytes to "dos"? 2) How can I grab a progress in order to show a progress dialog during the upload when it all happens at once?

/EDIT: 1) I set the Content-Length in the header which changes the problem a bit, but does not in any way solve it: Now the whole content is uploaded after the last byte is written into the stream. So, this doesn't change the situation that you can't grab the progress, because again, the data is written at once. 2) I tried MultipartEntity in Apache HttpClient v4. There you don't have an OutputStream at all, because all the data is written when you perform the request. So again, there is no way to grab a progress.

Is there anyone out there who has any other idea how to grab a process in a multipart/form upload?

Answer 1

I have tried quite a lot in the last days and I think I have the answer to the initial question:

It's not possible to grab a progress using HttpURLConnection because there is a bug / unusual behavior in Android: http://code.google.com/p/android/issues/detail?id=3164#c6

It will be fixed post Froyo, which is not something one can wait for...

So, the only other option I found is to use the Apache HttpClient. The answer linked by papleu is correct, but it refers to general Java. The classes that are used there are not available in Android anymore (HttpClient 3.1 was part of the SDK, once). So, what you can do is add the libraries from HttpClient 4 (specifically apache-mime4j-0.6.jar and httpmime-4.0.1.jar) and use a combination of the first answer (by Tuler) and the last answer (by Hamy):

import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.Charset;
import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.MultipartEntity;

public class CountingMultipartEntity extends MultipartEntity {

    private final ProgressListener listener;

    public CountingMultipartEntity(final ProgressListener listener) {
        super();
        this.listener = listener;
    }

    public CountingMultipartEntity(final HttpMultipartMode mode, final ProgressListener listener) {
        super(mode);
        this.listener = listener;
    }

    public CountingMultipartEntity(HttpMultipartMode mode, final String boundary,
            final Charset charset, final ProgressListener listener) {
        super(mode, boundary, charset);
        this.listener = listener;
    }

    @Override
    public void writeTo(final OutputStream outstream) throws IOException {
        super.writeTo(new CountingOutputStream(outstream, this.listener));
    }

    public static interface ProgressListener {
        void transferred(long num);
    }

    public static class CountingOutputStream extends FilterOutputStream {

        private final ProgressListener listener;
        private long transferred;

        public CountingOutputStream(final OutputStream out,
                final ProgressListener listener) {
            super(out);
            this.listener = listener;
            this.transferred = 0;
        }


        public void write(byte[] b, int off, int len) throws IOException {
            out.write(b, off, len);
            this.transferred += len;
            this.listener.transferred(this.transferred);
        }

        public void write(int b) throws IOException {
            out.write(b);
            this.transferred++;
            this.listener.transferred(this.transferred);
        }
    }
}

This works with HttpClient 4, but the downside is that my apk now has a size of 235 kb (it was 90 kb when I used the multipart upload described in my question) and, even worse, an installed app size of 735 kb (about 170 kb before). That's really awful. Only to get a progress during upload the app size is now more than 4 times as big as it was before.

Answer 2

Try this out. It works.

connection = (HttpURLConnection) url_stripped.openConnection();
    connection.setRequestMethod("PUT");
    String boundary = "---------------------------boundary";
    String tail = "\r\n--" + boundary + "--\r\n";
    connection.addRequestProperty("Content-Type", "image/jpeg");
    connection.setRequestProperty("Connection", "Keep-Alive");
    connection.setRequestProperty("Content-Length", ""
            + file.length());
    connection.setDoOutput(true);

    String metadataPart = "--"
            + boundary
            + "\r\n"
            + "Content-Disposition: form-data; name=\"metadata\"\r\n\r\n"
            + "" + "\r\n";

    String fileHeader1 = "--"
            + boundary
            + "\r\n"
            + "Content-Disposition: form-data; name=\"uploadfile\"; filename=\""
            + fileName + "\"\r\n"
            + "Content-Type: application/octet-stream\r\n"
            + "Content-Transfer-Encoding: binary\r\n";

    long fileLength = file.length() + tail.length();
    String fileHeader2 = "Content-length: " + fileLength + "\r\n";
    String fileHeader = fileHeader1 + fileHeader2 + "\r\n";
    String stringData = metadataPart + fileHeader;

    long requestLength = stringData.length() + fileLength;
    connection.setRequestProperty("Content-length", ""
            + requestLength);
    connection.setFixedLengthStreamingMode((int) requestLength);
    connection.connect();

    DataOutputStream out = new DataOutputStream(
            connection.getOutputStream());
    out.writeBytes(stringData);
    out.flush();

    int progress = 0;
    int bytesRead = 0;
    byte buf[] = new byte[1024];
    BufferedInputStream bufInput = new BufferedInputStream(
            new FileInputStream(file));
    while ((bytesRead = bufInput.read(buf)) != -1) {
        // write output
        out.write(buf, 0, bytesRead);
        out.flush();
        progress += bytesRead;
        // update progress bar
        publishProgress(progress);
    }

    // Write closing boundary and close stream
    out.writeBytes(tail);
    out.flush();
    out.close();

    // Get server response
    BufferedReader reader = new BufferedReader(
            new InputStreamReader(connection.getInputStream()));
    String line = "";
    StringBuilder builder = new StringBuilder();
    while ((line = reader.readLine()) != null) {
        builder.append(line);
    }

Reference: http://delimitry.blogspot.in/2011/08/android-upload-progress.html

Answer 3

It is possible to grab the progress from the DefaultHttpClient. The fact that it stays in the line

response = defaultHttpClient.execute( httpput, context );

until complete data is sent, does not necessarily mean that you cannot grab the progress. Namely, HttpContext changes while executing this line, so if you approach it via different Thread, you may observe changes in that. What you need is ConnAdapter. It has HttpConnectionMetrics embedded

final AbstractClientConnAdapter connAdapter = (AbstractClientConnAdapter) context.getAttribute(ExecutionContext.HTTP_CONNECTION);
final HttpConnectionMetrics metrics = connAdapter.getMetrics();
int bytesSent = (int)metrics.getSentBytesCount();

If you check this periodically, you will observe the progress. Be sure to correctly handle threads, and all try/catch guarded. Especially, handle the case when Context is not valid any more (IllegalStateException), which occurs when Put is canceled or completed.

Answer 4

Use WatchedInputStream instead. It nice, great, and easy to use.

1. use WatchedInputStream to wrap your inputstream.

2. watchedStream.setListener

   watchedStream.setListener(new WatchedInputStream.Listener() {

   public void notify(int read) { // do something to update UI.
   } }