Android how to post InputStream by chunks using okhttp

问题

I have an issue with FileInputStream on Android with Kotlin and Okhttp. I want to post a video file by using an API, but if this file is bigger than 128mb I have to upload it chunk by chunk.

So, in my request header I have to specified the Content-Range (Something like this: Content-Range bytes 0-10485759/189305151)

And I need to post only this part of the file, that why I'm using FileInputStream, then repeat for each chunks.

I'm also using FileInputStream to avoid to split the file locally or to put it in memory.

To do so I'm using the read() method of FileInputStream to get the chunk

fileStream.read(b, 0, chunkLength.toInt())

But when I'm uploading the FileInputStream it's not uploading the correct file length.

In my case where the file size is 189.305.151 the final file size uploaded is 1.614.427.758 ( 1.614.427.758 is all filestream.available() added together)

I don't know if I'm in the right way to do it, but this is the only solution that I found, I hope someone can help me 🙏.

Here is my full code :

private val chunkLength = (1024L * 1024L) * 10L
private fun uploadBigFile(videoId: String, file: File, callBack: CallBack<Video>){
    val fileLength = file.length()
    try {
        var b = ByteArray(chunkLength.toInt())
        for (offset in 0 until fileLength step chunkLength){
            val fileStream = file.inputStream()

            var currentPosition = (offset.toInt()) + chunkLength.toInt() - 1

            // foreach chunk except the first one
            if(offset > 0){
                // skip all the chunks already uploaded
                fileStream.skip(offset)
            }
            // if this is the last chunk
            if(currentPosition > fileLength){
                fileStream.read(ByteArray((currentPosition - fileLength).toInt()))
                currentPosition = file.length().toInt() - 1

            }else{
                fileStream.read(b, 0, chunkLength.toInt())
                Log.e("stream after read", fileStream.available().toString())
                Log.e("------", "------")

            }

            // FileInputStream as RequestBody
            val videoFile = RequestBodyUtil.create(fileStream)

            val body = MultipartBody.Builder()
                .setType(MultipartBody.FORM)
                .addFormDataPart("file", "file", videoFile)
                .build()

            Log.e("bytes", "${offset.toInt()}-$currentPosition/${file.length().toInt()}")

            val request = Request.Builder()
                .url("$baseUri/videos/$videoId/source")
                .addHeader(
                    "Content-Range",
                    "bytes ${offset.toInt()}-$currentPosition/${file.length().toInt()}"
                )
                .post(body)
                .build()

            executor.execute(request, videoChunkTransformer, callBack)
        }
    }catch (e: Exception){
        Log.e("error", e.toString())
    }
}

There is my RequestBodyUtil Class :

class RequestBodyUtil {
companion object{
    fun create(inputStream: InputStream): RequestBody {
        return object : RequestBody() {
            override fun contentType(): MediaType? {
                return null
            }

            override fun contentLength(): Long {
                return try {
                    inputStream.available().toLong()
                } catch (e: IOException){
                    0
                }
            }

            @Throws(IOException::class)
            override fun writeTo(sink: BufferedSink) {
                try {
                    var source = inputStream.source()
                    sink.writeAll(source.buffer())
                }catch (e: IOException){
                    e.printStackTrace()
                } finally {
                    inputStream.closeQuietly()
                }
            }
        }
    }
}}

EDIT: I added some changes on my code, but still have some issues.

I can't upload the input stream's chunk, so I have decided to copy each chunks in an byte array output stream, but with OkHttp and Okio I can't upload OutputStream (I know that store the chunk on RAM but this is the only solution that found). That why I changed it into an byte array input stream then I can upload it. For some files I can upload and it works well but when the file is too large I have an OutOfMemory Error such like that :

E/AndroidRuntime: FATAL EXCEPTION: main
Process: video.api.androidkotlinsdkexample, PID: 8627
java.lang.OutOfMemoryError: Failed to allocate a 52428816 byte allocation with 6291456 free bytes and 9999KB until OOM, target footprint 532923344, growth limit 536870912
    at java.util.Arrays.copyOf(Arrays.java:3161)
    at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
    at video.api.androidkotlinsdk.api.VideoApi.uploadBigFile(VideoApi.kt:159)
    at video.api.androidkotlinsdk.api.VideoApi.access$uploadBigFile(VideoApi.kt:25)
    at video.api.androidkotlinsdk.api.VideoApi$upload$1.onSuccess(VideoApi.kt:375)
    at video.api.androidkotlinsdk.api.VideoApi$upload$1.onSuccess(VideoApi.kt:357)
    at video.api.androidkotlinsdk.http.HttpRequestExecutor$execute$1$onResponse$1.run(HttpRequestExecutor.kt:35)
    at android.os.Handler.handleCallback(Handler.java:883)
    at android.os.Handler.dispatchMessage(Handler.java:100)
    at android.os.Looper.loop(Looper.java:214)
    at android.app.ActivityThread.main(ActivityThread.java:7356)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)

There is the code:

private fun uploadBigFile(videoId: String, file: File, callBack: CallBack<Video>){
    val fileLength = file.length()
    try {
        var b = ByteArray(chunkLength.toInt())
        var bytesReads = 0

        for (offset in 0 until fileLength step chunkLength){
            var readBytes: Int
            val fileStream = file.inputStream()
            var currentPosition = (offset.toInt()) + chunkLength.toInt() - 1

            // foreach chunk except the first one
            if(offset > 0){
                // skip all the chunks already uploaded
                fileStream.skip(offset)
            }

            // if this is the last chunk
            if(currentPosition > fileLength){
                readBytes = fileStream.read(b, 0, (fileLength - bytesReads).toInt())
                currentPosition = file.length().toInt() - 1


            }else{
                readBytes = fileStream.read(b, 0, chunkLength.toInt())

            }
            bytesReads += readBytes

            val byteArrayOutput = ByteArrayOutputStream()
            byteArrayOutput.write(b, 0, readBytes)

            val byteArrayInput = ByteArrayInputStream(byteArrayOutput.toByteArray())


            val videoFile = RequestBodyUtil.create(byteArrayInput)

            val body = MultipartBody.Builder()
                .setType(MultipartBody.FORM)
                .addFormDataPart("file", "file", videoFile)
                .build()

            Log.e("bytes", "${offset.toInt()}-$currentPosition/${file.length().toInt()}")

            val request = Request.Builder()
                .url("$baseUri/videos/$videoId/source")
                .addHeader(
                    "Content-Range",
                    "bytes ${offset.toInt()}-$currentPosition/${file.length().toInt()}"
                )
                .post(body)
                .build()

            executor.execute(request, videoChunkTransformer, callBack)
            byteArrayOutput.close()
            byteArrayInput.close()
            fileStream.close()

        }
    }catch (e: Exception){
        Log.e("error", e.toString())
    }
}

来源：https://stackoverflow.com/questions/64557764/android-how-to-post-inputstream-by-chunks-using-okhttp