Retrofit OutOfMemory exception while loading a files in BASE64 to server

问题

I had an OOM exception when tried to load a pdf file in base64 as a field in a model throw Retrofit. I know, than this is not a normal way to upload files, but it's not a third party realisation. How I can fix this kind of a problem?

An application crached when I switching off my network connection

Failed to allocate a 30544558 byte allocation with 2085152 free bytes and 26MB until OOM

@Streaming
@POST("/api/order/")
fun makeOrder(@Header("Authorization") token: String, @Body order: OrderMainModel): Single<Response<PhoneNumberResponse>>

回答1:

That's a pretty crazy way of sending files. But despite it being wrong (as you normally do those things with @Multipart), I found your problem an interesting exercise. My solution is full of hacks, but if you're absolutely sure you can't influence the API in any way, then maybe this can help you.

You'll need an InputStream. There's no other way you'll be able to send files without eventually running out of memory. But how to send an InputStream in a request like this?

I assume OrderMainModel has a String field that keeps your giant Base64 string. I'd start by changing this to the File (or an InputStream itself).

Now, assuming you're using Gson (that's a pretty bold assumption but that's a parser I use - I'm pretty sure you can achieve something similar with any reasonable json library), create a custom TypeAdapter for type File. This TypeAdapter will force you to implement this interface:

class Adapter : TypeAdapter<File>() {
    override fun write(out: JsonWriter, value: File) {
        // implement this

    }

    override fun read(`in`: JsonReader): File {
        // ignore
    }
}

You can leave read alone, you won't need it.

Now, what you have to do in the write method is to read it part by part, continuously writing it to JsonWriter. Ah, and whatever you read, you'll need to convert it to base64 on the fly. There's a Base64InputStream available in android.util but it doesn't seem to be capable of encoding, you can use this one, from commons-codec: https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64InputStream.html (remember to pass true to doEncode in a constructor, otherwise it'll work in decoding mode).

Now, everything you have to do is to make Base64InputStream wrap FileInputStream, which wraps File you received in your TypeAdapter.

// 0 means no new lines in your Base64. null means no line separator. 
Base64InputStream(FileInputStream(file)), true, 0, null)

And slowly re-write it back to your JsonWriter.

But wait, that's not easy either!

JsonWriter doesn't offer any reasonable way of writing a single string (value) in a streaming fashion. The only idea I have is to hack it with reflection.

In order to do that, you need to retrieve JsonWriter's inner out object, type of Writer. Then, in order to be able to write to do it without breaking the state that JsonWriter keeps internally, you need to get an access on Writer's two private methods - writeDeferredValue and beforeValue and invoke them in order. Everything gets quite complicated and unsafe. But hey, it's all about fun, isn't it?

Here's a small PoC that presents the idea, rather not prod ready. ;-)

fun main() {
    val model = Model(File("file.txt"))
    val gson = GsonBuilder().registerTypeAdapter(File::class.java, Adapter()).create()
    // System.out just for PoC, don't try with large files because output will be massive
    gson.toJson(model, System.out)
}

data class Model(@SerializedName("file") val file: File)

class Adapter : TypeAdapter<File>() {
    override fun write(out: JsonWriter, value: File) {
        Base64InputStream(FileInputStream(value), true, 0, null).use { 
            out.writeFromStream(it)
        }
    }

    override fun read(`in`: JsonReader): File {
        throw UnsupportedOperationException()
    }

    // JsonWriter only offers value(String), which would need you to load the whole file to the memory.
    private fun JsonWriter.writeFromStream(inputStream: InputStream) {
        val declaredField = javaClass.getDeclaredField("out")
        val deferredName = javaClass.getDeclaredMethod("writeDeferredName")
        val beforeValue = javaClass.getDeclaredMethod("beforeValue")

        declaredField.isAccessible = true
        deferredName.isAccessible = true
        beforeValue.isAccessible = true

        val actualWriter = declaredField.get(this) as Writer

        deferredName.invoke(this)
        beforeValue.invoke(this)
        actualWriter.write("\"")
        for (byte in inputStream.buffered()) {
            actualWriter.write(byte.toInt())
        }
        actualWriter.write("\"")
    }
}

You can probably achieve a similar behaviour by integrating some lower-level HTTP APIs, which will let you write to the OutputStream without reflection. Perhaps some other parsers (Jackson, maybe?) would make it slightly more convenient.

... or just fight for the API change.

Good luck!

来源：https://stackoverflow.com/questions/57981532/retrofit-outofmemory-exception-while-loading-a-files-in-base64-to-server