Webflux upload large files cause Java heap space

问题

Probably not many developers are facing the problem the same as me.
But I want to share the solution which I had been solved for almost 1 month.
I use Kubernetes and docker-compose, this Webflux service (container) is set the memory limit 1g mem_limit: 1g I am not allowed to increase the memory limit.

coRouter has been used as a controller which is in the container.

@Configuration
class WebController(
) {

    @Bean
    fun endpoints() = coRouter {
        contentType(MediaType.MULTIPART_FORM_DATA).nest {
            POST("/uploadFile") { requestParm ->
                requestParm.awaitMultipartData().multipartFormData["file"]?.first()
            }
        }
    }
}

If I upload files using the API http://localhost:9004/uploadFile 100MB, 10 files (using JS Ajax)
It will produce java.lang.OutOfMemoryError: Java heap space because it doesn't have enough memory.
Whenever you call requestParm.awaitMultipartData() it will stream those bytes into memory.

回答1:

I realized that, my current code will store the upload files in memory directly. There are 2 ways to solve this problem.

1st way, add @EnableWebFlux on your Router or Controller


@Configuration
@EnableWebFlux
class WebController(
) { }

2nd way, extending from DelegatingWebFluxConfiguration to your Router or Controller
Reference: https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#webflux-config-advanced-java

@Configuration
class WebController(
): DelegatingWebFluxConfiguration() { }

Why does it solve the uploading large file problem?

These 2 ways above will be automatically used a supper class's method
configureHttpMessageCodecs(configurer: ServerCodecConfigurer)

https://docs.spring.io/spring-framework/docs/current/reference/html/web-reactive.html#webflux-codecs-multipart

By default, the DefaultPartHttpMessageReader is used, but this can be changed through the ServerCodecConfigurer. For more information about the DefaultPartHttpMessageReader, refer to to the javadoc of DefaultPartHttpMessageReader.

As you can see ServerCodecConfigurer will use DefaultPartHttpMessageReader by default.

DefaultPartHttpMessageReader has this important properties MaxInMemorySize (if memory full, store in disk)

https://docs.spring.io/spring-framework/docs/5.3.1/javadoc-api/org/springframework/http/codec/multipart/DefaultPartHttpMessageReader.html#setMaxInMemorySize-int-

Configure the maximum amount of memory allowed per part. When the limit is exceeded: file parts are written to a temporary file. non-file parts are rejected with DataBufferLimitException. By default this is set to 256K.

The default memory for uploading files is 256K
It will save those files to disk for temporary if the memory is full.

You can custom the memory to store size files also Spring WebFlux: set max file(s) size for uploading files

来源：https://stackoverflow.com/questions/64780779/webflux-upload-large-files-cause-java-heap-space

标签

spring-webflux

large-files