Camel reverse proxy - no response stream caching

问题

I am trying to implement a memory efficient http reverse proxy that is only working on streams. The Jetty consumer places the input stream into the exchange and I can hook it up with a http producer to forward the request. No problem there.

However, all http producers that I am aware of (Jetty, http4, netty-http) read the response stream into heap memory and place its contents into the exchange in some form or another instead of a handle to the stream. And none of them seem to offer an option to make them do so.

I found this thread which describes the same problem and also suggests a solution. But looking at the code of the http4 HttpProducer in Camel 2.13.1, it does not look like the proposed change made it into the Camel code base after all.

Is there any way to achieve the stream-only approach with Camel? So, with a minimal memory footprint I could do something along the line of this:

<route id="reverse_proxy" streamCache="false">
    <from ref="jetty.http.server"/>
    <bean ref="streamHolder" method="enableCaching"/>
    <bean ref="streamHolder" method="parsePayloadHeaderInfoAndDoStuff"/>
    <bean ref="streamHolder" method="resetStream"/>
    <to ref="http.client"/> <!-- Register completion synchronization hook to close stream. -->
    <bean ref="streamHolder" method="enableCaching"/>
    <bean ref="streamHolder" method="parsePayloadResponseHeaderAndDoStuff"/>
    <bean ref="streamHolder" method="resetStream"/>
</route>

EDIT - Additional info on where exactly the input stream ends up in memory:

http4: Everything happens in org.apache.camel.component.http4.HttpProducer::process() -> populateResponse(..) -> extractResponseBody(..) -> doExtractResponseBodyAsStream(); and here the original stream is copied into an instance of CachedOutputStream.

Jetty: org.eclipse.jetty.client.AsyncHttpConnection::handle() -> org.eclipse.jetty.http.HttpParser::parseNext() will fill a byte array in org.eclipse.jetty.client.ContentExchange which is a CachedExchange which is a HttpExchange.

netty-http: Builds a pipeline that assembles the HttpResponse content as a composite ChannelBuffer. The wrapped channel buffers make up the complete response stream.

I have debugged all three clients and did not stumble across a branch not taken that would leave me with the original input stream as the exchange body.

This is reproducible with a route as simple as this:

<camelContext id="pep-poc">
    <endpoint id="jetty.http.server" uri="jetty:http://{{http.host.server}}:{{http.port.server}}/my/frontend?disableStreamCache=true"/>
    <endpoint id="http.client" uri="jetty:http://{{http.host.client}}:{{http.port.client}}/large_response.html?bridgeEndpoint=true&amp;throwExceptionOnFailure=false&amp;disableStreamCache=true"/>

    <route id="reverse_proxy" startupOrder="10" streamCache="false">
        <from ref="jetty.http.server"/>
        <to ref="http.client"/>
    </route>
</camelContext>

I have an Apache2 return a 750MB file as large_response.html.

EDIT 2

Indeed this is a problem with all available HTTP producers. See this thread on the Camel mailing list and the corresponding JIRA ticket.

回答1:

They do not read the stream into memory unless you access the message body on demand, and tell Camel to read it into memory as a String type etc.

See this cookbook example how to do a stream based proxy

http://camel.apache.org/how-to-use-camel-as-a-http-proxy-between-a-client-and-server.html

来源：https://stackoverflow.com/questions/24861164/camel-reverse-proxy-no-response-stream-caching

标签

apache-camel