Making Camel routes run in parallel

问题

I have an application which does some basic ETL using camel routes. Each route is configured to take some data from one table do some transformation and safe it into the same table on a different schema. So there is an one to one relationship between a camel route and a table.

Say I have these two routes:

from("direct:table_1").routeId(table1Route)
    .setBody("SELECT * FROM table_1)
    .to("jdbc:source_schema").split(body()).streaming()
    .process("someProcessor")
    .to("sql:INSERT INTO table_1 ... ?dataSource=target_schema");

from("direct:table_2").routeId(table2Route)
    .setBody("SELECT * FROM table_2)
    .to("jdbc:source_schema").split(body()).streaming()
    .process("someProcessor")
    .to("sql:INSERT INTO table_2 ... ?dataSource=target_schema");

Everything runs OK and the data is moved into target schema when sending a start processing message to both direct:table_1 and direct:table_2 end points.

However looking at the logs I can see table 2 records start being moved only after table 1 records are finished. That is definitely a no no for my application as some tables are quite big and and moving one table at a time would take a very long time to run.

My question is what I am doing wrong and how can I address this so the data movement happens in parallel.

回答1:

I would try something like this:

from("start").multicast().parallelProcessing().to("seda:table1", "seda:table2");

Basically I have:

Used multicast to send to multiple recipients and use parallellprocessing to try to send to both endpoints in parallell.
I have replaced your direct endpoints with seda endpoints. If you don't require synchronous endpoints it can be beneficial to use seda instead.

You can also experiment with .threads() syntax for multithreading.

If you want to compute your table endpoints at runtime you can replace .multicast() with .recipientlist()

回答2:

Alternatively, if using xml, this can be achieved by:

<routeContext id="xxxRoute" xmlns="http://camel.apache.org/schema/spring">
    <route id="xxxRouteId">
        <from uri="activemq:queue:{{xxx.queue}}" />
        <multicast parallelProcessing="true">
            <pipeline>
                <to uri="file://?fileExist=Append"></to>
            </pipeline>
            <pipeline>
                <to uri="sql:{{sql.xxxx.insertQuery}}"></to>
            </pipeline>
        </multicast>
    </route>
</routeContext>

来源：https://stackoverflow.com/questions/40567661/making-camel-routes-run-in-parallel

标签

apache-camel