Using Apache Camel ProducerTemplate in Apache Storm bolt

问题

I'm trying to write simple Storm + Camel project. My Storm topology analyzes tweets and one bolt should send tweet text to apache camel route, which in turn is using websocket to notify some webapp.

I cannot make it work due to NotSerializableExceptions received from bolts when trying to use build once CamelContext.

What I've already tried:

pass CamelContext in bolt's constructor - results in NotSerializableException
pass CamelContext in storm conf, and use it in bolt's prepare(...) method to gian access to it. Results in :

14484 [main] ERROR org.apache.storm.zookeeper.server.NIOServerCnxnFactory - Thread Thread[main,5,main] died java.lang.IllegalArgumentException: Topology conf is not json-serializable at backtype.storm.testing$submit_local_topology.invoke(testing.clj:262) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:43) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.LocalCluster.submitTopology(Unknown Source) ~[storm-core-0.9.4.jar:0.9.4]

Camel route:

public class MyRouteBuilder extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("direct:main")
                .to("websocket:localhost:8085/main?sendToAll=true");
    }
}

Storm Topology: Tweet Spout is spreading tweets using twitter4j stremaing API.

public class TwitterStreamTopology {

    public static void main(String[] args) {
        CamelContext producerTemplate = new RouteStarter().buildRoute();

        TopologyBuilder builder = new TopologyBuilder();
        builder.setSpout("tweetSpout", new TweetSpout(keywords), 1);
        builder.setBolt("websocket", new WebSocketBolt()).shuffleGrouping("tweetSpout");
        Config conf = new Config();
        conf.put("producerTemplate", producerTemplate.createProducerTemplate());
        conf.setDebug(true);

        LocalCluster cluster = new LocalCluster();
        cluster.submitTopology("mytopology", conf, builder.createTopology());

        Utils.sleep(20000);
        cluster.shutdown();
    }
}

WebsocketBolt:

public class WebSocketBolt extends BaseBasicBolt {
    private ProducerTemplate producerTemplate;

    @Override
    public void execute(Tuple input, BasicOutputCollector basicOutputCollector) {
        Status s = (Status) input.getValueByField("tweet");
        producerTemplate.sendBody("direct:main", s.getText());
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {

    }

    @Override
    public void prepare(Map stormConf, TopologyContext context) {
        super.prepare(stormConf, context);
        this.producerTemplate = (ProducerTemplate) stormConf.get("producerTemplate");
    }
}

Is there a way to do this nicely?

Or should I make camel route being accessed by http, and create some HttpClient in bolt prepare(...) method? This still looks like a little overkill, and there has to be a way to make it easier.

Thanks for all help!

回答1:

The root cause of your problem is that you're adding ProducerTemplate to your storm config and it is throwing an exception because it isn't serializable. If that were your own class, you could change the code to make it work but since that is a Camel class I would recommend a different approach.

WebSocketBolt: Change your producerTemplate private member to be transient: private transient ProducerTemplate producerTemplate; so that it will not attempt to be serialized (same problem you have with putting it into conf).
WebSocketBolt: Initialize producerTemplate inside your prepare method rather than in your topology.

Something like this:

public class WebSocketBolt extends BaseBasicBolt {
    private transient ProducerTemplate producerTemplate;

    @Override
    public void execute(Tuple input, BasicOutputCollector basicOutputCollector) {
        Status s = (Status) input.getValueByField("tweet");
        producerTemplate.sendBody("direct:main", s.getText());
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {

    }

    @Override
    public void prepare(Map stormConf, TopologyContext context) {
        super.prepare(stormConf, context);
        CamelContext producerTemplate = new RouteStarter().buildRoute();
        this.producerTemplate = producerTemplate.createProducerTemplate();
    }
}

来源：https://stackoverflow.com/questions/29913304/using-apache-camel-producertemplate-in-apache-storm-bolt

标签

java

apache-camel

apache-storm