How to run periodic tasks in an Apache Storm topology?

问题

I have an Apache Storm topology and would like to perform a certain action every once in a while. I'm not sure how to approach this in a way which would be natural and elegant.

Should it be a Bolt or a Spout using ScheduledExecutorService, or something else?

回答1:

Tick tuples are a decent option https://kitmenke.com/blog/2014/08/04/tick-tuples-within-storm/

Edit: Here's the essential code for your bolt

@Override
public Map<String, Object> getComponentConfiguration() {
    // configure how often a tick tuple will be sent to our bolt
    Config conf = new Config();
    conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 300);
    return conf;
}

Then you can use TupleUtils.isTick(tuple) in execute to check whether the received tuple is a tick tuple.

回答2:

I don't know if this is a correct approach, but it seems to be working fine: At the end of the prepare method of a Bolt, I added a call to intiScheduler(), which contains the following code:

Calendar calendar = Calendar.getInstance();
ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
        scheduler.scheduleAtFixedRate(new PeriodicAction() [class implementing Runnable], millisToFullHour(calendar) [wanna start at the top of the hour], 60*60*1000 [run every hour], TimeUnit.MILLISECONDS);

This needs to be used with caution though, because the bolt can have multiple instances depending on your setup.

来源：https://stackoverflow.com/questions/51072739/how-to-run-periodic-tasks-in-an-apache-storm-topology

标签

scheduled-tasks

apache-storm