Spring batch - running multiple jobs in parallel

Question

I am new to Spring batch and couldn\'t figure out how to do this..

Basically I have a spring file poller which runs every N mins to look for files with some name (ex

Answer 1

I believe that you can. Since you are new in spring batch (just like me) I would recommend that you go through the domain language of a batch if you haven't done so already.

Then you may start by configuring your own asynchronous JobLauncher. For example:

  @Bean
  public JobLauncher jobLauncher() throws Exception
  {
    SimpleJobLauncher jobLauncher = new SimpleJobLauncher();

    jobLauncher.setJobRepository(jobRepository);
    jobLauncher.setTaskExecutor(new SimpleAsyncTaskExecutor());
    jobLauncher.afterPropertiesSet();

    return jobLauncher;
  }

Pay special attention to SimpleAsyncTaskExecutor (the job repo can be autowired). This configuration will allow asynchronous execution as visualized next:

Compare it with the synchronous execution flow:

Maybe it would additionally help to quote the SimpleJobLauncher java doc:

Simple implementation of the JobLauncher interface. The Spring Core TaskExecutor interface is used to launch a Job. This means that the type of executor set is very important. If a SyncTaskExecutor is used, then the job will be processed within the same thread that called the launcher. Care should be taken to ensure any users of this class understand fully whether or not the implementation of TaskExecutor used will start tasks synchronously or asynchronously. The default setting uses a synchronous task executor.

More details and configuration options - here.

At the end just create the jobs with different names and/or launch them with different parameter set. Naive example would be:

  @Autowired
  public JobBuilderFactory jobBuilderFactory;

  public Job createJobA() {
    return jobBuilderFactory.get("A.txt")
                            .incrementer(new RunIdIncrementer())
                            .flow(step1())
                            .next(step2())
                            .end()
                            .build();
  }

  public Job createJobB() {
    return jobBuilderFactory.get("B.txt")
                            .incrementer(new RunIdIncrementer())
                            .flow(step1())
                            .next(step2())
                            .end()
                            .build();
  }

Executing these jobs with your asynchronous job launcher will create two job instances that which will execute in parallel. This is just one option, that may or may not be suitable for your context.