Spring batch execute dynamically generated steps in a tasklet

Question

I have a spring batch job that does the following...

Step 1. Creates a list of objects that need to be processed

Step 2. Creates a list of steps depending on how many items are in the list of objects created in step 1.

Step 3. Tries to executes the steps from the list of steps created in step 2.

The executing x steps is done below in executeDynamicStepsTasklet(). While the code runs without any errors it does not seem to be doing anything. Does what I have in that method look correct?

thanks

/* * */

@Configuration
public class ExportMasterListCsvJobConfig {

public static final String JOB_NAME = "exportMasterListCsv";
@Autowired
public JobBuilderFactory jobBuilderFactory;

@Autowired
public StepBuilderFactory stepBuilderFactory;

@Value("${exportMasterListCsv.generateMasterListRows.chunkSize}") 
public int chunkSize;

@Value("${exportMasterListCsv.generateMasterListRows.masterListSql}") 
public String masterListSql;

@Autowired
public DataSource onlineStagingDb;

@Value("${out.dir}") 
public String outDir;

@Value("${exportMasterListCsv.generatePromoStartDateEndDateGroupings.promoStartDateEndDateSql}") 
private String promoStartDateEndDateSql;


private List<DivisionIdPromoCompStartDtEndDtGrouping> divisionIdPromoCompStartDtEndDtGrouping;

private List<Step> dynamicSteps = Collections.synchronizedList(new ArrayList<Step>()) ;


@Bean
public Job exportMasterListCsvJob(
        @Qualifier("createJobDatesStep") Step createJobDatesStep,
        @Qualifier("createDynamicStepsStep") Step createDynamicStepsStep,
        @Qualifier("executeDynamicStepsStep") Step executeDynamicStepsStep) {

    return jobBuilderFactory.get(JOB_NAME)
            .flow(createJobDatesStep)
            .next(createDynamicStepsStep)
            .next(executeDynamicStepsStep)
            .end().build();
}   


@Bean
public Step executeDynamicStepsStep(
        @Qualifier("executeDynamicStepsTasklet")  Tasklet executeDynamicStepsTasklet) {

    return  stepBuilderFactory
                .get("executeDynamicStepsStep")
                .tasklet(executeDynamicStepsTasklet)
                .build();               
}

@Bean
public Tasklet executeDynamicStepsTasklet() {

    return new Tasklet() {

        @Override
        public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
            FlowStep flowStep = new FlowStep(createParallelFlow());
            SimpleJobBuilder jobBuilder = jobBuilderFactory.get("myNewJob").start(flowStep);
            return RepeatStatus.FINISHED;
        }
    };
}

public Flow createParallelFlow() {
    SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
    taskExecutor.setConcurrencyLimit(1); 

    List<Flow> flows = dynamicSteps.stream()
            .map(step -> new FlowBuilder<Flow>("flow_" + step.getName()).start(step).build())
            .collect(Collectors.toList());

    return new FlowBuilder<SimpleFlow>("parallelStepsFlow")
            .split(taskExecutor)
            .add(flows.toArray(new Flow[flows.size()]))
            .build();
}

@Bean

public Step createDynamicStepsStep(
        @Qualifier("createDynamicStepsTasklet")  Tasklet createDynamicStepsTasklet) {

    return  stepBuilderFactory
                .get("createDynamicStepsStep")
                .tasklet(createDynamicStepsTasklet)
                .build();               
}   

@Bean
@JobScope
public Tasklet createDynamicStepsTasklet() {

    return new Tasklet() {

        @Override
        public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
            for (DivisionIdPromoCompStartDtEndDtGrouping grp: divisionIdPromoCompStartDtEndDtGrouping){

                System.err.println("grp: " + grp);

                String stepName = "stp_" + grp;

                String fileName = grp + FlatFileConstants.EXTENSION_CSV;

                Step dynamicStep = 
                        stepBuilderFactory.get(stepName)
                        .<MasterList,MasterList> chunk(10)
                        .reader(queryStagingDbReader(
                                grp.getDivisionId(), 
                                grp.getRpmPromoCompDetailStartDate(), 
                                grp.getRpmPromoCompDetailEndDate()))
                        .writer(masterListFileWriter(fileName))                                
                        .build(); 

                dynamicSteps.add(dynamicStep);

            } 
            System.err.println("createDynamicStepsTasklet dynamicSteps: " + dynamicSteps);
            return RepeatStatus.FINISHED;
        }
    };
}


public FlatFileItemWriter<MasterList> masterListFileWriter(String fileName) {
    FlatFileItemWriter<MasterList> writer = new FlatFileItemWriter<>();
    writer.setResource(new FileSystemResource(new File(outDir, fileName )));
    writer.setHeaderCallback(masterListFlatFileHeaderCallback());
    writer.setLineAggregator(masterListFormatterLineAggregator());
    return writer;
}

So now I have a list of dynamic steps that need to be executed and I believe that they are in StepScope. Can someone advise me on how to execute them

Answer 1

This will not work. Your Tasklet just creates a job with a FlowStep as first Step. Using the jobBuilderfactory just creates the job. it does not launch it. The methodname "start" may be misleading, since this only defines the first step. But it does not launch the job.

You cannot change the structure of a job (its steps and substeps) once it is started. Therefore, it is not possible to configure a flowstep in step 2 based on things that are calculated in step 1. (of course you could do some hacking deeper inside the springbatch structure and directly modify the beans and so ... but you don't want to do that).

I suggest, that you use a kind of "SetupBean" with an appropriate postConstruct method which is injected into your class that configures your job. This "SetupBean" is responsible to calculate the list of objects being processed.

@Component
public class SetUpBean {

  private List<Object> myObjects;

  @PostConstruct
  public afterPropertiesSet() {
    myObjects = ...;
  }

  public List<Object> getMyObjects() {
   return myObjects;
  }
}

@Configuration
public class JobConfiguration {

   @Autowired
   private JobBuilderFactory jobBuilderFactory;

   @Autowired
   private StepBuilderFactory stepBuilderFactory;

   @Autowired
   private SetUpBean setup;

   ... 
}