Spring Batch: Writing data to multiple files with dynamic File Name

Question

I have a requirement to get the data from a database and write that data to files based on the filename given in the database.

This is how data is defined in the da

Answer 1

In spring-batch, you can do this using ClassifierCompositeItemWriter.

Since ClassifierCompositeItemWriter gives you access to your object during write, you can write custom logic to instruct spring to write to different files.

Take a look at below sample. The ClassifierCompositeItemWriter needs an implementation of Classifier interface. Below you can see that I have created a lambda where I am implementing the classify() method of the Classifier interface. The classify() method is where you will create your ItemWriter. In our example below, we have created a FlatFileItemWriter which gets the name of the file from the item itself and then creates a resource for that.

@Bean
public ClassifierCompositeItemWriter yourDataObjectItemWriter(
    Classifier> itemWriterClassifier
) {
  ClassifierCompositeItemWriter compositeItemWriter = new ClassifierCompositeItemWriter<>();
  compositeItemWriter.setClassifier(itemWriterClassifier);
  return compositeItemWriter;
}

@Bean
public Classifier> itemWriterClassifier() {
  return yourDataObject -> {
    String fileName = yourDataObject.getFileName();

    BeanWrapperFieldExtractor fieldExtractor = new BeanWrapperFieldExtractor<>();
    fieldExtractor.setNames(new String[]{"recId", "name"});
    DelimitedLineAggregator lineAggregator = new DelimitedLineAggregator<>();
    lineAggregator.setFieldExtractor(fieldExtractor);

    FlatFileItemWriter itemWriter = new FlatFileItemWriter<>();
    itemWriter.setResource(new FileSystemResource(fileName));
    itemWriter.setAppendAllowed(true);
    itemWriter.setLineAggregator(lineAggregator);
    itemWriter.setHeaderCallback(writer -> writer.write("REC_ID,NAME"));

    itemWriter.open(new ExecutionContext());
    return itemWriter;
  };
}

Finally, you can attach your ClassifierCompositeItemWriter in your batch step like you normally attach your ItemWriter.

@Bean
public Step myCustomStep(
    StepBuilderFactory stepBuilderFactory
) {
  return stepBuilderFactory.get("myCustomStep")
      .chunk(1000)
      .reader(myCustomReader())
      .writer(yourDataObjectItemWriter(itemWriterClassifier(null)))
      .build();
}

NOTE: As pointed out in comments by @Ping, a new writer will be created for each chunk, which is usually a bad practice and not an optimal solution. A better solution would be to maintain a hashmap of filename and writer so that you can reuse the writer.