Routing data to multiple files in item writer based on item's property as criteria

问题

I am getting a list of items in my reader.

There is a property called Code in each item object having several possible values not known to me before hand.

1) Based on the value of Codein each item, I want to write that particular item in a output file pertaining to that Code. For e.g. if my current item's Code is "abc", the item should be written in to abc.txt in the writer.

2) If there is a Code "xyz" in current item, for which the file is not present, a new file should get created and the item should go to that file.

3) For all such multiple files created based on Code, I also want to add a header and footer call back to enter some details e.g. count of items in each file.

Is it possible to have a writer, which satisfies above 3 requirements ?

I know that using multiresourceitemwriter, one can divide the data among multiple output files. But as far as I know, this division is based on the number of items. For e.g. first 10 items in file1, next 10 in file2 and so on.

But how to route data to output files based on an item property as mentioned in my question ?

I am well acquainted with Spring Batch and just need a little guidance since this is the first time I am facing this kind of issue.

Thanks for reading!

回答1:

i would try it with at least 2 strategies

the batch writes all data into a temporary database table and simple tool/batch/script creates the individual files - i'm not sure about header/footer but as always one could say "let's keep it cheap"
the itemWriter creates and manages the needed writers on the fly, maybe with an abstract pre-configured writer/bean as template, as long as you ignore restart scenarios it sounds "easy"

回答2:

If I understand your problem correctly, you need a few items.

First, a classifier that implements the Classifier interface

public class ItemCodeClassifier {
    @Classifier
    public String classify(Item item) {
        return item.getCode().getKey();// returns "abc", "xyz"
    }
}

Second a router implementation that consumes the above method

<bean id="classifier"  class="org.springframework.batch.classify.BackToBackPatternClassifier">
    <property name="routerDelegate">
        <bean class="ItemCodeClassifier" />
    </property>
    <property name="matcherMap">
        <map>
        <entry key="abc" value-ref="abcItemWriter" />
        <entry key="xyz" value-ref="xyzItemWriter" />
        </map>
    </property>
</bean>

And last of all, a ClassifierCompositeItemWriter

<bean id="ItemWriter" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter">
<property name="classifier" ref="classifier" />
</bean

Did not compile the above but hope that it helps.

回答3:

Below solution worked for me. Compiled the Code and it's working fine.

First, you will need a Classifier. Either implement the Classifier interface or annotate classify() method with @Classifier.

Here I have used annotation.

public class MyClassifier {
    @Classifier
    public String classify(Policy pol) {
        return pol.getPolCode;// returns "01", "02"
    }
}

Add a Classifier bean

<bean id="myClassifier"  class="org.springframework.batch.classify.BackToBackPatternClassifier">
    <property name="routerDelegate">
        <bean class="MyClassifier" />
    </property>
    <property name="matcherMap">
        <map>
        <entry key="01" value-ref="pol01ItemWriter" />
        <entry key="02" value-ref="pol02ItemWriter" />
        </map>
    </property>
</bean>

Add your writer bean as like below

<bean id="ItemWriter" class="org.springframework.batch.item.support.ClassifierCompositeItemWriter">
    <property name="myClassifier" ref="myClassifier" />
</bean>

The Above Code may throw WriterNotOpenException. For this you need to add batch:stream to the step.

<batch:step id="step1">
    <batch:tasklet>
        <batch:chunk reader="reader" processor="processor" writer="ItemWriter" commit-interval="3">
            <batch:streams>
                <batch:stream ref="pol01ItemWriter"/>
                <batch:stream ref="pol02ItemWriter"/>
            </batch:streams>
        </batch:chunk>
    </batch:tasklet>
</batch:step>

回答4:

Another option would be to build 3 lists containing the elements separated by their Code. Then, use a different step to write these lists to the filesystem.

One advantage is your elements are ready to be written, thus you can use a large buffer to increase your throughput writing (according to your hardware of course).

If the number of file you have to generate is dynamic, I suggest you to write them sequentially to avoid any multithreading issue.

You can't route elements dynamically. So the idea is to create a list of elements routed by yourself, and then work on these lists.

回答5:

Java Config for JDBC writer

This is how i have done it for jdbc writer. We Can have similar configuration for flatfile Writer

 @Bean
    public ItemWriter<Person> itemWriter(DataSource dataSource) {
        BackToBackPatternClassifier classifier = new BackToBackPatternClassifier();
        classifier.setRouterDelegate(new AggGroupClassifier());
        classifier.setMatcherMap(new HashMap<String, ItemWriter<? extends Person>>() {
            {
                put("A", writerA(dataSource));
                put("B", writerB(dataSource));
                put("C", writerC(dataSource));

            }
        });
        ClassifierCompositeItemWriter<Person> writer = new ClassifierCompositeItemWriter<Person>();
        writer.setClassifier(classifier);
        return writer;      
        }


public class AggGroupClassifier {

    @Classifier
    public String classify(Person person) {
        return person.getAgeGroup();

    }
}

回答6:

I encountered this same problem this morning. And finally I found that currently ClassifierCompositeItemWriter don't support FlatFileItemWriter as its delegate ItemWriter in Spring batch's latest release 2.1.9 version.

WriterNotOpenException is threw like below:

org.springframework.batch.item.WriterNotOpenException: Writer must be open before it can be written to
    at org.springframework.batch.item.file.FlatFileItemWriter.write(FlatFileItemWriter.java:236)
    at org.springframework.batch.item.support.ClassifierCompositeItemWriter.write(ClassifierCompositeItemWriter.java:65)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:171)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:150)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:269)
    at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:194)
    at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:74)
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:386)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:264)
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:76)
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:367)
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:214)
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:143)
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:250)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:281)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:114)

来源：https://stackoverflow.com/questions/12453254/routing-data-to-multiple-files-in-item-writer-based-on-items-property-as-criter

标签

spring-batch