How to apply partitioned count for MultiResourceItemReader?

问题

I have a file of 50K records. It takes close to 40 mins to insert it into a DB. So I thought of applying a partition to the step in such a way that the 50k records are partitioned between 10 threads (via gridSize) with each thread processing 1000 records in parallel.

All the forums show examples of using JDBCPagingItemReader and partitioned count set via execution context. Since I am using MultiResourceItemReader, how can I set the partition count(startingIndex and endingIndex - refer code snippet below) for MultiResourceItemReader?

Please advise.

Code snippet of partitioner below:

public Map partition(int gridSize) {
    LOGGER.debug("START: Partition");
    Map partitionMap = new HashMap();
    int startingIndex = 0;
    int endingIndex =  1000;

    for(int i=0; i< gridSize; i++){
        ExecutionContext ctxMap = new ExecutionContext();
        ctxMap.putInt("startingIndex",startingIndex);
        ctxMap.putInt("endingIndex", endingIndex);

        startingIndex = endingIndex+1;
        endingIndex += 1000; 

        partitionMap.put("Thread:-"+i, ctxMap);
    }
    LOGGER.debug("END: Created Partitions of size: "+ partitionMap.size());
    return partitionMap;
}

回答1:

You don't set the partition count on the MultiResourceItemReader. You use the MultiResourcePartitioner to create a partition per resource (file) and then have the reader pick up each file separately as it's own partition. With that configuration, you don't need the MultiResourceItemReader anymore as well (you can go straight to the delegate).

There is a sample of this use case in the Spring Batch samples and can be found here: https://github.com/spring-projects/spring-batch/blob/master/spring-batch-samples/src/main/resources/jobs/partitionFileJob.xml

来源：https://stackoverflow.com/questions/34372439/how-to-apply-partitioned-count-for-multiresourceitemreader

标签

multithreading

spring-batch