Spring batch Multithreaded processing for Single file to Multiple FIle

Question

My problem statement. Read a csv file with 10 million data and store it in db. with as minimal time as possible.

I had implemented it using Simple multi threaded

Answer 1

Here is how I solved the problem.

1. Read a file and chunk the file( split the file) using Buffered and File Channel reader and writer ( the fastest way of File read/write, even spring batch uses the same). I implemented such that this is executed before job is started( However it can be executed using job as step using method invoker)

2. Start the Job with directory location as job parameter.

3. Use multiResourcePartitioner which will get the directory location and for each file a slave step is created in separate thread
4. In the Slave step get the file passed from Partitioner and use spring batchs itemreader to read the file
5. Use the Database item writer( I'm using mybatis batch itemwriter) to push the data to Database.
6. Its better to use the split count equal to commit-count of step.