Spring batch Multithreaded processing for Single file to Multiple FIle

前端 未结 3 1131
一个人的身影
一个人的身影 2021-01-19 19:38

My problem statement. Read a csv file with 10 million data and store it in db. with as minimal time as possible.

I had implemented it using Simple multi threaded

3条回答
  •  自闭症患者
    2021-01-19 20:20

    1. About multi-thread read How to set up multi-threading in Spring Batch? answer; it will point you to right direction. Also, in this sample there are some consideration about restart for CSV file
    2. Job should automatically fails if some error on thread: I have never tried, but this should be the default behaviour
    3. Spring Batch How to set time interval between each call in a Chunk tasklet can be a start. Also, official doc about Backoff Policies - When retrying after a transient failure it often helps to wait a bit before trying again, because usually the failure is caused by some problem that will only be resolved by waiting. If a RetryCallback fails, the RetryTemplate can pause execution according to the BackoffPolicy in place.

    Let me known if this help or how you solve problem because I'm interested for my (future) work!
    I hope my indications can be helpful.

提交回复
热议问题