How to effectively and correctly load sequential activities to Database using Spring batch?

问题

I'm currently working on a project loading a .dat file info to database. However, this .dat file contains not only data but also actions. The first field indicates the action of the records and all else are just data. Below are some examples records: A key1 key2 data1 data2 D key1 key2 data1 data2 C key1 key2 data1 data2

let, A=add, D=delete, C=update The file size is roughly 5GB. In this case, order of the records to be process does matter. Is it possible to use Spring batch to batch process this? A previous implementation done by my predecessor was to create 3 lists: addList, deleteList, updateList, and generate dml statements for them separately. Although very efficient when loading, but it doesn't work because doesn't matter which order you execute these lists, it might cause data inconsistency. So to keep the data valid, I believe these records have to be executed in order.

One solution I came up with is to set reader fetch-size to 1. But that seems to defeat the purpose of spring batch.

Are there better ways to do this? perhaps not spring batch but something else?

回答1:

You should be able to achieve this (while preserving records order) using a combination of:

• PatternMatchingCompositeLineMapper: to map items according to your patterns (A, D and C)
• ClassifierCompositeItemWriter: configured with a PatternMatchingClassifier. This will classify items according to their type and use the corresponding writer (you should have one writer per type) to do the action.

回答2:

Have you investigated PatternMatchingCompositeLineTokenizer https://docs.spring.io/spring-batch/trunk/apidocs/org/springframework/batch/item/file/mapping/PatternMatchingCompositeLineMapper.html

来源：https://stackoverflow.com/questions/51736320/how-to-effectively-and-correctly-load-sequential-activities-to-database-using-sp