Spring MongoItemReader not reading all records on single execution

问题

@Bean
public Job orderJob() throws Exception {
    return jobBuilderFactory.get("orderJob").incrementer(new RunIdIncrementer()).listener(listener())
            .flow(orderStep()).end().build();
}

@Bean
public Step orderStep() throws Exception {
    return stepBuilderFactory.get("orderStep").<OrderCollection, Order>chunk(1000)
            .reader(orderReader()).processor(orderProcessor()).writer(orderWriter())
            .allowStartIfComplete(true).build();

}
@Bean
@StepScope
public MongoItemReader<OrderCollection> orderReader() throws Exception {
    MongoItemReader<OrderCollection> reader = new MongoItemReader<>();
    reader.setTemplate(mongoTemplate);
    reader.setCollection("order");
    Map<String, Sort.Direction> sort = new HashMap<>();
    sort.put("_id", Sort.Direction.ASC);
    reader.setSort(sort);
    reader.setTargetType(OrderCollection.class);
    reader.setQuery("{$or: [ {flag:false}, {flag:null} ]}");
    return reader;
}
@Bean
@StepScope
public OrderProcessor orderProcessor() {
    return new OrderProcessor();
}

@Bean
@StepScope
public ItemWriter<Order> orderWriter() {
    return new OrderWriter();
}

There are 5686 records in order Collection and for all records the flag if false .But the reader reads and process only 3000 records in first run. 1686 records in second run , and 1000 records in third run. There is no error FYI

回答1:

I'm guessing that you're probably updating the collection your reading from and your also updating a field that the query is using. If so then I had the same problem recently.

The MongoItemReader is a paged reader. So each time the writer updates those records the reader has a smaller pool but the page is still increasing.

So imagine that we have 20 items and read 5 items at a time:

1) Reads Items 1-5 from a total of 20.

2) Updates Items 1-5 and now there's a total of 15 possible items

3) Reads items 6-10 from a total 15.

4) Updates items 6-10 and now there's a total of 10 possible items.

5) Reads items 11-15 of 10 possible items

6) Read returns null because there's nothing returned for that page.

So now you've only processed half.

I followed the tutorial below to create a MongoDbCursorItemReader which solved this problem for me: https://blog.zenika.com/2012/05/23/spring-batch-and-mongodb-cursor-based-item-reader/

来源：https://stackoverflow.com/questions/45605056/spring-mongoitemreader-not-reading-all-records-on-single-execution

标签

mongodb

spring-batch