Spring Batch: ItemReader with Scrollable Resultset

问题

I have batch job in which I read more than 1 million records from database and I am accessing those records using Scrollable Resultset. Now i am converting that job to spring batch. Scrollable Resultset won't work in this situtation. I have tried but after reading records in first chunk Resultset closes and when batch tries to access it in next step it throws exception: "can not operate on close result set".

I am new to spring batch. Can any body please help me on how can i implement Scrollable Resultset logic in reader. As 1M records in memory is not such a good idea.

Regards,

回答1:

one possible solution is to look at a Continuable Tasklet. Tasklets can 'loop' as managed by Spring Batch, allowing for you to retrieve a record using ResultSet.next() and process in a single access without the framework closing the stream.

package de.incompleteco.spring.batch.step.tasklet;

import java.sql.ResultSet;
import java.sql.SQLException;

import javax.annotation.Resource;
import javax.sql.DataSource;

import org.springframework.batch.core.StepContribution;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.repeat.RepeatStatus;

public class ScrollableResultsetTasklet implements Tasklet {

    private boolean open = false;

    @Resource
    private DataSource dataSource;

    private String sql = "select * from test_table";

    private ResultSet rs;

    @Override
    public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
        if (!open) {
            //open the resultset
            rs = open();
            open = true;//set to open
        }//end if
        //move
        rs.next();
        if (!rs.isAfterLast()) {
            //show
            System.out.println(rs.getInt(1));
            return RepeatStatus.CONTINUABLE;
        }//end if
        //done
        return RepeatStatus.FINISHED;
    }

    protected ResultSet open() throws SQLException {
        return dataSource.getConnection().createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY).executeQuery(sql);
    }


}

this, of course, is not 'restartable' but then neither is a scrollable resultset (without some additional work).

in the case of reading from a chunk, you may look at 'externalizing' the resultset manager to a separate pogo where it will maintain it's own 'state' between the batch chunks. this may look something like this;

package de.incompleteco.spring.batch.service;

import java.sql.ResultSet;
import java.sql.SQLException;

import javax.annotation.Resource;
import javax.sql.DataSource;

public class ScrollableResultSetService {

    private boolean open = false;

    @Resource
    private DataSource dataSource;

    private String sql = "select * from test_table";

    private ResultSet rs;

    /**
     * retrieve the result set in a 'next' state
     * @return
     * @throws SQLException
     */
    public ResultSet getNext() throws SQLException {
        if (rs == null || !open) {
            //open the resulset
            rs =  dataSource.getConnection().createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE,ResultSet.CONCUR_READ_ONLY).executeQuery(sql);
        }//end if
        //move
        rs.next();
        //test
        if (rs.isAfterLast()) {
            return null;
        }//end if
        return rs;
    }

}

and the itemreader then looks a bit like this;

package de.incompleteco.spring.batch.step.item;

import java.sql.ResultSet;

import javax.annotation.Resource;

import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.NonTransientResourceException;
import org.springframework.batch.item.ParseException;
import org.springframework.batch.item.UnexpectedInputException;

import de.incompleteco.spring.batch.service.ScrollableResultSetService;

public class ScrollableResultSetItemReader implements ItemReader<T> {

    @Resource
    private ScrollableResultSetService service;

    @Override
    public T read() throws Exception, UnexpectedInputException, ParseException,NonTransientResourceException {
        ResultSet rs = service.getNext();
        if (rs == null) {
            return null;//don't continue
        }//end if
        //process the result set into your object
        //...
        //return object 
        return T;
    }

}

回答2:

You can use the out of the box JdbcPagingItemReader or JdbcCursorItemReader. Try them both and see which one fit the best for you.

Your Implementation will be the same with both.. only config change!!! ;-)

Both of them let your read very large number of record without putting them all in memory.

1M records is nothing with Spring Batch... if you use the right reader ;-)

EDIT : based on your comments:

Can you provide a code example of your existing interface/Impl?

Normally, when you want to use an existing service/Dao, you can use the ItemReaderAdapter. this will let you define what object and what method to delegate the read(). But the out-of-the box implementation is very basic and will call your custom service.find() on startup (InitializingBean). So you will end up with 1M object in a List in memory!

Because your find method probably run the query and map the resultSet to a List

I would suggest you move your SQL from your implementation class to a JdbcCursorItemreader and make uses of stepListener to verify the jobParameters and set the proper Sql inside the JdbcCursorItemreader.

Regards

来源：https://stackoverflow.com/questions/16169615/spring-batch-itemreader-with-scrollable-resultset

标签

spring-batch