Spring Batch - JdbcCursorItemReader throwing OutOfMemoryError with large MySQL table

问题

I am writing a program using Spring Batch to process 7,637,064 rows from a MySQL database table. I've had success with smaller tables, but the large number of rows in this table is causing OutOfMemoryError exceptions when the JdbcCursorItemReader attempts to open the cursor.

I could probably resolve this by throwing a larger Xmx at it, but it seems to me that Spring Batch should have a way to handle this and that I may simply be missing a key piece of configuration.

Spring Batch configuration:

  <job id="reportJob" xmlns="http://www.springframework.org/schema/batch">
    <step id="largeTableTransfer">
      <tasklet>
        <chunk reader="largeTableReader" processor="largeTableTransformer" writer="largeTableWriter"
          commit-interval="10" />
      </tasklet>
    </step>
  </job>

  <bean id="largeTableReader" class="org.springframework.batch.item.database.JdbcCursorItemReader">
    <property name="dataSource" ref="inputDataSource" />
    <property name="sql" value="select * from largeTable" />
    <property name="rowMapper">
      <bean class="myproject.reader.largeTableRowMapper" />
    </property>
  </bean>
  <bean id="largeTableTransformer" class="myproject.transformer.LargeTableTransformer" />
  <bean id="largeTableWriter" class="myproject.writer.JdbcLargeTableWriter">
    <property name="dataSource" ref="outputDataSource" />
  </bean>

Setting fetchSize on the JdbcCursorItemReader appears to have no effect. The only thing that allows it to run to completion is to set the maxRows to a small number, but then only that number of rows get processed.

The relevant stack trace:

2012-11-21 11:25:29,931 DEBUG [org.springframework.batch.core.repository.dao.JdbcStepExecutionDao] - <Truncating long message before update of StepExecution, original message is: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2734)
    at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
    at java.util.ArrayList.add(ArrayList.java:351)
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2821)
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1885)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
    at org.springframework.batch.item.database.JdbcCursorItemReader.openCursor(JdbcCursorItemReader.java:125)
    at org.springframework.batch.item.database.AbstractCursorItemReader.doOpen(AbstractCursorItemReader.java:401)
    at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:134)
    at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:93)
    at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:301)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:192)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:293)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:114)
    at ifpress.ams2amx.ExampleJobConfigurationTests.testLaunchJob(ExampleJobConfigurationTests.java:32)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

回答1:

There is an issue in MySql JDBC driver which causes loading of the whole dataset into memory regardless of the parameters you have passed to the statement creation method. See http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-implementation-notes.html for how properly open a cursor.

Here is how I do it:

<bean class="org.springframework.batch.item.database.JdbcCursorItemReader">
  <property name="verifyCursorPosition" value="false" />
   <property name="dataSource" ref="remoteDataSource" />
   <property name="rowMapper">
     <bean class="org.springframework.jdbc.core.SingleColumnRowMapper" />
   </property>
   <property name="fetchSize">
     <util:constant static-field="java.lang.Integer.MIN_VALUE" />
   </property>
   <property name="sql">
     <value>SELECT foo, bar FROM baz</value>
  </property>
</bean>

来源：https://stackoverflow.com/questions/13497822/spring-batch-jdbccursoritemreader-throwing-outofmemoryerror-with-large-mysql-t