Spring Batch - How to prevent batch from storing transactions in DB

Question

First the problem statement: I am using Spring-Batch in my DEV environment fine. When I move the code to a production environment I am running into a problem. In my DEV environment, Spring-Batch is able to create it's transaction data tables in our DB2 database server with out problem. This is not a option when we go to PROD as this is a read only job.

Attempted solution:

Search Stack Overflow I found this posting: Spring-Batch without persisting metadata to database?

Which sounded perfect, so I added

@Bean
public ResourcelessTransactionManager transactionManager() {
    return new ResourcelessTransactionManager();
}

@Bean
public JobRepository jobRepository(ResourcelessTransactionManager transactionManager) throws Exception {
    MapJobRepositoryFactoryBean mapJobRepositoryFactoryBean = new MapJobRepositoryFactoryBean(transactionManager);
    mapJobRepositoryFactoryBean.setTransactionManager(transactionManager);

    return mapJobRepositoryFactoryBean.getObject();
}

I also added it to my Job by calling .reporitory(jobRepository).

But I get

Caused by: java.lang.NullPointerException: null
    at       org.springframework.batch.core.repository.dao.MapJobExecutionDao.synchronizeStatus(MapJobExecutionDao.java:158) ~[spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]

So I am not sure what to do here. I am new to Spring so I am teaching myself as I go. I am open to other solutions, such as an in memory database, but I have not been able to get them to work either. I do NOT need to save any state or session information between runs, but the data base query I am running will return around a million or so rows, so I will need to get that in chunks.

Any suggestions or help would be greatly appreciated.

Answer 1

Add this beans to AppClass

@Bean
public PlatformTransactionManager transactionManager() {
    return new ResourcelessTransactionManager();
}


@Bean
public JobExplorer jobExplorer() throws Exception {
    MapJobExplorerFactoryBean jobExplorerFactory = new MapJobExplorerFactoryBean(mapJobRepositoryFactoryBean());
    jobExplorerFactory.afterPropertiesSet();
    return jobExplorerFactory.getObject();
}

@Bean
public MapJobRepositoryFactoryBean mapJobRepositoryFactoryBean() {
    MapJobRepositoryFactoryBean mapJobRepositoryFactoryBean = new MapJobRepositoryFactoryBean();
    mapJobRepositoryFactoryBean.setTransactionManager(transactionManager());
    return mapJobRepositoryFactoryBean;
}

@Bean
public JobRepository jobRepository() throws Exception {
    return mapJobRepositoryFactoryBean().getObject();
}

@Bean
public JobLauncher jobLauncher() throws Exception {
    SimpleJobLauncher simpleJobLauncher = new SimpleJobLauncher();
    simpleJobLauncher.setJobRepository(jobRepository());
    return simpleJobLauncher;
}

Answer 2

This doesn't directly answer your question, but that is not a good solution; the map-based repository is supposed to be used only for testing. It will grow in memory indefinitely.

I suggest you use an embedded database like sqlite. The main problem in using a separate database for job metadata is that you should then coordinate the transactions between the two databases that you use (so that the state of metadata matches that of the data), but since it seems you're not even writing in the main database, that probably won't be a problem for you.

Answer 3

You could use an in-memory database (for example H2 or HSQL) quite easily. Examples of that you can find for example here: http://www.mkyong.com/spring/spring-embedded-database-examples/.

As for the Map-backed job repository, it does provide a method to clear its contents:

public void clear()

Convenience method to clear all the map DAOs globally, removing all entities.

Be aware that a Map-based job repository is not fit for use in partitioned steps and other multi-threading.

Answer 4

The following seems to have done the job for me:

@Bean
public DataSource dataSource() {        

    EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
    EmbeddedDatabase db = builder
        .setType(EmbeddedDatabaseType.HSQL) 
        .build();
    return db;
}

Now Spring is not creating tables in our production database, and when the JVM exits state is lost so nothing seems to be hanging around.

UPDATE: The above code has caused concurrency errors for us. We have addressed this by abandoning the EmbeddedDatabaseBuilder and declaring the HSQLDB this way instead:

@Bean
    public BasicDataSource dataSource() {
        BasicDataSource dataSource = new BasicDataSource();
          dataSource.setDriverClassName("org.hsqldb.jdbcDriver");
          dataSource.setUrl("jdbc:hsqldb:mem:testdb;sql.enforce_strict_size=true;hsqldb.tx=mvcc");
          dataSource.setUsername("sa");
          dataSource.setPassword("");
        return dataSource;
    }   

The primary difference is that we are able to specify mvcc (Multiversion concurrency control) in connection string which resolves the issue.