My main job does only read operations and the other one does some writing but on MyISAM engine which ignores transactions, so I wouldn't require necessarily transaction support. How can I configure Spring Batch to have its own datasource for the JobRepository, separate from the one holding the business data? The initial one datasource-configurations is done like the following:
@Configuration
public class StandaloneInfrastructureConfiguration {
@Autowired
Environment env;
@Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() {
LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
em.setDataSource(dataSource());
em.setPackagesToScan(new String[] { "org.podcastpedia.batch.*" });
JpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
em.setJpaVendorAdapter(vendorAdapter);
em.setJpaProperties(additionalJpaProperties());
return em;
}
Properties additionalJpaProperties() {
Properties properties = new Properties();
properties.setProperty("hibernate.hbm2ddl.auto", "none");
properties.setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5Dialect");
properties.setProperty("hibernate.show_sql", "true");
return properties;
}
@Bean
public DataSource dataSource(){
return DataSourceBuilder.create()
.url(env.getProperty("db.url"))
.driverClassName(env.getProperty("db.driver"))
.username(env.getProperty("db.username"))
.password(env.getProperty("db.password"))
.build();
}
@Bean
public PlatformTransactionManager transactionManager(EntityManagerFactory emf){
JpaTransactionManager transactionManager = new JpaTransactionManager();
transactionManager.setEntityManagerFactory(emf);
return transactionManager;
}
}
and then it is imported in the Job's configuration class where the @EnableBatchProcessing annotation automagically makes use of it. My initial thought was to try to set the configuration class extend the DefaultBatchConfigurer, but then I get a
BeanCurrentlyInCreationException ( org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name jobBuilders: Requested bean is currently in creation: Is there an unresolvable circular reference?):
@Configuration
@EnableBatchProcessing
@Import({StandaloneInfrastructureConfiguration.class, NotifySubscribersServicesConfiguration.class})
public class NotifySubscribersJobConfiguration extends DefaultBatchConfigurer {
@Autowired
private JobBuilderFactory jobBuilders;
@Autowired
private StepBuilderFactory stepBuilders;
@Autowired
private DataSource dataSource;
@Autowired
Environment env;
@Override
@Autowired
public void setDataSource(javax.sql.DataSource dataSource) {
super.setDataSource(batchDataSource());
}
private DataSource batchDataSource(){
return DataSourceBuilder.create()
.url(env.getProperty("batchdb.url"))
.driverClassName(env.getProperty("batchdb.driver"))
.username(env.getProperty("batchdb.username"))
.password(env.getProperty("batchdb.password"))
.build();
}
@Bean
public ItemReader<User> notifySubscribersReader(){
JdbcCursorItemReader<User> reader = new JdbcCursorItemReader<User>();
String sql = "select * from users where is_email_subscriber is not null";
reader.setSql(sql);
reader.setDataSource(dataSource);
reader.setRowMapper(rowMapper());
return reader;
}
........
}
Any thoughts are more than welcomed. The project is available on GitHub - https://github.com/podcastpedia/podcastpedia-batch
Thanks a bunch.
Ok, this is strange but it works. Moving the datasources to it's own configuration class works just fine and one is able to autowire.
The example is a multi-datasource version of Spring Batch Service Example:
DataSourceConfiguration:
public class DataSourceConfiguration {
@Value("classpath:schema-mysql.sql")
private Resource schemaScript;
@Bean
@Primary
public DataSource hsqldbDataSource() throws SQLException {
final SimpleDriverDataSource dataSource = new SimpleDriverDataSource();
dataSource.setDriver(new org.hsqldb.jdbcDriver());
dataSource.setUrl("jdbc:hsqldb:mem:mydb");
dataSource.setUsername("sa");
dataSource.setPassword("");
return dataSource;
}
@Bean
public JdbcTemplate jdbcTemplate(final DataSource dataSource) {
return new JdbcTemplate(dataSource);
}
@Bean
public DataSource mysqlDataSource() throws SQLException {
final SimpleDriverDataSource dataSource = new SimpleDriverDataSource();
dataSource.setDriver(new com.mysql.jdbc.Driver());
dataSource.setUrl("jdbc:mysql://localhost/spring_batch_example");
dataSource.setUsername("test");
dataSource.setPassword("test");
DatabasePopulatorUtils.execute(databasePopulator(), dataSource);
return dataSource;
}
@Bean
public JdbcTemplate mysqlJdbcTemplate(@Qualifier("mysqlDataSource") final DataSource dataSource) {
return new JdbcTemplate(dataSource);
}
private DatabasePopulator databasePopulator() {
final ResourceDatabasePopulator populator = new ResourceDatabasePopulator();
populator.addScript(schemaScript);
return populator;
}
}
BatchConfiguration:
@Configuration
@EnableBatchProcessing
@Import({ DataSourceConfiguration.class, MBeanExporterConfig.class })
public class BatchConfiguration {
@Autowired
private JobBuilderFactory jobs;
@Autowired
private StepBuilderFactory steps;
@Bean
public ItemReader<Person> reader() {
final FlatFileItemReader<Person> reader = new FlatFileItemReader<Person>();
reader.setResource(new ClassPathResource("sample-data.csv"));
reader.setLineMapper(new DefaultLineMapper<Person>() {
{
setLineTokenizer(new DelimitedLineTokenizer() {
{
setNames(new String[] { "firstName", "lastName" });
}
});
setFieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {
{
setTargetType(Person.class);
}
});
}
});
return reader;
}
@Bean
public ItemProcessor<Person, Person> processor() {
return new PersonItemProcessor();
}
@Bean
public ItemWriter<Person> writer(@Qualifier("mysqlDataSource") final DataSource dataSource) {
final JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<Person>();
writer.setItemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)");
writer.setDataSource(dataSource);
return writer;
}
@Bean
public Job importUserJob(final Step s1) {
return jobs.get("importUserJob").incrementer(new RunIdIncrementer()).flow(s1).end().build();
}
@Bean
public Step step1(final ItemReader<Person> reader,
final ItemWriter<Person> writer, final ItemProcessor<Person, Person> processor) {
return steps.get("step1")
.<Person, Person> chunk(1)
.reader(reader)
.processor(processor)
.writer(writer)
.build();
}
}
I have my data sources in a separate configuration class. In the batch configuration, we extend DefaultBatchConfigurer and override the setDataSource method, passing in the specific database to use with Spring Batch with a @Qualifier. I was unable to get this to work using the constructor version, but the setter method worked for me.
My Reader, Processor, and Writer's are in their own self contained classes, along with the steps.
This is using Spring Boot 1.1.8 & Spring Batch 3.0.1. Note: We had a different setup for a project using Spring Boot 1.1.5 that did not work the same on the newer version.
package org.sample.config.jdbc;
import javax.sql.DataSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.env.Environment;
import com.atomikos.jdbc.AtomikosDataSourceBean;
import com.mysql.jdbc.jdbc2.optional.MysqlXADataSource;
/**
* The Class DataSourceConfiguration.
*
*/
@Configuration
public class DataSourceConfig {
private final static Logger log = LoggerFactory.getLogger(DataSourceConfig.class);
@Autowired private Environment env;
/**
* Siphon data source.
*
* @return the data source
*/
@Bean(name = "mainDataSource")
@Primary
public DataSource mainDataSource() {
final String user = this.env.getProperty("db.main.username");
final String password = this.env.getProperty("db.main.password");
final String url = this.env.getProperty("db.main.url");
return this.getMysqlXADataSource(url, user, password);
}
/**
* Batch data source.
*
* @return the data source
*/
@Bean(name = "batchDataSource", initMethod = "init", destroyMethod = "close")
public DataSource batchDataSource() {
final String user = this.env.getProperty("db.batch.username");
final String password = this.env.getProperty("db.batch.password");
final String url = this.env.getProperty("db.batch.url");
return this.getAtomikosDataSource("metaDataSource", this.getMysqlXADataSource(url, user, password));
}
/**
* Gets the mysql xa data source.
*
* @param url the url
* @param user the user
* @param password the password
* @return the mysql xa data source
*/
private MysqlXADataSource getMysqlXADataSource(final String url, final String user, final String password) {
final MysqlXADataSource mysql = new MysqlXADataSource();
mysql.setUser(user);
mysql.setPassword(password);
mysql.setUrl(url);
mysql.setPinGlobalTxToPhysicalConnection(true);
return mysql;
}
/**
* Gets the atomikos data source.
*
* @param resourceName the resource name
* @param xaDataSource the xa data source
* @return the atomikos data source
*/
private AtomikosDataSourceBean getAtomikosDataSource(final String resourceName, final MysqlXADataSource xaDataSource) {
final AtomikosDataSourceBean atomikos = new AtomikosDataSourceBean();
atomikos.setUniqueResourceName(resourceName);
atomikos.setXaDataSource(xaDataSource);
atomikos.setMaxLifetime(3600);
atomikos.setMinPoolSize(2);
atomikos.setMaxPoolSize(10);
return atomikos;
}
}
package org.sample.settlement.batch;
import javax.sql.DataSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.DefaultBatchConfigurer;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.transaction.PlatformTransactionManager;
/**
* The Class BatchConfiguration.
*
*/
@Configuration
@EnableBatchProcessing
public class BatchConfiguration extends DefaultBatchConfigurer {
private final static Logger log = LoggerFactory.getLogger(BatchConfiguration.class);
@Autowired private JobBuilderFactory jobs;
@Autowired private StepBuilderFactory steps;
@Autowired private PlatformTransactionManager transactionManager;
@Autowired @Qualifier("processStep") private Step processStep;
/**
* Process payments job.
*
* @return the job
*/
@Bean(name = "processJob")
public Job processJob() {
return this.jobs.get("processJob")
.incrementer(new RunIdIncrementer())
.start(processStep)
.build();
}
@Override
@Autowired
public void setDataSource(@Qualifier("batchDataSource") DataSource batchDataSource) {
super.setDataSource(batchDataSource);
}
}
Per https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-two-datasources:
@Bean
@Primary
@ConfigurationProperties("app.datasource.first")
public DataSourceProperties firstDataSourceProperties() {
return new DataSourceProperties();
}
@Bean
@Primary
@ConfigurationProperties("app.datasource.first")
public DataSource firstDataSource() {
return firstDataSourceProperties().initializeDataSourceBuilder().build();
}
@Bean
@ConfigurationProperties("app.datasource.second")
public DataSourceProperties secondDataSourceProperties() {
return new DataSourceProperties();
}
@Bean
@ConfigurationProperties("app.datasource.second")
public DataSource secondDataSource() {
return secondDataSourceProperties().initializeDataSourceBuilder().build();
}
In the application properties, you can use regular datasource properties:
app.datasource.first.type=com.zaxxer.hikari.HikariDataSource
app.datasource.first.maximum-pool-size=30
app.datasource.second.url=jdbc:mysql://localhost/test
app.datasource.second.username=dbuser
app.datasource.second.password=dbpass
app.datasource.second.max-total=30
Have you tried something like this already?
@Bean(name="batchDataSource")
public DataSource batchDataSource(){
return DataSourceBuilder.create()
.url(env.getProperty("batchdb.url"))
.driverClassName(env.getProperty("batchdb.driver"))
.username(env.getProperty("batchdb.username"))
.password(env.getProperty("batchdb.password"))
.build();
}
and then mark the other datasource with a @Primary, and use an @Qualifier in your batch config to specify that you want to auotwire the batchDataSource bean.
Assuming you have 2 data sources, one for spring batch metadata such as job details[lets say CONFIGDB] and other for your business data [lets say AppDB]:
Inject CONFIGDB into jobRepository, like this:
<bean id="jobRepository"
class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
<property name="transactionManager" ref="transactionManager" />
<property name="dataSource" ref="CONFIGDB" />
<property name="databaseType" value="db2" />
<property name="tablePrefix" value="CONFIGDB.BATCH_" />
</bean>
Now you can inject the AppDB dartasource into your DAO's OR Writers if any like..
<bean id="DemoItemWriter" class="com.demoItemWriter">
<property name="dataSource" ref="AppDB" />
</bean>
OR
you can do define a resource and Inject this AppDB with jndi lookup in the class where its needed like:
public class ExampleDAO {
@Resource(lookup = "java:comp/env/jdbc/AppDB")
DataSource ds;
}
来源：https://stackoverflow.com/questions/25256487/how-to-java-configure-separate-datasources-for-spring-batch-data-and-business-da