spring-batch

Spring Batch: Which ItemReader implementation to use for high volume & low latency

由 余生颓废 提交于 2019-11-30 08:35:32
Use case: Read 10 million rows [10 columns] from database and write to a file (csv format). Which ItemReader implementation among JdbcCursorItemReader & JdbcPagingItemReader would be suggested? What would be the reason? Which would be better performing (fast) in the above use case? Would the selection be different in case of a single-process vs multi-process approach? In case of a multi-threaded approach using TaskExecutor, which one would be better & simple? To process that kind of data, you're probably going to want to parallelize it if that is possible (the only thing preventing it would be

How to process logically related rows after ItemReader in SpringBatch?

由 会有一股神秘感。 提交于 2019-11-30 08:28:36
Scenario To make it simple, let's suppose I have an ItemReader that returns me 25 rows. The first 10 rows belong to student A The next 5 belong to student B and the 10 remaining belong to student C I want to aggregate them together logically say by studentId and flatten them to end up with one row per student. Problem If I understand correctly, setting the commit interval to 5 will do the following: Send 5 rows to the Processor (which will aggregate them or do any business logic I tell it to). After Processed will write 5 rows. Then it will do it again for the next 5 rows and so on. If that is

Spring Batch how to filter duplicated items before send it to ItemWriter

由 僤鯓⒐⒋嵵緔 提交于 2019-11-30 07:18:15
问题 I read a flat file (for example a .csv file with 1 line per User, Ex: UserId;Data1;Date2 ). But how to handle duplicated User item in the reader (where is no list of previus readed users...) stepBuilderFactory.get("createUserStep1") .<User, User>chunk(1000) .reader(flatFileItemReader) // FlatFileItemReader .writer(itemWriter) // For example JDBC Writer .build(); 回答1: Filtering is typically done with an ItemProcessor . If the ItemProcessor returns null, the item is filtered and not passed to

How does Spring Batch transaction management work?

由 烂漫一生 提交于 2019-11-30 06:38:31
问题 I'm trying to understand how Spring Batch does transaction management. This is not a technical question but more of conceptual one: what approach does Spring Batch use and what are the consequences of that approach? Let me try to clarify this question a bit. For instance, looking at the TaskletStep, I see that generally a step execution looks something like this: several JobRepository transactions to prepare the step metadata a business transaction for every chunk to process more

Tasklet to delete a table in spring batch

由 给你一囗甜甜゛ 提交于 2019-11-30 06:04:04
问题 I have steps in the batch job that does different things. But before I begin all these steps, I need to clear a table. Is there any simple way to write a tasklet that will delete the table directly from the job xml file ? I am using ibatis as ORM 回答1: you mean even more simple than a tasklet, e.g. like this pseudocode ? <!-- xml bean config --> <bean id="deleteTableTaskletStep" class="..."> <property name="dataSource" ref="dataSource" /> <property name="sql" value="delete from ..." /> </bean>

What are the Spring Batch “default” Context Variables?

由 南笙酒味 提交于 2019-11-30 04:05:06
问题 In the Spring Batch step-scope documentation, there are three unexplained spring-batch context maps: jobParameters , jobExecutionContext , and stepExecutionContext . Springsource sample code, combined: <bean id="flatFileItemReader" scope="step" class="org.springframework.batch.item.file.FlatFileItemReader"> <property name="var1" value="#{jobParameters['input.file.name']}" /> <property name="var2" value="#{jobExecutionContext['input.file.name']}" /> <property name="var3" value="#

Reading line breaks in CSV which are quoted in the file in FlatfileItemReader of spring batch

由 我们两清 提交于 2019-11-30 03:47:08
问题 I am trying to parse a CSV file with FlatFileItemReader. This CSV contains some quoted newline characters as shown below. email, name abc@z.com, "NEW NAME ABC" But this parsing is failing with required fields are 2 but actual is 1. What I am missing in my FlatFileReader configuration? <property name="lineMapper"> <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper"> <!-- The lineTokenizer divides individual lines up into units of work --> <property name="lineTokenizer">

H2 SQL Grammar Exception

由 偶尔善良 提交于 2019-11-30 03:22:00
问题 I am trying to import a sql script for H2. This script is provided by spring-batch and it's used to store the jobs metadata. When I execute this script directly in the H2 console, I have no syntax errors, but I referenced the same script in Hibernate/JPA to be imported at the initialization phase, I got this exception : org.hibernate.tool.hbm2ddl.ImportScriptException: Error during statement execution (file: 'org/springframework/batch/core/schema-h2.sql'): CREATE TABLE BATCH_JOB_INSTANCE ( ..

Why does destroy method 'close' fail for JPAPagingItemReader configured with Java config?

由 北战南征 提交于 2019-11-30 03:07:57
We are trying to convert our Spring-Batch jobs from XML configuration to Java configuration. We are using Spring 4.0.1.RELEASE and Spring Batch 2.2.1.RELEASE. After converting one job, the following warning started to appear in the log file: 15-Apr-2014 09:59:26.335 [Thread-2] WARN o.s.b.f.s.DisposableBeanAdapter - Invocation of destroy method 'close' failed on bean with name 'fileReader': org.springframework.batch.item.ItemStreamException: Error while closing item reader The full stacktrace is: org.springframework.batch.item.ItemStreamException: Error while closing item reader at org

Transaction management with Spring Batch

由 爱⌒轻易说出口 提交于 2019-11-30 02:59:13
I am discovering actually Spring and I am able to setup some jobs. Now, I would like to save my imported datas in a database using Hibernate/JPA and I keep getting this error : 14:46:43.500 [main] ERROR o.s.b.core.step.AbstractStep - Encountered an error executing the step javax.persistence.TransactionRequiredException: no transaction is in progress I see that the problem is with the transaction. Here is my spring java config for the entityManager and the transactionManager : @Configuration public class PersistenceSpringConfig implements EnvironmentAware { @Bean public
订阅 spring-batch