spring-batch

Best approach using Spring batch to process big file

北战南征 提交于 2019-11-28 06:33:09
问题 I am using Spring batch to download a big file in order to process it. the scenario is pretty simple: 1. Download the file via http 2. process it(validations,transformations) 3. send it into queue no need to save the input file data. we might have multiple job instances(of the same scenario) running in the same time I am looking for best practice to handle this situation. Should I create Tasklet to download the file locally and than start processing it via regular steps? in that case I need

Stopping a file inbound channel adapter after one one file is read

喜你入骨 提交于 2019-11-28 06:27:51
问题 Our application uses a Spring Integration file:inbound-channel-adapter to poll a directory to listen to when a file is dropped there. Spring Integration then starts a Spring Batch job, handing over to the job the path and name of the file to process. Obviously, the file poller continues to run even after a file has been processed by the Spring Batch job. So, the Spring context remains open and the application does not terminate. Is there a way, programatically or through configuration

Spring Batch - How to prevent batch from storing transactions in DB

孤街醉人 提交于 2019-11-28 05:42:41
问题 First the problem statement: I am using Spring-Batch in my DEV environment fine. When I move the code to a production environment I am running into a problem. In my DEV environment, Spring-Batch is able to create it's transaction data tables in our DB2 database server with out problem. This is not a option when we go to PROD as this is a read only job. Attempted solution: Search Stack Overflow I found this posting: Spring-Batch without persisting metadata to database? Which sounded perfect,

How Spring Boot run batch jobs

核能气质少年 提交于 2019-11-28 05:15:17
I followed this sample for Spring Batch with Boot. When you run the main method the job is executed. This way I can't figure out how one can control the job execution. For example how you schedule a job, or get access to the job execution, or set job parameters. I tried to register my own JobLauncher @Bean public JobLauncher jobLauncher(JobRepository jobRepo){ SimpleJobLauncher simpleJobLauncher = new SimpleJobLauncher(); simpleJobLauncher.setJobRepository(jobRepo); return simpleJobLauncher; } but when I try to use it in the main method: public static void main(String[] args) {

Why Spring's jdbcTemplate.batchUpdate() so slow?

给你一囗甜甜゛ 提交于 2019-11-28 05:02:38
I'm trying to find the faster way to do batch insert . I tried to insert several batches with jdbcTemplate.update(String sql) , where sql was builded by StringBuilder and looks like: INSERT INTO TABLE(x, y, i) VALUES(1,2,3), (1,2,3), ... , (1,2,3) Batch size was exactly 1000. I inserted nearly 100 batches. I checked the time using StopWatch and found out insert time: min[38ms], avg[50ms], max[190ms] per batch I was glad but I wanted to make my code better. After that, I tried to use jdbcTemplate.batchUpdate in way like: jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {

Triggering spark jobs with REST

馋奶兔 提交于 2019-11-28 04:43:31
I have been of late trying out apache spark . My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my requirement. I have a REST end point where I expose API to trigger Jobs, I have used Spring4.0 for Rest Implementation. Now going ahead I thought of implementing Jobs as Service in Spring where I would submit Job programmatically, meaning when the endpoint is triggered, with given parameters I would trigger the job. I have now few design options. Similar to the below written job, I need to maintain

Integrating Spring Batch Admin into an existing application

亡梦爱人 提交于 2019-11-28 04:35:46
I have an application which uses Spring Batch and Spring MVC. I am able to deploy Spring Batch Admin as a separate war and use it against the same DB my application uses, though I would like to integrate it into my own application, possibly modify some of the views as well. Is there an easy way to do this or do I have to fork it and go from there? There is an easy way apparently according to this thread ; Define a DispatcherServlet for Batch Admin in web.xml : <servlet> <servlet-name>Batch Servlet</servlet-name> <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>

Spring batch repeat step ending up in never ending loop

半腔热情 提交于 2019-11-28 04:30:51
问题 I have a spring batch job that I'd like to do the following... Step 1 - Tasklet - Create a list of dates, store the list of dates in the job execution context. Step 2 - JDBC Item Reader - Get list of dates from job execution context. Get element(0) in dates list. Use is as input for jdbc query. Store element(0) date is job execution context Remove element(0) date from list of dates Store element(0) date in job execution context Flat File Item Writer - Get element(0) date from job execution

How to Override Spring-boot application.properties programmatically

百般思念 提交于 2019-11-28 03:56:18
I have jdbc property files which I take from external configuration web-service In spring boot in order to set mysql props it's easy as adding those to application.properties: spring.datasource.url=jdbc:mysql://localhost/mydb spring.datasource.username=root spring.datasource.password=root spring.datasource.driver-class-name=com.mysql.jdbc.Driver How could I override those programticlly in my app? same goes for Spring-batch props: database.driver=com.mysql.jdbc.Driver database.url=jdbc:mysql://localhost/mydv database.username=root database.password=root Lukas Hinsch You can add additional

org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step

ぃ、小莉子 提交于 2019-11-28 02:20:07
I am learning spring batch and I was able to create simple single step application( github repo link ) This application contains a job which does following: 1. reads persons from csv file 2. lowercase their names 3. Save them into databse Now I want to learn partition feature so I added following partitioner: @Component public class MyPartitioner implements Partitioner { @Override public Map<String, ExecutionContext> partition(int gridSize) { Map<String, ExecutionContext> map = new HashMap<>(gridSize); for (int k = 0; k < gridSize; k++) { ExecutionContext context = new ExecutionContext();