I've created: https://jira.spring.io/browse/BATCH-2778
I am developing Spring Batch + Redis (Spring Data Redis) example. In this example, I'm reading student.csv
file and storing all the data as is in Redis DB. I wanted to used dateOfBirth
as Date
and I am sure that I need to do some date logic conversion to store value Date in Redis
.
As per my analysis, it looks like I wont be able to use @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd")
, because I am not dealing with JSON at all. I think I should be using the Converter etc to convert the Byte[]
to Date
and Date
to Byte[]
etc.
Infact I tried these options, but doesn't working. Any quick help?
Note: I don't want to save dateOfBirth as String value.
org.springframework.batch.item.file.FlatFileParseException: Parsing error at line: 2 in resource=[URL [file:c:/Videos/student.csv]], input=[1,John,Doe,05-12-1988 12:34:45] at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:184) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.read(AbstractItemCountingItemStreamItemReader.java:89) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader$$FastClassBySpringCGLIB$$ebb633d0.invoke(<generated>) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.batch.item.file.FlatFileItemReader$$EnhancerBySpringCGLIB$$87932856.read(<generated>) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.item.SimpleChunkProvider.doRead(SimpleChunkProvider.java:91) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.item.SimpleChunkProvider.read(SimpleChunkProvider.java:157) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.item.SimpleChunkProvider$1.doInIteration(SimpleChunkProvider.java:116) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.item.SimpleChunkProvider.provide(SimpleChunkProvider.java:110) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:69) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) ~[spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:394) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:308) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:141) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:134) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_162] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_162] at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_162] at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) [spring-aop-5.0.10.RELEASE.jar:5.0.10.RELEASE] at com.sun.proxy.$Proxy69.run(Unknown Source) [na:na] at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:163) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:179) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:134) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:128) [spring-boot-autoconfigure-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:792) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:776) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1242) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1230) [spring-boot-2.0.6.RELEASE.jar:2.0.6.RELEASE] at com.myexample.DateBatchPocApplication.main(DateBatchPocApplication.java:12) [classes/:na] Caused by: org.springframework.validation.BindException: org.springframework.validation.BeanPropertyBindingResult: 1 errors Field error in object 'target' on field 'dateOfBirth': rejected value [05-12-1988 12:34:45]; codes [typeMismatch.target.dateOfBirth,typeMismatch.dateOfBirth,typeMismatch.java.util.Date,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [target.dateOfBirth,dateOfBirth]; arguments []; default message [dateOfBirth]]; default message [Failed to convert property value of type 'java.lang.String' to required type 'java.util.Date' for property 'dateOfBirth'; nested exception is java.lang.IllegalStateException: Cannot convert value of type 'java.lang.String' to required type 'java.util.Date' for property 'dateOfBirth': no matching editors or conversion strategy found] at org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper.mapFieldSet(BeanWrapperFieldSetMapper.java:200) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.item.file.mapping.DefaultLineMapper.mapLine(DefaultLineMapper.java:43) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:180) ~[spring-batch-infrastructure-4.0.1.RELEASE.jar:4.0.1.RELEASE] ... 56 common frames omitted
StudentJob.java
@Configuration public class StudentJob { @Autowired private JobBuilderFactory jobBuilderFactory; @Autowired private StepBuilderFactory stepBuilderFactory; @Autowired private FlatFileItemReader<Student> studentReader; @Autowired private StudentWritter studentWritter; @Bean public Job readstudentCSVFileJob() { return jobBuilderFactory.get("readstudentCSVFileJob").incrementer(new RunIdIncrementer()) .start(countryCurrStepOne()).build(); } @Bean public Step countryCurrStepOne() { return stepBuilderFactory.get("studentStepOne").<Student, Student>chunk(5).reader(studentReader) .writer(studentWritter).build(); } }
StudentBatchConfig.java
@Slf4j @Configuration public class StudentBatchConfig { @Bean(destroyMethod="") @StepScope public FlatFileItemReader<Student> studentReader(@Value("${input.student.path}") Resource resource) throws IOException { log.debug("Resource Path : "+resource.getFile()); FlatFileItemReader<Student> itemReader = new FlatFileItemReader<>(); itemReader.setName("STUDENT_READER"); itemReader.setResource(resource); itemReader.setLineMapper(studentlineMapper()); itemReader.setLinesToSkip(1); return itemReader; } @Bean public LineMapper<Student> studentlineMapper() { // Delimited Line Tokenizer DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer(); lineTokenizer.setNames("id", "firstName", "lastName", "dateOfBirth"); lineTokenizer.setIncludedFields(new int[] {0,1,2,3}); // Bean Wrapper Field SetMapper BeanWrapperFieldSetMapper<Student> fieldSetMapper = new BeanWrapperFieldSetMapper<>(); fieldSetMapper.setTargetType(Student.class); DefaultLineMapper<Student> lineMapper = new DefaultLineMapper<>(); lineMapper.setLineTokenizer(lineTokenizer); lineMapper.setFieldSetMapper(fieldSetMapper); return lineMapper; } @Bean public StudentWritter studentWritter() { return new StudentWritter(); } }
StudentWritter.java
public class StudentWritter implements ItemWriter<Student>{ @Autowired private StudentRepository studentRepository; @Override public void write(List<? extends Student> students) throws Exception { studentRepository.saveAll(students); } }
EDIT-1:
@Slf4j @Configuration public class StudentBatchConfig { private static final SimpleDateFormat format = new SimpleDateFormat("dd-MM-yyyy HH:mm"); @Bean(destroyMethod="") @StepScope public FlatFileItemReader<Student> countryReader(@Value("${input.student.path}") Resource resource) throws IOException { log.debug("Resource Path : "+resource.getFile()); FlatFileItemReader<Student> itemReader = new FlatFileItemReader<>(); itemReader.setName("STUDENT_READER"); itemReader.setResource(resource); itemReader.setLineMapper(studentLineMapper()); itemReader.setLinesToSkip(1); return itemReader; } @SuppressWarnings("rawtypes") @Bean public LineMapper<Student> studentLineMapper() { DefaultLineMapper<Student> lineMapper = new DefaultLineMapper<>(); // Delimited Line Tokenizer DelimitedLineTokenizer lineTokenizer = new DelimitedLineTokenizer(); lineTokenizer.setNames("id", "firstName", "lastName", "dateOfBirth"); lineTokenizer.setIncludedFields(new int[] {0,1,2,3}); // Date parsing logic has been added CustomDateEditor customDateEditor = new CustomDateEditor(format, false); HashMap<Class, PropertyEditor> customEditors = new HashMap<>(); customEditors.put(Date.class, customDateEditor); // Bean Wrapper Field SetMapper BeanWrapperFieldSetMapper<Student> fieldSetMapper = new BeanWrapperFieldSetMapper<>(); fieldSetMapper.setTargetType(Student.class); fieldSetMapper.setCustomEditors(customEditors); lineMapper.setLineTokenizer(lineTokenizer); lineMapper.setFieldSetMapper(fieldSetMapper); return lineMapper; } @Bean public StudentWritter studentWritter() { return new StudentWritter(); } }
Student.java
@Builder @Data @AllArgsConstructor @NoArgsConstructor @RedisHash("student") public class Student { @Id @Indexed private String id; private String firstName; private String lastName; private Date dateOfBirth; }
But dates are not storing correctly.. seem like formatting issue..
NOTE: If I open this file in CSV editor of Microsoft it shows me dd-MM-yyyy HH:mm
But If I edit this file in Notepad++, then it shows
id,firstName,lastName,dateOfBirth 1,John,Doe,25-11-2018 14:48:10 2,Sameer,Kumbhare,25-11-2018 14:48:10