Reading file from HDFS using Spring batch

Question

I've to write a Spring batch which will read a file from HDFS and will update the data in MySQL DB.

The source file in HDFS contains some report data, in CSV format.

Can someone point me to an example of reading a file from HDFS?

Thanks.

Answer 1

The FlatFileItemReader in Spring Batch works with any Spring Framework Resource implementation:

@Bean
public FlatFileItemReader<String> itemReader() {
    Resource resource; // get (or autowire) resource
    return new FlatFileItemReaderBuilder<String>()
            .resource(resource)
            // set other reader properties
            .build();
}

So if you manage to have a Resource handle pointing to a HDFS file, your are done.

Now in order to have a HDFS resource, you can:

• Use Spring for Hadoop. Once the HDFS file system is configured, you would be able to get the resource from the application context with applicationContext.getResource("hdfs:data.csv");
• Implement your own Resource using Hadoop APIs (like shown in the answer by Michael Simons). I see that some folks already did this here

Hope this helps.