Spring Batch - Read files from Aws S3

Question

I am trying to read files from AWS S3 and process it with Spring Batch:

Can a Spring Itemreader process this Task? If so, How do I pass the credentials to S3 client

Answer 1

More simple steps are:

1. Create AWSS3 client bean.
2. Create ResourceLoader bean.
3. Use ResourceLoader to set S3 resources.

Firstly, you need to create AWSS3 client and ResourceLoader bean in your aws configuration file, like this.

@Configuration
@EnableContextResourceLoader
public class AWSConfiguration {

@Bean
@Primary
public AmazonS3 getAmazonS3Cient() {

    ClientConfiguration config = new ClientConfiguration();
    
    config.setConnectionTimeout(5000 * 10);
    config.setSocketTimeout(5000 * 10);

    return AmazonS3ClientBuilder.standard()
            .withClientConfiguration(config).build();
}


@Bean
@Autowired
public static ResourceLoaderBeanPostProcessor resourceLoaderBeanPostProcessor(
        AmazonS3 amazonS3EncryptionClient) {
    return new ResourceLoaderBeanPostProcessor(amazonS3EncryptionClient);
}

}

Then use resourceloader bean in ItemReader to set S3 resources.

@Autowired
private ResourceLoader resourceLoader;

@Bean
public FlatFileItemReader fileItemReader() {

FlatFileItemReader reader = new FlatFileItemReader<>();
reader.setLineMapper(new JsonLineMapper()); //Change line mapper as per your need
reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
return reader;
}