spring batch file writer to write directly to amazon s3 storage without PutObjectRequest

问题

I'm trying to upload a file to amazon s3. Instead of uploading, I want to read the data from database using spring batch and write the file directly into the s3 storage. Is there anyway we can do that ?

回答1:

Spring Cloud AWS adds support for the Amazon S3 service to load and write resources with the resource loader and the s3 protocol. Once you have configured the AWS resource loader, you can write a custom Spring Batch writer like:

import java.io.OutputStream;
import java.util.List;

import org.springframework.batch.item.ItemWriter;
import org.springframework.core.io.ResourceLoader;
import org.springframework.core.io.WritableResource;

public class AwsS3ItemWriter implements ItemWriter<String> {

    private ResourceLoader resourceLoader;

    private WritableResource resource;

    public AwsS3ItemWriter(ResourceLoader resourceLoader, String resource) {
        this.resourceLoader = resourceLoader;
        this.resource = (WritableResource) this.resourceLoader.getResource(resource);
    }

    @Override
    public void write(List<? extends String> items) throws Exception {
        try (OutputStream outputStream = resource.getOutputStream()) {
            for (String item : items) {
                outputStream.write(item.getBytes());
            }
        }
    }
}

Then you should be able to use this writer with an S3 resource like s3://myBucket/myFile.log.

Is there anyway we can do that ?

Please note that I did not compile/test the previous code. I just wanted to give you a starting point of how to do it.

Hope this helps.

回答2:

The problem is that the OutputStream will only write the last List items sent by the step... I think you might need to write a temporary file on file system and then send the whole file in a separate tasklet

See this example : https://github.com/TerrenceMiao/AWS/blob/master/dynamodb-java/src/main/java/org/paradise/microservice/userpreference/service/writer/CSVFileWriter.java

来源：https://stackoverflow.com/questions/50685891/spring-batch-file-writer-to-write-directly-to-amazon-s3-storage-without-putobjec