Is there any direct way to copy one s3 directory to another in java or scala?

问题

I want to archive all the files and sub directories in a s3 directory to some other s3 location using java. Is there any direct way to copy one s3 directory to another in java or scala?

回答1:

There is no API call to operate on whole directories in Amazon S3.

In fact, directories/folders do not exist in Amazon S3. Rather, each object stores the full path in its filename (Key).

If you wish to copy multiple objects that have the same prefix in their Key, your code will need to loop through the objects, copying one object at a time.

回答2:

A bit wordy, but does the job: reasonable logging, multithreading via TransferManager, handling continuation token for "folders" with more than 1000 keys:

/**
 * Copies all content from s3://sourceBucketName/sourceFolder to s3://destinationBucketName/destinationFolder.
 */
public void copyAll(String sourceBucketName, String sourceFolder, String destinationBucketName, String destinationFolder) {
    log.info("Copying data from s3://{}/{} to s3://{}/{}", sourceBucketName, sourceFolder, destinationBucketName, destinationFolder);
    TransferManager transferManager = TransferManagerBuilder.standard()
            .withS3Client(client)
            .build();

    try {
        ListObjectsV2Request request = new ListObjectsV2Request()
                .withBucketName(sourceBucketName)
                .withPrefix(sourceFolder);

        ListObjectsV2Result objects;
        do {
            objects = client.listObjectsV2(request);

            List<Copy> transfers = new ArrayList<>();
            for (S3ObjectSummary object : objects.getObjectSummaries()) {
                String sourceKey = object.getKey();
                String sourceRelativeKey = sourceKey.substring(sourceFolder.length());
                String destinationKey = destinationFolder + sourceRelativeKey;

                transfers.add(transferManager.copy(sourceBucketName, sourceKey, destinationBucketName, destinationKey));
            }
            for (Copy transfer : transfers) {
                log.debug(transfer.getDescription());
                transfer.waitForCompletion();
            }

            log.info("Copied batch of {} objects. Last object: {}", transfers.size(), transfers.isEmpty() ? "None" : transfers.get(transfers.size() - 1).getDescription());
            request.setContinuationToken(objects.getNextContinuationToken());
        } while (objects.isTruncated());
        log.info("Copy operation completed successfully from s3://{}/{} to s3://{}/{}", sourceBucketName, sourceFolder, destinationBucketName, destinationFolder);
    } catch (InterruptedException e) {
        // Resetting interrupt flag and returning control to the caller.
        Thread.currentThread().interrupt();
        throw new RuntimeException(e);
    } finally {
        transferManager.shutdownNow(false);
    }
}

来源：https://stackoverflow.com/questions/56970836/is-there-any-direct-way-to-copy-one-s3-directory-to-another-in-java-or-scala