Is it possible to copy all files from one S3 bucket to another with s3cmd?

Question

I\'m pretty happy with s3cmd, but there is one issue: How to copy all files from one S3 bucket to another? Is it even possible?

EDIT: I\'ve found a way to copy files

Answer 1

The answer with the most upvotes as I write this is this one:

s3cmd sync s3://from/this/bucket s3://to/this/bucket

It's a useful answer. But sometimes sync is not what you need (it deletes files, etc.). It took me a long time to figure out this non-scripting alternative to simply copy multiple files between buckets. (OK, in the case shown below it's not between buckets. It's between not-really-folders, but it works between buckets equally well.)

# Slightly verbose, slightly unintuitive, very useful:
s3cmd cp --recursive --exclude=* --include=file_prefix* s3://semarchy-inc/source1/ s3://semarchy-inc/target/

Explanation of the above command:

• –recursive
  In my mind, my requirement is not recursive. I simply want multiple files. But recursive in this context just tells s3cmd cp to handle multiple files. Great.
• –exclude
  It’s an odd way to think of the problem. Begin by recursively selecting all files. Next, exclude all files. Wait, what?
• –include
  Now we’re talking. Indicate the file prefix (or suffix or whatever pattern) that you want to include.
  s3://sourceBucket/ s3://targetBucket/
  This part is intuitive enough. Though technically it seems to violate the documented example from s3cmd help which indicates that a source object must be specified:
  s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]