How do I bulk upload to s3?

Question

I recently refactored some of my code to stuff rows into a db using \'load data\' and it works great -- however for each record I have I must upload 2 files to s3 -- this to

Answer 1

You can use the [putObjects][1] function of JetS3t to upload multiple files at once.

Alternatively you could use a background thread to upload to S3 from a queue, and add files to the queue from your code that loads the data into the database.

[1]: http://jets3t.s3.amazonaws.com/api/org/jets3t/service/multithread/S3ServiceMulti.html#putObjects(org.jets3t.service.model.S3Bucket, org.jets3t.service.model.S3Object[])

Answer 2

I just found a nice solution to upload an entire directory with php:

$client->uploadDirectory(
  SOURCE_FOLDER,
  YOUR_BUCKET_NAME,
  DESTINATION,
  array(
    'concurrency' => 5,
    'debug'          => TRUE,
    'force'            => FALSE,
    'params'        => array(
      'ServerSideEncryption' => 'AES256',
    ),
  )
);