How to make 10,000 files in S3 public

Question

I have a folder in a bucket with 10,000 files. There seems to be no way to upload them and make them public straight away. So I uploaded them all, they\'re private, and I ne

Answer 1

If you are uploading for the first time, you can set the files to be public on upload on the command line:

aws s3 sync . s3://my-bucket/path --acl public-read

As documented in Using High-Level s3 Commands with the AWS Command Line Interface

Unfortunately it only applies the ACL when the files are uploaded. It does not (in my testing) apply the ACL to already uploaded files.

If you do want to update existing objects, you used to be able to sync the bucket to itself, but this seems to have stopped working.

[Not working anymore] This can be done from the command line:

aws s3 sync s3://my-bucket/path s3://my-bucket/path --acl public-read

(So this no longer answers the question, but leaving answer for reference as it used to work.)

Answer 2

I had the same problem, solution by @DanielVonFange is outdated, as new version of SDK is out.

Adding code snippet that works for me right now with AWS Ruby SDK:

require 'aws-sdk'

Aws.config.update({
  region: 'REGION_CODE_HERE',
  credentials: Aws::Credentials.new(
    'ACCESS_KEY_ID_HERE',
    'SECRET_ACCESS_KEY_HERE'
  )
})
bucket_name = 'BUCKET_NAME_HERE'

s3 = Aws::S3::Resource.new
s3.bucket(bucket_name).objects.each do |object|
  puts object.key
  object.acl.put({ acl: 'public-read' })
end

Answer 3

Have a look at BucketExplorer it manages bulk operations very well and is a solid S3 Client.

Answer 4

Using the cli:

aws s3 ls s3://bucket-name --recursive > all_files.txt && grep .jpg all_files.txt > files.txt && cat files.txt | awk '{cmd="aws s3api put-object-acl --acl public-read --bucket bucket-name --key "$4;system(cmd)}'

Answer 5

You can generate a bucket policy (see example below) which gives access to all the files in the bucket. The bucket policy can be added to a bucket through AWS console.

{
    "Id": "...",
    "Statement": [ {
        "Sid": "...",
        "Action": [
            "s3:GetObject"
        ],
        "Effect": "Allow",
        "Resource": "arn:aws:s3:::bucket/*",
        "Principal": {
            "AWS": [ "*" ]
        }
    } ]
}

Also look at following policy generator tool provided by Amazon.

http://awspolicygen.s3.amazonaws.com/policygen.html

Answer 6

Just wanted to add that with the new S3 Console you can select your folder(s) and select Make public to make all files inside the folders public. It works as a background task so it should handle any number of files.