Node.js & Amazon S3: How to iterate through all files in a bucket?

Question

Is there any Amazon S3 client library for Node.js that allows listing of all files in S3 bucket?

The most known aws2js and knox don\'t seem to have this functionalit

Answer 1

If you want to get list of keys only within specific folder inside a S3 Bucket then this will be useful.

Basically, listObjects function will start searching from the Marker we set and it will search until maxKeys: 1000 as limit. so it will search one by one folder and get you first 1000 keys it find from different folder in a bucket.

Consider i have many folders inside my bucket with prefix as prod/some date/, Ex: prod/2017/05/12/ ,prod/2017/05/13/,etc.

I want to fetch list of objects (file names) only within prod/2017/05/12/ folder then i will specify prod/2017/05/12/ as my start and prod/2017/05/13/ [your next folder name] as my end and in code i'm breaking the loop when i encounter the end.

Each Keyin data.Contents will look like this.

{      Key: 'prod/2017/05/13/4bf2c675-a417-4c1f-a0b4-22fc45f99207.jpg',
       LastModified: 2017-05-13T00:59:02.000Z,
       ETag: '"630b2sdfsdfs49ef392bcc16c833004f94ae850"',
       Size: 134236366,
       StorageClass: 'STANDARD',
       Owner: { } 
 }

Code:

var list = [];

function listAllKeys(s3bucket, start, end) {
  s3.listObjects({
    Bucket: s3bucket,
    Marker: start,
    MaxKeys: 1000,
  }, function(err, data) {
      if (data.Contents) {
        for (var i = 0; i < data.Contents.length; i++) {
         var key = data.Contents[i].Key;    //See above code for the structure of data.Contents
          if (key.substring(0, 19) != end) {
             list.push(key);
          } else {
             break;   // break the loop if end arrived
          }
       }
        console.log(list);
        console.log('Total - ', list.length);      
     }
   });
 }

listAllKeys('BucketName', 'prod/2017/05/12/', 'prod/2017/05/13/');

Output:

[ 'prod/2017/05/12/05/4bf2c675-a417-4c1f-a0b4-22fc45f99207.jpg',
  'prod/2017/05/12/05/a36528b9-e071-4b83-a7e6-9b32d6bce6d8.jpg',
  'prod/2017/05/12/05/bc4d6d4b-4455-48b3-a548-7a714c489060.jpg',
  'prod/2017/05/12/05/f4b8d599-80d0-46fa-a996-e73b8fd0cd6d.jpg',
  ... 689 more items ]
Total - 692