Extract Links Within Specific Folder in AWS S3 Buckets

问题

I am trying to get my AWS S3 API to list objects that I have stored in my S3 buckets. I have successfully used the code below to pull some of the links from my S3 buckets.

aws s3api list-objects --bucket my-bucket --query Contents[].[Key] --output text

The problem is the output in my command prompt is not listing the entire S3 Bucket inventory list. Is it possible to alter this code so that the output on my CLI lists the full inventory? If not, is there a way to alter the code to target specific file names within the S3 Bucket? For example, all the file names in my bucket are dates, so I would try and pull all the links from the file titled 3_15_20 Videos within the "my-bucket" bucket. Thanks in advance!

回答1:

From list-objects — AWS CLI Command Reference:

list-objects is a paginated operation. Multiple API calls may be issued in order to retrieve the entire data set of results. You can disable pagination by providing the --no-paginate argument.

Therefore, try using --no-paginate and see whether it returns all objects.

If you are regularly listing a bucket that contains a huge number of objects, you could also consider using Amazon S3 Inventory, which can provide a daily CSV file listing the contents of a bucket.

来源：https://stackoverflow.com/questions/60699363/extract-links-within-specific-folder-in-aws-s3-buckets