How to get more than 1000 objects from S3 by using list_objects_v2?

Question

I have more than 500,000 objects on s3. I am trying get the size of each object. I am using the following python code for that

Answer 1

If you don't NEED to use the boto3.client you can use boto3.resource to get a complete list of your files:

s3r = boto3.resource('s3')
bucket = s3r.Bucket('bucket_name')
files_in_bucket = list(bucket.objects.all())

Then to get the size just:

sizes = [f.size for f in files_in_bucket]

Depending on the size of your bucket this might take a minute.