How do you search an amazon s3 bucket?

Question

I have a bucket with thousands of files in it. How can I search the bucket? Is there a tool you can recommend?

Answer 1

Search by Prefix in S3 Console

directly in the AWS Console bucket view.

Copy wanted files using s3-dist-cp

When you have thousands or millions of files another way to get the wanted files is to copy them to another location using distributed copy. You run this on EMR in a Hadoop Job. The cool thing about AWS is that they provide their custom S3 version s3-dist-cp. It allows you to group wanted files using a regular expression in the groupBy field. You can use this for example in a custom step on EMR

[
    {
        "ActionOnFailure": "CONTINUE",
        "Args": [
            "s3-dist-cp",
            "--s3Endpoint=s3.amazonaws.com",
            "--src=s3://mybucket/",
            "--dest=s3://mytarget-bucket/",
            "--groupBy=MY_PATTERN",
            "--targetSize=1000"
        ],
        "Jar": "command-runner.jar",
        "Name": "S3DistCp Step Aggregate Results",
        "Type": "CUSTOM_JAR"
    }
]