How do you search an amazon s3 bucket?

Question

I have a bucket with thousands of files in it. How can I search the bucket? Is there a tool you can recommend?

Answer 1

If you're on Windows and have no time finding a nice grep alternative, a quick and dirty way would be:

aws s3 ls s3://your-bucket/folder/ --recursive > myfile.txt

and then do a quick-search in myfile.txt

The "folder" bit is optional.

P.S. if you don't have AWS CLI installed - here's a one liner using Chocolatey package manager

choco install awscli

P.P.S. If you don't have the Chocolatey package manager - get it! Your life on Windows will get 10x better. (I'm not affiliated with Chocolatey in any way, but hey, it's a must-have, really).

Answer 2

Not a technical answer, but I have built an application which allows for wildcard search: https://bucketsearch.net/

It will asynchronously index your bucket and then allow you to search the results.

It's free to use (donationware).

Answer 3

The way I did it is: I have thousands of files in s3. I saw the properties panel of one file in the list. You can see the URI of that file and I copy pasted that to the browser - it was a text file and it rendered nicely. Now I replaced the uuid in the url with the uuid that I had at hand and boom there the file is.

I wish AWS had a better way to search a file, but this worked for me.

Answer 4

Given that you are in AWS...I would think you would want to use their CloudSearch tools. Put the data you want to search in their service...have it point to the S3 keys.

http://aws.amazon.com/cloudsearch/

Answer 5

Use Amazon Athena to query S3 bucket. Also, load data to Amazon Elastic search. Hope this helps.

Answer 6

AWS released a new Service to query S3 buckets with SQL: Amazon Athena https://aws.amazon.com/athena/