How to efficiently import many large JSON files directly from S3 into MongoDB

Question

I have compressed JSON files in S3 and I would like to set up MongoDB in EC2 to server json documents contained in these files. The compressed files are >100M and there are 1000

Answer 1

You don't need to store intermediate files, you can pipe the output of s3 file to stdout and you can get input to mongoimport from stdin.

Your full command would look something like:

s3cmd get s3:// - | mongoimport -d  -c 

note the - which says send the file to stdout rather than to a filename.