Shared storage between multiple AWS EC2 instances

Question

I\'m facing a problem to share a storage between multiple EC2 instances. I\'m going to have to run heavy jobs so I\'ll need a lot of instances to do it. On one side ,I have

Answer 1

Thanks for the answers. but finally I'm using NFS between the instances and it works pretty well!

Answer 2

Did you consider the option of having each worker write its logs on a local disk (maybe even on the ephemeral partition), and then make each worker upload its own big log file to S3 after it finishes?

This is somewhat similar to what happens when you use Elastic MapReduce to run some distributed tasks on a Hadoop cluster.

You'd get high write throughput (since it is writing to a local disk, if you use the ephemeral partition), and also high upload throughput to send the files to S3 (since you'd have the bandwidth of many workers available).

Answer 3

Not entirely sure of the context, but would writing objects directly on mounted S3 be feasible?

Answer 4

As is described in this thread and in some of the answers already provided, two common ways to accomplish this goal were to use S3 or NFS to share data access between instances.

On April 9th 2015, Amazon announced Amazon Elastic File System (Amazon EFS), which provides a much better solution to the problem you're trying to solve.