Python multiprocessing BETWEEN Amazon cloud instances

Question

I\'m looking to run a long-running python analysis process on a few Amazon EC2 instances. The code already runs using the python multiprocessing module and can

Answer 1

I would use dumbo. It is a python wrapper for Hadoop that is compatible with Amazon Elastic MapReduce. Write a little wrapper around your code to integrate with dumbo. Note that you probably need a map-only job with no reduce step.