Anatomy of a Distributed System in PHP

Question

I\'ve a problem which is giving me some hard time trying to figure it out the ideal solution and, to better explain it, I\'m going to expose my scenario here.

Answer 1

The setup of a master server and several workers looks right in your case.

On the master server I would install MySQL (Percona InnoDB version is stable and fast) in master-master replication so you won't have a single point of failure. The master server will host an API which the workers will pull at every N seconds. The master will check if there is a job available, if so it has to flag that the job has been assigned to the worker X and return the appropriate input to the worker (all of this via HTTP). Also, here you can store all the script files of the workers.

On the workers, I would strongly suggest you to install a Linux distro. On Linux it's easier to set up scheduled tasks and in general I think it's more appropriate for the job. With Linux you can even create a live cd or iso image with a perfectly configured worker and install it fast and easy on all the machines you want. Then set up a cron job that will RSync with the master server to update/modify the scripts. In this way you will change the files in just one place (the master server) and all the workers will get the updates.

In this configuration you don't care of the IPs or the number of workers because the workers are connecting to the master, not vice-versa.

The worker job is pretty easy: ask the API for a job, do it, send back the result via API. Rinse and repeat :-)