Distributed Job scheduling, management, and reporting

Question

I recently had a play around with Hadoop and was impressed with it\'s scheduling, management, and reporting of MapReduce jobs. It appears to make the distribution and execut

Answer 1

I guess you are looking for a workflow engine for CPU intensive tasks (also know "scientific workflow", e.g. http://www.extreme.indiana.edu/swf-survey). But I'm not sure how distributed do you want it to be. Usually all workflow engines have a "single point of failure".

Answer 2

I have since found Spring Batch and Spring Batch Integration which appear to address many of my requirements. I will let you know how I get on.

Answer 3

ProActive Scheduler seems to fit your requirements, especially the complex workflows you mentionned with tasks coordination. It is open source and Java based. You can use it to run anything, Hadoop jobs, scripts, Java code,...

Disclaimer: I work for the company behind it

Answer 4

I believe quite a few problems can be expressed as map-reduce problems.

For problems that you can't modify to fit the structure your can look at setting up your own using Java's ExecutorService. But it will be limited to one JVM and it will be quite low level. It will allow for easy coordination and synchronization however.

Answer 5

Take a look at Quartz. I think it supports stuff like managing jobs remotely and clustering several machines to run jobs.

Answer 6

Try Redisson framework. It provides easy api to execute and schedule java.util.concurrent.Callable and java.lang.Runnable tasks. Here is documentation about distributed Executor service and Scheduler service