DAG(directed acyclic graph) dynamic job scheduler

Question

I need to manage a large workflow of ETL tasks, which execution depends on time, data availability or an external event. Some jobs may fail during execution of the workflow and

Answer 1

Have you looked at Ruffus?

I have no experience with it but it appears to do some of the items on your list. It also looks quite hackable so you might be able to implement your other requirements yourself.