directed-acyclic-graphs

I have two weighted DAGs (directed acyclic graphs) and need to merge them into one, so I can get a topological ordering (it could be more than two in some cases). The problem is that the graphs are acyclic each, but can form a cycle together. Also, the graphs are large (100k+ nodes, 500k+ edges). Is there a clever way to merge the graphs? Equally good would be an algorithm to traverse all graphs "at once". Edit: By "merge" I mean combining all edges and vertices of both graphs together (preserving weights of course), if they do not create cycles. If an edge already exists I want to use the

I need to manage a large workflow of ETL tasks, which execution depends on time, data availability or an external event. Some jobs may fail during execution of the workflow and the system should have the ability to restart a failed workflow branch without waiting for whole workflow to finish execution. Are there any frameworks in python that can handle this? I see several core functions: DAG bulding Execution of nodes (run shell cmd with wait,logging etc.) Ability to rebuild sub-graph in parent DAG during execution Ability to manual execute nodes or sub-graph while parent graph is running

I have seen DAG getting generated whenever we perform any operation on RDD but what happens when we perform operations on our dataframe? When executing multiple operations on dataframe, Are those lazily evaluated just like RDD? When the catalyst optimizer comes into the picture? I am sort of confused between these. If anyone can throw some light on these topics, it would be really of great help. Thanks Every operation on a Dataset , continuous processing mode notwithstanding, is translated into a sequence of operations on internal RDDs . Therefore concept of DAG is by all means applicable. By

With Airflow, is it possible to restart an upstream task if a downstream task fails? This seems to be against the "Acyclic" part of the term DAG. I would think this is a common problem though. Background I'm looking into using Airflow to manage a data processing workflow that has been managed manually. There is a task that will fail if a parameter x is set too high, but increasing the parameter value gives better quality results. We have not found a way to calculate a safe but maximally high parameter x. The process by hand has been to restart the job if failed with a lower parameter until it

I am planning on creating a Node Graph in pyqt. The abstract models that qt provides work for 1D, 2D and Tree data but the abstract class seems to break down for something like a node graph. In particular the "parent" function in QAbstractModel returns QModelIndex of a single parent. In a DAG I will may have multiple parents. One resource I found was this blog post: http://invalidmagic.wordpress.com/2009/12/10/qgraphicsscene-used-as-a-qabstractitemmodel/ It provides some useful information, but I can't seem figure how the model represents the concept of multiple parents. I'm looking for

I have an acyclic directed graph. I would like to assign levels to each vertex in a manner that guarantees that if the edge (v1,v2) is in the graph, then level(v1) > level(v2). I would also like it if level(v1) = level(v3) whenever (v1,v2) and (v3,v2) are in the graph. Also, the possible levels are discrete (might as well take them to be the natural numbers). The ideal case would be that level(v1) = level(v2) + 1 whenever (v1,v2) is in the graph and there is no other path from v1 to v2, but sometimes that isn't possible with the other constraints - e.g, consider a graph on five vertices with