Airflow: how to extend SubDagOperator?

问题

When I try to extend the SubDagOperator provided in airflow API, airflow webserver GUI does not recognize it as SubDagOperator thereby disabling me to zoom in to the subdag.

How can I extend the SubDagOperator while preserving the ability to zoom in to it as a subdag? Am I missing something?

回答1:

Please see the example below on how to extend the SubDagOperator. The key in your case was to override the task_type function

from airflow import DAG
from airflow.operators.subdag_operator import SubDagOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.utils.decorators import apply_defaults


class ExampleSubdagSubclassOperator(SubDagOperator):
  template_fields = ()
  template_ext = ()

  @apply_defaults
  def __init__(self, *args, **kwargs):

    dag = kwargs.get('dag')
    task_id = kwargs.get('task_id')

    subdag = DAG(
        '{}.{}'.format(dag.dag_id, task_id),
        schedule_interval=dag.schedule_interval,
        start_date=dag.start_date
    )

    # Replace the following 3 lines with code to automatically generate the desired tasks in the subdag
    t1 = DummyOperator(dag=subdag, task_id='t1')
    t2 = DummyOperator(dag=subdag, task_id='t2')
    t3 = DummyOperator(dag=subdag, task_id='t3')

    super(ExampleSubdagSubclassOperator, self).__init__(subdag=subdag, *args, **kwargs)

  # This property needs to be overridden so that the airflow UI recognises the task as a subdag and enables
  # the "Zoom into Sub Dag" button
  @property
  def task_type(self):
      return 'SubDagOperator'

回答2:

If you define the subdag as a dag itself this should work.

There is a good example from airflow. You also have to pay attention to the naming convention, the subdag name has to be of parent_name.child_name format.

the main dag

https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_subdag_operator.py

the subdags https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/subdags/subdag.py

来源：https://stackoverflow.com/questions/42256675/airflow-how-to-extend-subdagoperator

标签

airflow