How to get the JobID for the airflow dag runs?

问题

When we do a dagrun, on the Airflow UI, in the "Graph View" we get details of each job run.

JobID is something like "scheduled__2017-04-11T10:47:00".

I need this JobID for tracking and log creation in which I maintain time each task/dagrun took.

So my question is how can i get the JobID within the same dag that is being run.

Thanks,Chetan

回答1:

This value is actually called run_id and can be accessed via the context or macros.

In the python operator this is accessed via context, and in the bash operator this is accessed via jinja templating on the bash_command field.

More info on what's available in macros:

https://airflow.incubator.apache.org/code.html#macros

More info on jinja:

https://airflow.incubator.apache.org/concepts.html#jinja-templating

from airflow.models import DAG
from datetime import datetime
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator


dag = DAG(
    dag_id='run_id',
    schedule_interval=None,
    start_date=datetime(2017, 2, 26)
)

def my_func(**kwargs):
    context = kwargs
    print(context['dag_run'].run_id)

t1 = PythonOperator(
    task_id='python_run_id',
    python_callable=my_func,
    provide_context=True,
    dag=dag
    )

t2 = BashOperator(
    task_id='bash_run_id',
    bash_command='echo {{run_id}}',
    dag=dag)

t1.set_downstream(t2)

Use this dag as an example, and check the log for each operator, you should see the run_id printed in the log.

来源：https://stackoverflow.com/questions/43345991/how-to-get-the-jobid-for-the-airflow-dag-runs

标签

python

pyspark

airflow

apache-airflow