Apache Airflow does not pickle DAGs

问题

I would like to recover DAG objects so that I can better inspect certain dependencies after DAG runs (e.g. what data is consumed by specific operators). I am using postgres:9.6 as metadata database backend.

This seems to be supported via the donot_pickle configuration variable, which by default indicates all DAGs must be pickled:

[core]
# Whether to disable pickling dags
donot_pickle = False

I have some test DAGs (3) available but their corresponding pickle_id is empty:

> select pickle_id from dag;
pickle_id
---------


(3 rows)

Pickles table is also empty:

> select count(*) from dag_pickle;
 count
------
    0
(1 row)

What might be going wrong here? I was not able to find any reference in the docs.

回答1:

There are 2 ways to enable pickling:

• DONT_PICKLE=False in scheduler config is only relevant for backfill jobs
• -p, -do_pickle in scheduler command line arguments enables pickling for scheduled jobs (https://airflow.apache.org/cli.html#Named%20Arguments_repeat18)

来源：https://stackoverflow.com/questions/48786437/apache-airflow-does-not-pickle-dags