Adding logs to Airflow Logs

问题

How can I add my own logs onto the Apache Airflow logs that are automatically generated? any print statements wont get logged in there, so I was wondering how I can add my logs so that it shows up on the UI as well?

回答1:

I think you can work around this by using the logging module and trusting the configuration to Airflow.

Something like:

import ...

dag = ...

def print_params_fn(**kwargs):
    import logging
    logging.info(kwargs)
    return None

print_params = PythonOperator(task_id="print_params",
                              python_callable=print_params_fn,
                              provide_context=True,
                              dag=dag)

回答2:

If you look at the PythonOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/python_operator.py#L80-L81, looks like there is no way to log STDOUT/STDERR from the python callable into the airflow logs.

However, if you look at the BashOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/bash_operator.py#L79-L94, the STDOUT/STDERR from there is logged along with the airflow logs. So, if logs are important to you, I suggest adding the python code in a separate file and calling it using the BashOperator.

来源：https://stackoverflow.com/questions/40120467/adding-logs-to-airflow-logs