airflow

问题 I am trying to pass a list of strings from one task to another one via XCom but I do not seem to manage to get the pushed list interpreted back as a list. For example, when I do this in some function blah that is run in a ShortCircuitOperator : paths = ['gs://{}/{}'.format(bucket, obj) for obj in my_list] kwargs['ti'].xcom_push(key='return_value', value=full_paths) and then I want to use such list as a parameter of an operator. For example, run_task_after_blah = AfterBlahOperator( task_id=

问题 This is actually two questions combined into one. My AIRFLOW_HOME is structured like airflow +-- dags +-- plugins +-- __init__.py +-- hooks +-- __init__.py +-- my_hook.py +-- another_hook.py +-- operators +-- __init__.py +-- my_operator.py +-- another_operator.py +-- sensors +-- utils I've been following astronomer.io's examples here https://github.com/airflow-plugins. My custom operators use my custom hooks , and all the imports are relative to the top level folder plugins . # my_operator.py

问题 Problem : I want to copy files from a folder in Google Cloud Storage Bucket (e.g Folder1 in Bucket1) to another Bucket (e.g Bucket2). I can't find any Airflow Operator for Google Cloud Storage to copy files. 回答1: I know this is an old question but I found myself dealing with this task too. Since I'm using the Google Cloud-Composer, GoogleCloudStorageToGoogleCloudStorageOperator was not available in the current version. I managed to solve this issue by using a simple BashOperator from airflow

问题 Is it possible to customize the format that Airflow uses for logging? I tried adding a LOG_FORMAT variable in $AIRFLOW_HOME/airflow.cfg, but it doesn't seem to take effect LOG_FORMAT = "%(asctime)s logLevel=%(levelname)s logger=%(name)s - %(message)s" 回答1: You need to change the settings.py file in the airflow package to change the log format Update settings.py (after LOGGING_LEVEL add below line): LOG_FORMAT = os.path.expanduser(conf.get('core', 'LOG_FORMAT')) Update airflow.cfg

问题 How to generate tasks dynamically based on the list returned from an upstream task. I have tried the following: Using an external file to write and read from the list - this option works but I am looking for a more elegant solution. Xcom pull inside a subdag factory. It does not work. I am able to pass a list from the upstream task to a subdag but that xcom is only accessible inside of a subdag's task and cannot be used to loop/iterate over the returned list and generate tasks. for e.g.

问题 I'm running airflow v 1.9.0. I am trying to get some form of authentication working but have so far failed to get github auth and password auth working. The password auth feels like it should be pretty straight forward and I'm hoping someone can point me in the right direction. My airflow.cfg has the following [webserver] authenticate = True auth_backend = airflow.contrib.auth.backends.password_auth Following the instructions here https://airflow.incubator.apache.org/security.html#password I

问题 I am using airflow on an EC2 instance using the LocalScheduler option. I've invoked airflow scheduler and airflow webserver and everything seems to be running fine. That said, after supplying the cron string to schedule_interval for "do this every 10 minutes," '*/10 * * * *' , the job continue to execute every 24 hours by default. Here's the header of the code: from datetime import datetime import os import sys from airflow.models import DAG from airflow.operators.python_operator import

问题 I want to try to use Airflow instead of Cron. But schedule_interval doesn't work as I expected. I wrote the python code like below. And in my understanding, Airflow should have ran on "2016/03/30 8:15:00" but it didn't work at that time. If I changed it like this "'schedule_interval': timedelta(minutes = 5)", it worked correctly, I think. The "notice_slack.sh" is just to call slack api to my channels. # -*- coding: utf-8 -*- from __future__ import absolute_import, unicode_literals import os

问题 I am trying to run a airflow DAG and need to pass some parameters for the tasks. How do I read the JSON string passed as the --conf parameter in the command line trigger_dag command, in the python DAG file. ex: airflow trigger_dag 'dag_name' -r 'run_id' --conf '{"key":"value"}' 回答1: Two ways. From inside a template field or file: {{ dag_run.conf['key'] }} Or when context is available, e.g. within a python callable of the PythonOperator : context['dag_run'].conf['key'] 来源： https:/

问题 Good Morning. I'm trying to setup a DAG too Watch/sense for a file to hit a network folder Process the file Archive the file Using the tutorials online and stackoverflow I have been able to come up with the following DAG and Operator that successfully achieves the objectives, however I would like the DAG to be rescheduled or rerun on completion so it starts watching/sensing for another file. I attempted to set a variable max_active_runs:1 and then a schedule_interval: timedelta(seconds=5)