airflow

问题 I have been using the Postgres to S3 operator to load data from Postgres to S3. But recently, I had to export a very large table and my Airflow composer fails without any logs, this could be because we are using the NamedTemporaryFile function of Python's tempfile module to create a temporary file and we are using this temporary file to load to S3. Since we are using Composer, this will be loaded to Composer's local memory, and since the size of the file is very large, it is failing. Refer

问题 I am trying to install airflow on an AWS EC2 instance. The process seems to be pretty well documented by various sources on the web, however, I have run into a problem after I 'pip install' airflow; I get the below error when I execute the command 'airflow initdb': [2019-09-25 13:22:02,329] {__init__.py:51} INFO - Using executor SequentialExecutor Traceback (most recent call last): File "/home/cloud-user/.local/bin/airflow", line 22, in <module> from airflow.bin.cli import CLIFactory File "

问题 I am trying to install airflow on an AWS EC2 instance. The process seems to be pretty well documented by various sources on the web, however, I have run into a problem after I 'pip install' airflow; I get the below error when I execute the command 'airflow initdb': [2019-09-25 13:22:02,329] {__init__.py:51} INFO - Using executor SequentialExecutor Traceback (most recent call last): File "/home/cloud-user/.local/bin/airflow", line 22, in <module> from airflow.bin.cli import CLIFactory File "

问题 I am trying to install airflow on an AWS EC2 instance. The process seems to be pretty well documented by various sources on the web, however, I have run into a problem after I 'pip install' airflow; I get the below error when I execute the command 'airflow initdb': [2019-09-25 13:22:02,329] {__init__.py:51} INFO - Using executor SequentialExecutor Traceback (most recent call last): File "/home/cloud-user/.local/bin/airflow", line 22, in <module> from airflow.bin.cli import CLIFactory File "

问题 I want to create a snippet that passes the correct date based on whether the DAG was scheduled or whether it was triggered manually. The DAG runs monthly. The DAG generates a report (A SQL query) based on the data of the previous month. If I run the DAG scheduled, I can fetch the previous month with the following jinja snippet: execution_date.month given that the DAG is scheduled at the end of the previous period (last month) the execution_date will correctly return the last month. However on

问题 If I have multiple airflow dags with some overlapping python package dependencies, how can I keep each of these project deps. decoupled? Eg. if I had project A and B on same server I would run each of them with something like... source /path/to/virtualenv_a/activate python script_a.py deactivate source /path/to/virtualenv_b/activate python script_b.py deactivate Basically, would like to run dags with the same situation (eg. each dag uses python scripts that have may have overlapping package

问题 I want to store data from SQL to Pandas dataframe and do some data transformations and then load to another table suing airflow Issue that I am facing is that connection string to tables are accessbale only through Airflow. So I need to use airflow as medium to read and write data. How can this be done ? MY code Task1 = PostgresOperator( task_id='Task1', postgres_conn_id='REDSHIFT_CONN', sql="SELECT * FROM Western.trip limit 5 ", params={'limit': '50'}, dag=dag The output of task needs to be

问题 I have a python DAG Parent Job and DAG Child Job . The tasks in the Child Job should be triggered on the successful completion of the Parent Job tasks which are run daily. How can add external job trigger ? MY CODE from datetime import datetime, timedelta from airflow import DAG from airflow.operators.postgres_operator import PostgresOperator from utils import FAILURE_EMAILS yesterday = datetime.combine(datetime.today() - timedelta(1), datetime.min.time()) default_args = { 'owner': 'airflow',

问题 I have following pods: NAME READY STATUS RESTARTS AGE airflow-database-init-job-ggk95 0/1 Completed 0 3h airflow-redis-0 1/1 Running 0 3h airflow-scheduler-7594cd584-mlfrt 2/2 Running 9 3h airflow-sqlproxy-74f64b8b97-csl8h 1/1 Running 0 3h airflow-worker-5fcd4fffff-7w2sg 2/2 Running 0 3h airflow-worker-5fcd4fffff-m44bs 2/2 Running 0 3h airflow-worker-5fcd4fffff-mm55s 2/2 Running 0 3h composer-agent-0034135a-3fed-49a6-b173-9d3f9d0569db-ktwwt 0/1 Completed 0 3h composer-agent-0034135a-3fed-49a6

问题 I am trying to using the kubernetes pod operator in airflow, and there is a directory that I wish to share with kubernetes pod on my airflow worker, is there is a way to mount airflow worker's directory to kubernetes pod? I tried with the code below, and the volumn seems not mounted successfully. import datetime import unittest from unittest import TestCase from airflow.operators.kubernetes_pod_operator import KubernetesPodOperator from airflow.kubernetes.volume import Volume from airflow