airflow

问题 We are experimenting with Apache Airflow (version 1.10rc2, with python 2.7) and deploying it to kubernetes, webserver and scheduler to different pods, and the database is as well using cloud sql, but we have been facing out of memory problems with the scheduler pod. At the moment of the OOM, we were running only 4 example Dags (approximately 20 tasks). The memory for the pod is 1Gib. I've seen in other posts that a task might consume approximately 50Mib of memory when running, and all task

问题 We are experimenting with Apache Airflow (version 1.10rc2, with python 2.7) and deploying it to kubernetes, webserver and scheduler to different pods, and the database is as well using cloud sql, but we have been facing out of memory problems with the scheduler pod. At the moment of the OOM, we were running only 4 example Dags (approximately 20 tasks). The memory for the pod is 1Gib. I've seen in other posts that a task might consume approximately 50Mib of memory when running, and all task

问题 I have some failing DAGs, let's say from 1st-Feb to 20th-Feb. From that date upword, all of them succeeded. I tried to use the cli (instead of doing it twenty times with the Web UI): airflow clear -f -t * my_dags.my_dag_id But I have a weird error: airflow: error: unrecognized arguments: airflow-webserver.pid airflow.cfg airflow_variables.json my_dags.my_dag_id EDIT 1: Like @tobi6 explained it, the * was indeed causing troubles. Knowing that, I tried this command instead: airflow clear -u -d

问题 I have a scenario wherein a particular dag upon completion needs to trigger multiple dags,have used TriggerDagRunOperator to trigger single dag,is it possible to pass multiple dags to the TriggerDagRunOperator to trigger multiple dags? And is it possible to trigger only upon successful completion of the current dag. 回答1: I have faced the same problem. And there is no solution out of the box, but we can write a custom operator for it. So here the code of a custom operator, that get python

问题 I have a scenario wherein a particular dag upon completion needs to trigger multiple dags,have used TriggerDagRunOperator to trigger single dag,is it possible to pass multiple dags to the TriggerDagRunOperator to trigger multiple dags? And is it possible to trigger only upon successful completion of the current dag. 回答1: I have faced the same problem. And there is no solution out of the box, but we can write a custom operator for it. So here the code of a custom operator, that get python

问题 What precisely is the difference between the "airflow initdb" command and the "airflow resetdb" command? Is it really necessary to have 2 different commands? When is it appropriate to use one vs the other? The doc says ... airflow initdb : Initialize the metadata database airflow resetdb : Burn down and rebuild the metadata database This doesn't tell me much. My best guess is that airflow initdb is to be used only the first time that the database is created from the airflow.cfg airflow

问题 What precisely is the difference between the "airflow initdb" command and the "airflow resetdb" command? Is it really necessary to have 2 different commands? When is it appropriate to use one vs the other? The doc says ... airflow initdb : Initialize the metadata database airflow resetdb : Burn down and rebuild the metadata database This doesn't tell me much. My best guess is that airflow initdb is to be used only the first time that the database is created from the airflow.cfg airflow

问题 What precisely is the difference between the "airflow initdb" command and the "airflow resetdb" command? Is it really necessary to have 2 different commands? When is it appropriate to use one vs the other? The doc says ... airflow initdb : Initialize the metadata database airflow resetdb : Burn down and rebuild the metadata database This doesn't tell me much. My best guess is that airflow initdb is to be used only the first time that the database is created from the airflow.cfg airflow

问题 What precisely is the difference between the "airflow initdb" command and the "airflow resetdb" command? Is it really necessary to have 2 different commands? When is it appropriate to use one vs the other? The doc says ... airflow initdb : Initialize the metadata database airflow resetdb : Burn down and rebuild the metadata database This doesn't tell me much. My best guess is that airflow initdb is to be used only the first time that the database is created from the airflow.cfg airflow

来源： https://stackoverflow.com/questions/64535476/import-error-in-vscode-despite-setting-the-pythonpath