airflow

My Cloud Composer managed Airflow got stuck for hours since I've canceled a Task Instance that was taking too long (Let's call it Task A) I've cleared all the DAG Runs and task instances, but there are a few jobs running and one job with Shutdown state (I suppose the job of Task A) ( snapshot of my Jobs ). Besides, it seems that the scheduler is not running since recently deleted DAGs keep appearing in the dashboard Is there a way to kill the jobs or reset the scheduler? Any idea to un-stuck the composer will be welcomed. You can restart the scheduler as follows: From your cloud shell: 1

I'm trying to cluster my Airflow setup and I'm using this article to do so. I just configured my airflow.cfg file to use the CeleryExecutor , I pointed my sql_alchemy_conn to my postgresql database that's running on the same master node, I've set the broker_url to use AWS SQS (I didn't set the access_key_id or secret_key since it's running on an EC2-Instance it doesn't need those), and I've set the celery_result_backend to my postgresql server too. I saved my new airflow.cfg changes, I ran airflow initdb , and then I ran airflow scheduler which worked. I went to the UI and turned on one of my

I am trying to run some hive job in airflow. I made custome jdbc connection which you can see in the image. I could query hive tables through airflow web ui(data profiling->ad hoc query). Also I want to run some sample dag file from Internet: #File Name: wf_incremental_load.py from airflow import DAG from airflow.operators import BashOperator, HiveOperator from datetime import datetime, timedelta default_args = { 'owner': 'airflow', 'start_date': datetime(2019, 3, 13), 'retries': 1, 'retry_delay': timedelta(minutes=5) } dag = DAG('hive_test', default_args=default_args,schedule_interval='* */5

I am using Airflow to run Spark jobs on Google Cloud Composer. I need to Create cluster (YAML parameters supplied by user) list of spark jobs (job params also supplied by per job YAML) With the Airflow API - I can read YAML files, and push variables across tasks using xcom. But, consider the DataprocClusterCreateOperator() cluster_name project_id zone and a few other arguments are marked as templated. What if I want to pass in other arguments as templated (which are currently not so)? - like image_version , num_workers , worker_machine_type etc? Is there any workaround for this? Not sure what

When I run a docker image using KubernetesPodOperator in Airflow version 1.10 Once the pod finishes the task successfullly, airflow tries to get the xcom value by making a connection to the pod via k8s stream client. Following is the error which I encountered: [2018-12-18 05:29:02,209] {{models.py:1760}} ERROR - (0) Reason: Handshake status 403 Forbidden Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/kubernetes/stream/ws_client.py", line 249, in websocket_call client = WSClient(configuration, get_websocket_url(url), headers) File "/usr/local/lib/python3.6/site

I'm using Apache Airflow standalone to submit my Spark jobs with SSHExecutorOperator to connect to the edge node and submit jobs with a simple BashCommand . It's mostly working well but sometimes some random tasks are running undefinitly. My job succeeds, but is still running according to Airflow. When I check the logs, it's like Airflow has stopped following the job as if it didn't get the return value. Why could this happen? Some jobs run for 10h+ and Airflow watches them successfully, while others fail. I have only Spark's logs (at INFO level) without anything printed by the job driver. It