EMR Cluster Creation using Airflow dag run, Once task is done EMR will be terminated

Question

I have Airflow jobs, which are running fine on the EMR cluster. what I need is, let\'s say if I have a 4 airflow jobs which required an EMR cluster for let\'s say 20 min to

Answer 1

The best way to do this is probably to have a node at the root of your Airflow DAG that creates the EMR cluster, and then another node at the very end of the DAG that spins the cluster down after all of the other nodes have completed.