How to terminate AWS EMR Cluster automatically after some time

Question

I currently have a task at hand to Terminate a long-running EMR cluster after a set period of time (based on some metric). Google Dataproc has this capability in something

Answer 1

I had to do a similar implementation and just considering Cluster Elapsed time was not solving our problem.

so we came up with a approach to hit the Hadoop API, you can find them here

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API

So here is what we did,

1. Ask the user who brings up a cluster to add a Tag like "AutoShutDown":"True:BufferMinutes", here "AutoShutDown" is the key and "True:BufferMinutes" is the value of the Tag

2. Here BufferMinutes is the time in minutes (30, 60 etc.)

3. create a Lambda to hit the hadoop api of all those clusters configured with step 1 (if the user does not add the Tag then the cluster is untouched) and fetch the end time of the last job that was completed (only if all jobs are either completed / terminated), if any job is still running then do nothing and exit.

4. now

   datetime_difference = (current_time - lastFinished) if(datetime_difference > requested_time) { terminate_cluster }

5. Create a cloud watch trigger and add the lambda created as target to it, schedule the trigger to run as required.

Note: Lambda is written in python, so boto3 is used and client will be "emr" same like what abdullahkhawer mentioned in his solution above.

This implementation gives flexibility to the user to choose and reduces a great deal of burden on dev-ops.