Source: medium.com
Monitor jobs in a periodically spawn AWS EMR cluster
Monitoring EMR jobs is important, because we run several jobs on a cluster and failure of one doesn’t mean all others failed. It is enough to spin up the cluster, run the jobs and then terminate them.
The use case discussed in this post is incomplete without the below background information: In this post, you will learn, how to recieve notification for failure of a job run on an EMR cluster whose id changes.
Tricky part run_job_flow method of emr client creates a cluster and runs the jobs, but with a different clusterid each time.
After completion of jobs or failure of all jobs, the cluster is terminated.
The use case discussed in this post is incomplete without the below background information: In this post, you will learn, how to recieve notification for failure of a job run on an EMR cluster whose id changes.
Tricky part run_job_flow method of emr client creates a cluster and runs the jobs, but with a different clusterid each time.
After completion of jobs or failure of all jobs, the cluster is terminated.
Related Articles
Community Partners
DevOps Careers