Source: towardsdatascience.com
Airflow in Docker Metrics ReportingCategory: Data, Kubernetes, Docker
An unsettling yet likely familiar situation: you deployed Airflow successfully, but find yourself constantly refreshing the webserver UI to make sure everything is running smoothly. You rely on certain alerting tasks to execute upon upstream failures, but if the queue is full and tasks are stalling, how will you be notified?
To start, I’ll assume basic understanding of Airflow functionality and containerization using Docker and Docker Compose. More resources can be found here for Airflow, here for Docker, and here for Docker Compose.
When starting the containers, there is a Prometheus datasource already configured in localhost:3000/datasources and an Airflow Metrics dashboard listed in localhost:3000/dashboards.
To start, I’ll assume basic understanding of Airflow functionality and containerization using Docker and Docker Compose. More resources can be found here for Airflow, here for Docker, and here for Docker Compose.
When starting the containers, there is a Prometheus datasource already configured in localhost:3000/datasources and an Airflow Metrics dashboard listed in localhost:3000/dashboards.
Related Articles
Community Partners
DevOps Careers