DevOps Articles

Auto-Diagnosis and Remediation in Netflix Data Platform

https://netflixtechblog.medium.com/?source=post_page-----5bcc52d853d1-----------------------------------By https://www.linkedin.com/in/vikram-srivastava-8641ba6/ and https://www.linkedin.com/in/mayworm/ Netflix has one of the most complex data platforms in the cloud on which our data scientists and engineers run batch and streaming workloads. As our subscribers grow worldwide and https://about.netflix.com/en/news/let-the-games-begin-a-new-way-to-experience-entertainment-on-mobile, the number of batch workflows and real-time data pipelines increases rapidly.

Streaming Pensive ArchitectureApache Flink powers real-time stream processing jobs in the Netflix data platform.

The issue may occur in the source Kafka stream, the main Flink job, or the sinks to which the Flink job is writing data.