Source: open.nytimes.com
Failover Plans, Outage Playbooks and Resilience GapsCategory: Kubernetes, nginx, artificial-intelligence
By Alexandra Shaheen, Megan Araula and Shawn BowerThis is part two of an ongoing series about how The Times’s is working to improve the resilience of its systems. When we prepare for an election, we seek to fill resilience gaps that put our most important workflows at risk.
We expected every election event in 2020 to bring exceptionally high levels of traffic to our systems, and we needed to create alternate mechanisms that would enable us to perform our most critical workflows.
We needed to design a new pathway for machine-generated content that could handle this larger rate of publishing, and both our primary and failover stacks needed to account for this pathway.
We began investigating an active-active multi-CDN solution, but we quickly found that this was impossible because of our tight timeline before the election and the complexity of our caching layer.
We expected every election event in 2020 to bring exceptionally high levels of traffic to our systems, and we needed to create alternate mechanisms that would enable us to perform our most critical workflows.
We needed to design a new pathway for machine-generated content that could handle this larger rate of publishing, and both our primary and failover stacks needed to account for this pathway.
We began investigating an active-active multi-CDN solution, but we quickly found that this was impossible because of our tight timeline before the election and the complexity of our caching layer.
Related Articles
Community Partners
DevOps Careers