- Cloudflare admits it caused its own outage: it wasn’t a cyberattack
- Fluctuating bug reports made the issue difficult to identify at first
- “Unacceptable” disruption created some learning opportunities
Cloudflare has shared more details about its November 18 outage, its worst outage since 2019, confirming that it was not the result of an attack or any other type of malicious activity.
In a blog post, the company’s co-founder and CEO Matthew Prince explained that a database permission change caused the system to generate a “feature file” that doubled in size, before propagating to all the machines on its network, causing the software to crash.
Because Cloudflare was able to identify what had gone wrong, normal operation resumed just over three hours after the outage, with a full recovery a few hours later.
Cloudflare confirms the outage was not an attack
“Main traffic was largely flowing normally at 2:30 p.m.,” Prince wrote, as confirmed by a chat showing a large drop in HTTP error 5xx status codes around that time.
However, Cloudflare needed to dig a little deeper to find out what exactly was going on due to a fairly high fluctuation range in the bug reports. This was because the problematic file was generated every five minutes.
“In addition to returning HTTP 5xx errors, we saw significant increases in the latency of our CDN responses during the impact period,” Prince added, noting that “large amounts of CPU” were being used during debugging and observability.
Cloudflare’s status page also went down during the attack, a page that is completely independent of Cloudflare’s infrastructure. Apparently, this was little more than a coincidence.
However, the outage at least served as a learning opportunity for Cloudflare, which now promises to enable more global disconnection features.
“A blackout like today’s is unacceptable,” Prince concluded, before putting a slightly positive spin on it: “When we’ve had blackouts in the past, it’s always led us to build new, more resilient systems.”
Follow TechRadar on Google News and add us as a preferred source to receive news, reviews and opinions from our experts in your feeds. Be sure to click the Follow button!
And of course you can also follow TechRadar on TikTok for news, reviews, unboxings in video form and receive regular updates from us on WhatsApp also.
