Cloudflare wrongly suspected that the widespread outage that took many websites offline on November 18 was caused by a DDoS attack, the company’s CEO admitted. However, in his blog post describing what happened, Matthew Prince explained that after realizing his mistake, his team was able to fix the problem. “The issue was not caused, directly or indirectly, by a cyberattack or malicious activity of any kind,” he wrote. Rather, it was due to a change in permissions for its database systems, which resulted in a problem with a file used by its bot management system.
The company’s bot management system uses a machine learning model to evaluate bots for each query they make as they explore Cloudflare’s network. Its clients rely on these scores to decide whether to allow or block certain bots from accessing their websites. One of the uses of bot scores is to be able to block bots from AI companies so that they cannot use content from a website to train their LLMs. In July, Cloudflare launched an experiment called “pay-per-crawl,” which lets website owners let an AI bot crawl their pages if they get paid for access.
Prince said the model relies on a “feature” configuration file to predict whether or not a bot request has been automated. The features file is refreshed every few minutes and a change in the underlying mechanism generating this file resulted in a change in its size which triggered the error. “As a result, HTTP 5xx error codes were returned by the primary proxy system that handles traffic processing for our customers, for any traffic dependent on the bot module,” Prince wrote.
This recent event marks Cloudflare’s worst outage in years. The company said it had not had an outage that would have “prevented the majority of primary traffic from flowing through the network.” [its] network” since 2019. Prince apologized for the issue on behalf of his team.
