After last week’s discovery of a massive security hole in the widely used online security service Cloudflare, the company reacted quickly, reflecting the severity of the flaw. It’s a reminder that response is often just as important as security in the first place.
A long time ago—probably five-plus years back—I received a T-shirt in the mail.
It featured a list of names of the earliest users of Cloudflare, done up in the shape of the company’s logo. In microscopic type, my website’s URL was on it. I had seen immediately the value of the platform, which helps mitigate online threats—it was an extension of an older service called Project Honey Pot, which helped highlight bad, often automated, online actors. Cloudflare did more than highlight the bad actors; it prevented them from visiting your site.
In the roughly eight years since its launch, Cloudflare has become one of the internet’s most basic security services—a fact that was reinforced recently after a major security hole affecting WordPress users was discovered. Because it worked as a go-between that separated WordPress sites that used Cloudflare from the larger internet, it was able to automatically block the hole for those sites.
Now, it’s Cloudflare’s turn to be on the receiving end of a security hole.
“Cloudbleed,” as the exploit has already been unfortunately nicknamed, functions a lot like Heartbleed, a serious security flaw from two years ago involving the similarly widely used OpenSSL protocol. Basically, due to a memory leak in Cloudflare’s system, information that was intended to be secure bled into files on different webpages throughout the internet.
The team has worked continuously to ensure that this bug and its consequences are fully dealt with.
The leaked data was fairly random, but it included sensitive information such as passwords, personal chat logs, application data, and credit card info. The odds of it happening to a single data request were similar to winning the lottery (1 in 3.3 million HTTP requests were affected), but because so many HTTP requests are made in a day, the odds are high that it affected a number of users over the six-month period the bug was in the wild. (Extensions for Chrome and Firefox can help you determine if you were hit; or you can try sites like DoesItUseCloudflare.)
On the plus side, Cloudflare had the right instincts about dealing with this problem: After being informed of the bug, the company quickly went into action. It says it mitigated the bug within 47 minutes and completely fixed the problem within seven hours.
“Having a global team meant that, at 12-hour intervals, work was handed over between offices, enabling staff to work on the problem 24 hours a day,” the company’s John Graham-Cumming wrote. “The team has worked continuously to ensure that this bug and its consequences are fully dealt with.”
But even with these high standards, this bug—which was caused, literally, by a single mistyped character—caused Cloudflare’s reputation to take a huge hit, and things could have been worse. There are always things to tweak. A company the size of Cloudflare probably needs to fortify its bug bounty program a bit more.
Whether you’re Cloudflare or a small-staff association, there is always room for edge cases, even if you do almost everything right. You could have a well-organized plan and a website locked down like Fort Knox, but things happen.
What if, for example, a phishing email hits your finance department? You can tell your team to be careful until you’re blue in the face, but those messages are designed to fool your employees—and a simple misread of a dangerous message means that you’ve fallen for the bait.
As I noted in a blog post last year about reported weaknesses in the two-factor authentication system, even things seen as secure are fallible. Holes will always be poked even in the most secure infrastructures, and small mistakes or signs of decay can unwittingly bring a system to its knees.
The Cloudflare bug highlighted something that was working perfectly even as the broader system broke: The company had effective systems, both organizationally and technologically, that could respond to the problem.
That doesn’t discount the seriousness of the problem uncovered. It does, however, take some of the sting out of it. You can trust that Cloudflare is working to do the right thing—even if a problem arises.
That’s a reputation that your association, with its broad array of member data and sizable online presence, should strive for.