Cloudflare became a widely used content delivery network (CDN) over the past few years by offering attractive pricing and features such as DDoS protection. Many popular sites utilize Cloudflare for its feature set as well as its ease of integration. For many content management platforms (e.g. WordPress, Joomla and Drupal), it is trivial to integrate. Recently, a gaping security hole was found in Cloudflare which means you need to go update your passwords.
What is the Cloudflare Security Issue?
At its essence, the Cloudflare security issue was an issue with a parser where a memory leak could make public sensitive information. Some of this information was so public it was even being indexed by Google and other search engines. This, combined with the popularity of Cloudflare for handling sensitive transactional data, means that this breach got a nickname of “Cloudbleed” much like the venerable OpenSSL Heartbleed bug from years ago.
You can read more about the Cloudflare issues on their official blog post. We recommend reading that post.
How Widespread is the Cloudflare Issue?
Very. From what we gather, all Cloudflare proxy customers are vulnerable. In terms of official numbers, the company’s blog post states 1 in every 3,300,000 HTTP requests was impacted. When you have a massive number of requests, having a 0.00003% rate of requests being impacted means that a huge number of requests were hit. Note, this is not a 0.00003% chance of a leak, this is a 0.00003% are being claimed to actually have been leaked.
Just to see how massive it is, you can see a list of top 10,000 worldwide sites (per Alexa) that use Cloudflare here. The site also has links to tools so you can check millions of sites by name instead of downloading the entire list.
What Should You Do?
We are going to suggest it is time to change passwords. At STH we do not use Cloudflare as a CDN and therefore are not directly impacted. At the same time, many popular sites our readers likely visit were impacted. It is best to take this as an opportunity to update passwords. Also, this is being used as a potential nudge to start using two-factor authentication on any important accounts.
It’s not just passwords that are at risk. The leaked data often included POST data and if you were unlucky enough to be signing up for a site/service when your POST data was leaked your credit card details and anything else included on the page would be visible. One of the scrapes posted by Tavis Ormandy’s (the Google researcher who reported the bug) included an OkCupid user answering questions about sexual preferences (Ormandy heavily redacted it of course).
With that said… it’s russian roulette. While everyone should err on the side of caution, the odds that any given user’s data (your data) was compromised is statistically pretty slim. The data that was returned was random as was the site it was return from. e.g. issuing a request that triggered the error to Uber might return data from Uber or from completely unrelated sites and there was no assurances that the returned data would actually contain anything useful. Beyond that, the site had to be using one of the three services listed in the CloudFlare RCA. i.e. this wasn’t a full on breach where an attacker could pull targeted data at a whim, they’d have to repeatedly scrape and parse, scrape and parse. To fully exploit this an attacker would need to make as many requests as possible and store the data somewhere for analysis afterwards. I’d almost compare it to mining bitcoins. I’m not trying to downplay it, just put it in proper context.
As a CloudFlare user, I was encouraged by their response. They responded to Ormandy almost immediately. They disabled the services that were causing leak as soon as they figured out what was going on and had it completely fixed less than 24 hours after Ormandy/Google reported it. They were also very open about the cause and the remediation. It’s a stark contrast to how other companies have handled issues of similar magnitude and a great example of how it should be handled.
BTW, an easy way to tell if a site you’re using is proxied by CloudFlare and you might be at risk is to install a browser extension like Wapplyzer which will give you a breakdown of what CDNs, server software and frameworks a site is utilizing. They’re not 100% accurate, especially if the site is actively hiding/obfuscating details (like not reporting web server version) but identifying CF proxied sites is pretty easy if you know what to look for.
This could’ve been exploited by any interesting party by signing up with CloudFlare, and placing a specifically crafted file that would be served by CloudFlare by the same node as the target’s. You just then your page over and over again, and as a side-effect get a nice collection of data intended for the target site. (Limited to geographic region, because those “frontends” or “leaf nodes” are hit by users close to them.)
So it’s not only “one stray password” or “one answer from a single-matching page”. CloudFlare has no insight into how often, how systematically, and who has exploited this to amass a collection of data.
As it is well-known that they run “application level filters” and firewalls, and change served contents to optimize them for their clients, a competent interested party did well to look into any flaws and bugs. It’s nothing arcane Travis stumbled upon here, no hard-to-trigger bug in a specialized niche.
CF addressed this professionally and in a commendable way. Bugs happen, and something like this will likely happen again.
You on the other hand should not (a) entrust any secrets to a third party, including CDNs run by third parties; so design your systems around this! Sensitive data should go to (and from!) your systems and yours only. And (b) be sceptical if a service provider plays TLS-MITM for half the internet. And (c) ask if there’s a bug bounty scheme in place and if it is serious. Someone who discovered a glitch is more likely to sit on it or sell it to someone else than go through the hassle of writing it up (more so if it’s not his native language) and defending the case before a vendor that will award him—best-case—some ballpens, pair of socks, three AAA batteries and/or a US-size t-shirt.
Mark. You can’t select the nodes that are servicing your site/page. Clodflare dynamically picks them based on geographic region and load. It’s also not possible to determine a targets node for the same reason. Beyond that the external iPs aren’t bound to a specific machine, they just load balance among the farm at that node/pop. So the idea that you could target a specific site is not correct. You would end up getting a lot of noise from every site service by that node/pop. You most certainly would get data from the site you wanted but it would not be every time. You’d get what ever was last in the memory space that’s being overrun on the server that handled the request. There are people claiming they have and did but they haven’t offered any proof and the detail offered by CF and the nature of their service/infrastructure just don’t allow for that.
I’m not trying to downplay it, it’s a very serious bug which is why Google basically hunted down CF staff to report it immediately. It’s just important that everyone impacted is very clear on what happened and why.
Mike, I am well aware that I cannot just select a single node and force it to serve the unrelated target’s traffic and mine (the specifically crafted one to trigger the memory leak). But you can dramatically increase odds by targeting a geographic region of interest and loop through alternate requests to the machines (floating IPs etc.) to increase the odds to fish something, dramatically so. You’re correct, of course there will be noise.
I’d love to go into details about how CDNs are set up, and the possible structures. Just chat me up on Twitter or email. I’ve some experience in this field.
On a unrelated note, I am looking forward to CF’s move from their fork of Nginx+LUA to whatever they build on top of their fork of Go’s TLS stack. (The latter is being audited these days they said.) Although that will mitigate memory leaks like we’ve experienced this week, Go does not use most of Linux’ attack mitigation mechanisms (ASLR…) and does not implement any of its own (no stack canaries…), and holds the occasional unfortunate timing flaw (math.bigint) which could bite us in future if they or Go’s community does not address it properly.