The Internet broke today: Facebook, Verizon, and more see major outages

Enlarge / The Internet this week, more or less (credit: torange.biz (modified))

Last week, Verizon caused a major BGP misroute that took large chunks of the Internet, including CDN company Cloudflare, partially down for a day. This week, the rest of the Internet has apparently asked Verizon to hold its beer.

Cloudflare went down again for half an hour yesterday, and this time, it was the company’s own fault—we’re still waiting on a full post-mortem, but the short version is that a firewall regular expression rule targeting malicious Javascript spiked the firewalls’ CPU usage, crippling throughput and causing widespread HTTP 502 errors. Microsoft’s Office365 also seems to have experienced a multi-hour partial outage yesterday, with the service working over some ISPs and routes but not others for about four hours.

Office365’s partial outage on July 2 was reported by third-party users at downdetector.com (credit: downdetector.com)

Facebook and its properties WhatsApp and Instagram have suffered widespread outages relating to image display for most of today. The problem seems to be bad timestamp data being fed to the company’s CDN in some image tags; when I looked into the broken images littering my own Facebook timeline, I discovered different timestamp arguments embedded in the same URLs. Loading an image from fbcdn.net with bad “oh=” and “oe=” arguments—or no arguments at all—results in an HTTP 403 “Bad URL timestamp”.