Today was an interesting day, it started with a DDoS attack to Twitter and is ending with boobies. Richard Stiennon (over at Threat Chaos) and I started looking into the attacks and dug a little deeper into Twitter’s architecture. We noticed that the status.twitter.com page did not go down. That’s because it’s hosted in RackSpace (far away from their web servers) on some guy’s computer. What’s fun about that (as Stiennon noticed) it also hosts Fuckyeahboobies.com (WARNING: link NSFW). Yes, that’s right, Twitter’s corporate status page is also the same server that hosts Fuckyeahboobies.com. www.fuckyeahboobies.com. 3600 IN CNAME fuckyeahboobies.com. fuckyeahboobies.com. 3600 IN A 72.32.231.8 ;; ANSWER SECTION: status.twitter.com. 60 IN A 72.32.231.8

Let’s be fair, hosting the status page away from Twitter’s hosting infrastructure is a very good idea. However, mixing it with 14,476 other hosted sites on the same machine may not be so bright. Those other sites can attract problems. If someone were to hack the group hosted machine and modify the status.twitter.com page, it could be harmful to the value of Twitter. One needs to ask the question: “Does this bring value to Twitter? Does saving a few thousand a year in hosting cost outweigh the risks?” Personally, I would expect a company that has $55 million in funding to strive towards world class design. At this point it’s not bringing value to their shareholders by cutting corners on critical infrastructure.

Anyhoo, over the morning, I constructed this basic diagram of Twitter’s hosting architecture (below) to help people understand what happened during their DDoS attacks. I must admit, it started out well with the boobies site, but when I started looking into their network I was a bit disappointed. They have a rather flat network that appears to be completely managed by NTT/Verio.

The sections in red are the paths that the DDoS would have taken. I would guess something in their load balancing farm was not configured to deal with the attack or this would have just been absorbed without much notice. The upstream routers were doing just fine when I ran tests during the DDoS attack. I get the feeling that their load balancers are doing most of the request validation.

When you look at the TCP handshakes for www.twitter.com, it responds on any port, which indicates they are running some sort of blind syn cookies (weeds out spoofed SYN floods). Again, I am assuming their load balancer or an upstream firewall is doing this. Along with their SYN cookies, they are doing a 302 redirect cookie (web server lingo for, “hey get the page over here instead”):

GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020

The assumption is that a real web browser will follow the new location cookie (”?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed to be a bot and denied access to the site. In some cases I have seen this setup eventually put repeat offenders on a blacklist, this could be determinate to Twitter.

This is a cleaver DDoS defense mechanism, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script, Tweetdeck, Power Twitter, Seesmic, and the list goes on. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks.

It’s pretty clear they are ready for a redesign. They need their own autonomous network, bring in bandwidth from many different providers, and have several layers of security. Building a strong ACL border and a nice mitigation layer would make a lot of sense. I also don’t think abandoning the shared cloud services model is a good idea, but having control over the heart of the operation makes sense.

Facebook has been doing a much better job at this as they were not crippled from the same attacks.

It just goes to show, there needs to be a blend of cloud services (like AWS) along with good in-house design.

UPDATE: 302 redirects were countermeasure fail

Posted on Google Developer Talk, Twitter developer Alex Payne stated:

Alex Payne
Aug 11, 5:00 pm
We’re aware of these issues; sorry.
Our ops team tells me that the countermeasures that are being put in place
should not cause the 302 redirect behavior that impacted OAuth and other
services late last week. If you’re seeing that behavior, please post here
and we’ll coordinate with them to eliminate it.