Twitter Fail, image by hellvetica |
At a presentation I gave at an International Terrorism and Intelligence conference, I discussed how Twitter is an obvious DDoS target. Well about 30 days later they’re in the thick of it.
Twitter is down and their network has clear signs of massive failure. In the several hundred (if not more) cases of DDoS I have had experience with, this looks like a very clear case of an attack. Congestion is a very clear sign of a DDoS attack. In this case you will see on a traceroute clean hops up to the last few, where the network starts to get congested. Basically that means each step of the network is clean until things concentrate at the end. The assumption is the congestion is caused by DDoS and not a system administrator creating a routing loop or something whacky like that. They also only appear to have ONE network provider (NTT), which is rather insane these days. It also makes targeting Twitter a much less complicated task. Using very basic tools it is possible to see that the congestion on their network is rather extreme. It’s possible to deduce that the congestion is probably due to a DDoS attack. |
DDoS Clue 1: UDP now blocked
6 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 21.497 ms 18.386 ms 19.277 ms
7 128.121.150.245 (128.121.150.245) 19.289 ms 20.950 ms 17.331 ms
8 * * *
9 128.121.150.245 (128.121.150.245) 20.178 ms !X * *
10 128.121.150.245 (128.121.150.245) 20.731 ms !X * *
11 128.121.150.245 (128.121.150.245) 19.777 ms !X * *
12 128.121.150.245 (128.121.150.245) 27.217 ms !X * *
13 * 128.121.150.245 (128.121.150.245) 24.115 ms !X *
14 * * *
The !X in the traceroute tells us that someone has placed an ACL or a filter to block certain types of traffic. In this case the traffic they are blocking is UDP, which is what traceroute generates to test each hop.
DDoS Clue 2: Massive and erratic latency
When you look at a TCP data flow, with a tool like tcptraceroute, it’s possible to get a little deeper into the twitter network. You can see easily that there’s something very wrong at hop 6, where it goes from 10ms to over 700ms.
This is really strong evidence that someone is attacking Twitter:
4 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 5.471 ms 10.941 ms 10.987 ms
5 128.121.150.133 (128.121.150.133) 10.988 ms 10.050 ms 10.988 ms
6 128.121.146.165 (128.121.146.165) 713.595 ms 1927.409 ms 1954.990 ms
One step further you can see the ICMP data is also showing massive struggle with the upstream network:
— twitter.com ping statistics —
248 packets transmitted, 68 packets received, 72.6% packet loss
round-trip min/avg/max/stddev = 1.080/424.280/2216.415/625.497 ms
This shows that the max response time has been 2.2 seconds (should be in milliseconds) and that the average is almost half a second. In my experience, this is very clear evidence of DDoS.
UPDATE 1: Twitter’s status page is reporting DDoS
Apparently they operate segmented networks, thus the www.twitter.com servers and load balancers are different than search.twitter.com which is different from status.twitter.com. Both status.twitter.com and search.twitter.com are up, I would assume also some of their API stuff is up, here’s what they say on status.twitter.com right now:
“Ongoing denial-of-service attack 4 minutes ago
We are defending against a denial-of-service attack, and will update status again shortly.Site is down 1 hour ago
We are determining the cause and will provide an update shortly.Update: we are defending against a denial-of-service attack.”
UPDATE 2: Twitter is struggling
The site continues to bounce up and down, it’s pretty clear they are trying to use DDoS mitigation techniques. The technique I see right now is a HTTP redirect with the assumption that Bots do not follow redirects:
GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020
The assumption is that a real web browser will follow the new location cookie (“?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed a bot.
This is a good, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks. There may have been too many cooks in the kitchen.





