 Twitter Fail, image by hellvetica |
At a presentation I gave at an International Terrorism and Intelligence conference, I discussed how Twitter is an obvious DDoS target. Well about 30 days later they’re in the thick of it.
Twitter is down and their network has clear signs of massive failure. In the several hundred (if not more) cases of DDoS I have had experience with, this looks like a very clear case of an attack. Congestion is a very clear sign of a DDoS attack. In this case you will see on a traceroute clean hops up to the last few, where the network starts to get congested. Basically that means each step of the network is clean until things concentrate at the end. The assumption is the congestion is caused by DDoS and not a system administrator creating a routing loop or something whacky like that. They also only appear to have ONE network provider (NTT), which is rather insane these days. It also makes targeting Twitter a much less complicated task. Using very basic tools it is possible to see that the congestion on their network is rather extreme. It’s possible to deduce that the congestion is probably due to a DDoS attack. |
DDoS Clue 1: UDP now blocked
6 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 21.497 ms 18.386 ms 19.277 ms
7 128.121.150.245 (128.121.150.245) 19.289 ms 20.950 ms 17.331 ms
8 * * *
9 128.121.150.245 (128.121.150.245) 20.178 ms !X * *
10 128.121.150.245 (128.121.150.245) 20.731 ms !X * *
11 128.121.150.245 (128.121.150.245) 19.777 ms !X * *
12 128.121.150.245 (128.121.150.245) 27.217 ms !X * *
13 * 128.121.150.245 (128.121.150.245) 24.115 ms !X *
14 * * *
The !X in the traceroute tells us that someone has placed an ACL or a filter to block certain types of traffic. In this case the traffic they are blocking is UDP, which is what traceroute generates to test each hop.
DDoS Clue 2: Massive and erratic latency
When you look at a TCP data flow, with a tool like tcptraceroute, it’s possible to get a little deeper into the twitter network. You can see easily that there’s something very wrong at hop 6, where it goes from 10ms to over 700ms.
This is really strong evidence that someone is attacking Twitter:
4 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 5.471 ms 10.941 ms 10.987 ms
5 128.121.150.133 (128.121.150.133) 10.988 ms 10.050 ms 10.988 ms
6 128.121.146.165 (128.121.146.165) 713.595 ms 1927.409 ms 1954.990 ms
One step further you can see the ICMP data is also showing massive struggle with the upstream network:
— twitter.com ping statistics —
248 packets transmitted, 68 packets received, 72.6% packet loss
round-trip min/avg/max/stddev = 1.080/424.280/2216.415/625.497 ms
This shows that the max response time has been 2.2 seconds (should be in milliseconds) and that the average is almost half a second. In my experience, this is very clear evidence of DDoS.
UPDATE 1: Twitter’s status page is reporting DDoS
Apparently they operate segmented networks, thus the www.twitter.com servers and load balancers are different than search.twitter.com which is different from status.twitter.com. Both status.twitter.com and search.twitter.com are up, I would assume also some of their API stuff is up, here’s what they say on status.twitter.com right now:
“Ongoing denial-of-service attack 4 minutes ago
We are defending against a denial-of-service attack, and will update status again shortly.Site is down 1 hour ago
We are determining the cause and will provide an update shortly.Update: we are defending against a denial-of-service attack.”
UPDATE 2: Twitter is struggling
The site continues to bounce up and down, it’s pretty clear they are trying to use DDoS mitigation techniques. The technique I see right now is a HTTP redirect with the assumption that Bots do not follow redirects:
GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020
The assumption is that a real web browser will follow the new location cookie (“?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed a bot.
This is a good, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks. There may have been too many cooks in the kitchen.
Thank you for your insight on this. I was doing some investigation also using traceroute and came up with the same guess. Looks like you were right!
Great write-up. You’ve got some interesting posts on your blog. FYI – I came over here from your comment in TechCrunch.
Nice write up I just saw post about it on MWD
http://www.mwd.com about this whole DDoS attack..
Your deep observations are very interesting! Can I translate this text and publish in my blog?
Thanks.
Hi Dave, sure… I am happy for people to learn about this stuff. If you post it on your blog, please be as kind and link back. Cheers
Meh… You were right, but only because DDoS is infinitely more likely than stupidity or hardware failure. No investigation required… nor does your investigation really differentiate between the available options.
Only the likelihood allows you to draw your conclusions, not any ‘information’ you may have gathered.
Found that delete button yet?
So you think DDoS is more likely than a hardware misconfiguration, software upgrade fail, or a network outage? I think you’re rather wrong buddy…
I’d be happy to dig further if I had access to more data.
Cheers
Barrett, they’re attacking facebook too. I can’t believe someone do this kind of thing for fun, what about you?
Makes sense, however, Facebook has their shit together. The guys that are building out Facebook’s infrastructure are seasoned engineers. Are there any reports of major Facebook issues at the moment?
I’m pretty sure these attacks are forms of censorship, however, someone could also do this for the ‘lulz’. DDoS attacks are not hard to do, so it could just be a 15 y/o kid that’s pissed about someone’s Tweet. There’s no real way to know without more information.
Iran?
Iran needs ‘lulz’ so yeah, they are the prize culprits – I can just see the mullahs now huddled around the Koran and finding a section they can interpret into meaning “Allah (peace be upon him) has decreed that all social networking sites are a form of worshipping false gods”. They have now issued a fatwa proclaiming Jihad against Twitter, Facebook and Bebo. Look out for Bin Laden cancelling his @BinLadenRulz twitter account asap.
[...] e fazendo bagunça. Traduzido e modificado por mim com uma pressa absurda. Versão original em Verbophobia:Twitter down due to DDoS por Barrett [...]
Hey Barrett, what about Gawker.com? It has entered in a failure at the same time of Twitter. Is there any connections between these outages? Maybe Gawker.com is using NTT link too?
Cheers.
I don’t think NTT had any problems on their network due to this. Also, Gawker uses Internap for bandwidth, and is hosted in NYC vs. California.
[...] I’ve seen a number of great posts about these attacks that I wanted to share. Take a look at this blog post by Barrett Lyon on the security aspects of this attack on Twitter: http://www.blyon.com/blog/index.php/2009/08/06/twitter-down-due-to-ddos/ [...]
No doubt about your research, it doesn’t take a rocket scientist to see this but it does help to know a little about computers.
Are there any indication for the origin of the packets? – I personally think it would be sourced by some of the Russian networks, i.e. RBN control points. These are readily available if you have the money to pay for them, and there’s a lot of them out there.
We constantly see these botnets grow exponentially and with still more and more flaws in OS’s being exposed, I don’t foresee a change in this trend.
I have a feeling the real source of this is Iran/North Korea/China – i.e. “axis of evil” that tries to bully the west.
Is there any way “we” can use this to our advantage. I.e. can someone infect the government of “axis of Evil” with exploits that will then propagate to the various gov computers to allow unrestricted network access to/from the outside?
There has to be some internal computers that have viral/bot infections that “we” can use to start the investigation into the networks and return the favor.
Sorry but i find some of the panic hilarious. OH NO IM not going to be able to tweet today my life is ruined! But in all seriousness i hope they get it blocked out
Wade, would you feel differently if you could not do online banking?
[...] Lyons writes on bylon.com how it could be a DDoS attack: At a presentation I gave at an International Terrorism and [...]
Hey Berret.. Thanks for a nice writeup on twitters DDoS mitigation FAIL.
Why is it that people have to be burned sometimes badly and repeatedly to learn…?
Do we know if this was a web based DDOS or DNS related attack? I would imagine for a hacker it is easier to DDOS DNS zones rather than the content servers, no?
The guys at Twiier should realize that they are in a different place than they were some 6 months ago. Since the, the popularity of Twitter has grown so much that they should have been prepared for this, and should have built in a lot of preventative measures like avoiding single points of failure (NTT alone?), and building up multiple levels of redundancy. I would fire their CTO if I were the boss. Harsh, huh?
[...] Twitter down due to DDoS [...]
[...] Twitter ddos http://www.blyon.com/blog/index.php/2009/08/06/twitter-down-due-to-ddos/ [...]
[...] Facebook has also been giving me problems for a few days, so my guess is that there is another attack happening . . . of course that is just [...]