Archive for August, 2009

IP Crunch Time!

Saturday, August 22nd, 2009

Some people have noticed I have slowed down on my blog a bit. I’m sorry! It’s true, I have put writing on the side burner (for a couple of weeks) while I complete four more patent applications.

I’m an idea factory, without a specific employer to absorb my steady stream of thoughts and energy, I decided to create my own intellectual property. What will I do with it? The answer to that question will be rather interesting but far in the future.

I’ll be back here writing soon! I promise! Meanwhile, here’s a link to my DDoS mitigation patent that was granted recently: US 20060075491A1

Tags:
Posted in Opinion | No Comments »

Major Cable Cuts in Asia Pacific

Wednesday, August 12th, 2009

Sometime last night there were a series of major submarine cable cuts to the Asia Pacific region, somewhere around Taiwan.

These cable systems are important for the communication of the Asia Pacific region, if you live in Taiwan or Hong Kong, the Internet today may not be as snappy as you’re used to. Luckily, so far there are no reports of complete outage.

The cuts so far are on the cable systems: C2C, APCN2, and EAC. I’ve put together a very basic map (accurate enough to get a general feel) of the confirmed cuts:


Background Map Provided by Google Maps

There are other reports of issues on the other major systems SMW2 and SMW3. Both SMW2 and SMW3 were cut December 19th 2008 and were down for over a week. However, current reports about these systems may be due to traffic congestion (load from the other systems failing over).

The remaining systems, TGN and FLAG cable systems also are running well as of now.

So far there are no reports on the cause of the damage to the systems, yet the speculation is typhoon Morakot. I wish the men and women working on routing around them and fixing them good luck!

Update: Suspected cause

Debris flows damage undersea cables off southeastern Taiwan
Taipei, Aug. 12 (CNA) Deepsea debris flows off southeastern Taiwan, which were triggered by Typhoon Morakot, seriously damaged undersea cables and disrupted Internet services between Taiwan, China and Southeast Asia, according to Taiwan’s main telecommunications company.”

Update: Intra-Asia Cable (IAC)

As another important note, The Intra-Asia Cable (IAC) system from Tata Communications (the only system in the region which has planned a route not passing thru the earthquake prone region around Taiwan) was also one of the few systems up and running in the region the entire time.

To learn more, check out these links:

Tags: ,
Posted in Cable Systems | 1 Comment »

More Twitter Woes?

Tuesday, August 11th, 2009

Around noon today, Twitter had some stutters and on their status page they wrote:

We’re working to recover from a site outage and will update as we learn more.

Update (12:17p): We’re back up and analyzing the traffic data to determine the nature of this attack.

I am guessing as time goes on, the attacks to Twitter will be much more targeted. Attackers can update their bots to attack specific user accounts or other process-intensive working parts of Twitter which could create processing bottlenecks within the internal Twitter application itself.

It is very difficult to build large scaling databases, judging by Twitter’s current network design, I would assume their database design would be similar in stature — lack luster.

Rather than just basic attacks to Twitter’s hosting partner NTT, attackers will target the Twitter database/application weak spots via their API or via their web interface. A targeted attack on Twitter’s own application weaknesses would bring more bang for the buck and be more difficult to defend against.

As seen in the image on the left, some evidence of application fail is apparent. Twitter’s web site is doing some odd stuff; at times my followers/following stats and the trending topics applet disappear.

The strange behavior myself and others are seeing could be caused by more targeted application layer attacks. Of course this is speculation. However, something is not working right over there and people are attacking them.

Tags: , , ,
Posted in DDoS, Opinion, Security, Twitter Research, Uncategorized | 2 Comments »

Twitter’s Hosting Illustrated: F*ckyeahboobies.com

Thursday, August 6th, 2009
Today was an interesting day, it started with a DDoS attack to Twitter and is ending with boobies.

Richard Stiennon (over at Threat Chaos) and I started looking into the attacks and dug a little deeper into Twitter’s architecture.

We noticed that the status.twitter.com page did not go down. That’s because it’s hosted in RackSpace (far away from their web servers) on some guy’s computer. What’s fun about that (as Stiennon noticed) it also hosts Fuckyeahboobies.com (WARNING: link NSFW). Yes, that’s right, Twitter’s corporate status page is also the same server that hosts Fuckyeahboobies.com.

www.fuckyeahboobies.com. 3600 IN CNAME fuckyeahboobies.com.
fuckyeahboobies.com. 3600 IN A 72.32.231.8

;; ANSWER SECTION:
status.twitter.com. 60 IN A 72.32.231.8

Let’s be fair, hosting the status page away from Twitter’s hosting infrastructure is a very good idea. However, mixing it with 14,476 other hosted sites on the same machine may not be so bright. Those other sites can attract problems. If someone were to hack the group hosted machine and modify the status.twitter.com page, it could be harmful to the value of Twitter. One needs to ask the question: “Does this bring value to Twitter? Does saving a few thousand a year in hosting cost outweigh the risks?” Personally, I would expect a company that has $55 million in funding to strive towards world class design. At this point it’s not bringing value to their shareholders by cutting corners on critical infrastructure.

Anyhoo, over the morning, I constructed this basic diagram of Twitter’s hosting architecture (below) to help people understand what happened during their DDoS attacks. I must admit, it started out well with the boobies site, but when I started looking into their network I was a bit disappointed. They have a rather flat network that appears to be completely managed by NTT/Verio.

The sections in red are the paths that the DDoS would have taken. I would guess something in their load balancing farm was not configured to deal with the attack or this would have just been absorbed without much notice. The upstream routers were doing just fine when I ran tests during the DDoS attack. I get the feeling that their load balancers are doing most of the request validation.

When you look at the TCP handshakes for www.twitter.com, it responds on any port, which indicates they are running some sort of blind syn cookies (weeds out spoofed SYN floods). Again, I am assuming their load balancer or an upstream firewall is doing this. Along with their SYN cookies, they are doing a 302 redirect cookie (web server lingo for, “hey get the page over here instead”):

GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020

The assumption is that a real web browser will follow the new location cookie (”?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed to be a bot and denied access to the site. In some cases I have seen this setup eventually put repeat offenders on a blacklist, this could be determinate to Twitter.

This is a cleaver DDoS defense mechanism, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script, Tweetdeck, Power Twitter, Seesmic, and the list goes on. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks.

It’s pretty clear they are ready for a redesign. They need their own autonomous network, bring in bandwidth from many different providers, and have several layers of security. Building a strong ACL border and a nice mitigation layer would make a lot of sense. I also don’t think abandoning the shared cloud services model is a good idea, but having control over the heart of the operation makes sense.

Facebook has been doing a much better job at this as they were not crippled from the same attacks.

It just goes to show, there needs to be a blend of cloud services (like AWS) along with good in-house design.

UPDATE: 302 redirects were countermeasure fail

Posted on Google Developer Talk, Twitter developer Alex Payne stated:

Alex Payne
Aug 11, 5:00 pm
We’re aware of these issues; sorry.
Our ops team tells me that the countermeasures that are being put in place
should not cause the 302 redirect behavior that impacted OAuth and other
services late last week. If you’re seeing that behavior, please post here
and we’ll coordinate with them to eliminate it.

Tags: , , ,
Posted in DDoS, Hackers, Opinion, Security, Twitter Research | 8 Comments »

Twitter down due to DDoS

Thursday, August 6th, 2009
Twitter Fail, image by hellvetica At a presentation I gave at an International Terrorism and Intelligence conference, I discussed how Twitter is an obvious DDoS target. Well about 30 days later they’re in the thick of it.

Twitter is down and their network has clear signs of massive failure. In the several hundred (if not more) cases of DDoS I have had experience with, this looks like a very clear case of an attack.

Congestion is a very clear sign of a DDoS attack. In this case you will see on a traceroute clean hops up to the last few, where the network starts to get congested. Basically that means each step of the network is clean until things concentrate at the end.

The assumption is the congestion is caused by DDoS and not a system administrator creating a routing loop or something whacky like that.

They also only appear to have ONE network provider (NTT), which is rather insane these days. It also makes targeting Twitter a much less complicated task.

Using very basic tools it is possible to see that the congestion on their network is rather extreme. It’s possible to deduce that the congestion is probably due to a DDoS attack.

DDoS Clue 1: UDP now blocked

6 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 21.497 ms 18.386 ms 19.277 ms
7 128.121.150.245 (128.121.150.245) 19.289 ms 20.950 ms 17.331 ms
8 * * *
9 128.121.150.245 (128.121.150.245) 20.178 ms !X * *
10 128.121.150.245 (128.121.150.245) 20.731 ms !X * *
11 128.121.150.245 (128.121.150.245) 19.777 ms !X * *
12 128.121.150.245 (128.121.150.245) 27.217 ms !X * *
13 * 128.121.150.245 (128.121.150.245) 24.115 ms !X *
14 * * *

The !X in the traceroute tells us that someone has placed an ACL or a filter to block certain types of traffic. In this case the traffic they are blocking is UDP, which is what traceroute generates to test each hop.

DDoS Clue 2: Massive and erratic latency

When you look at a TCP data flow, with a tool like tcptraceroute, it’s possible to get a little deeper into the twitter network. You can see easily that there’s something very wrong at hop 6, where it goes from 10ms to over 700ms.

This is really strong evidence that someone is attacking Twitter:

4 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 5.471 ms 10.941 ms 10.987 ms
5 128.121.150.133 (128.121.150.133) 10.988 ms 10.050 ms 10.988 ms
6 128.121.146.165 (128.121.146.165) 713.595 ms 1927.409 ms 1954.990 ms

One step further you can see the ICMP data is also showing massive struggle with the upstream network:

— twitter.com ping statistics —
248 packets transmitted, 68 packets received, 72.6% packet loss
round-trip min/avg/max/stddev = 1.080/424.280/2216.415/625.497 ms

This shows that the max response time has been 2.2 seconds (should be in milliseconds) and that the average is almost half a second. In my experience, this is very clear evidence of DDoS.

UPDATE 1: Twitter’s status page is reporting DDoS

Apparently they operate segmented networks, thus the www.twitter.com servers and load balancers are different than search.twitter.com which is different from status.twitter.com. Both status.twitter.com and search.twitter.com are up, I would assume also some of their API stuff is up, here’s what they say on status.twitter.com right now:

“Ongoing denial-of-service attack 4 minutes ago
We are defending against a denial-of-service attack, and will update status again shortly.

Site is down 1 hour ago
We are determining the cause and will provide an update shortly.

Update: we are defending against a denial-of-service attack.”

UPDATE 2: Twitter is struggling

The site continues to bounce up and down, it’s pretty clear they are trying to use DDoS mitigation techniques. The technique I see right now is a HTTP redirect with the assumption that Bots do not follow redirects:

GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020

The assumption is that a real web browser will follow the new location cookie (“?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed a bot.

This is a good, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks. There may have been too many cooks in the kitchen.

Tags: , ,
Posted in DDoS, Hackers, Security, Twitter Research | 25 Comments »

The Google Bubble

Sunday, August 2nd, 2009
TechCrunch just released an article about their ad revenue and openly discussed the current online ad recession.

The amount of money advertisers are willing to pay for a keyword (you know, search for the word “Disney” and you get ads related to the keyword “Disney”) has been drastically reduced.

The less advertisers pay for keywords, the less people make from online advertising. Basic stuff, the cause for weakness in keyword pricing may be due to what I call, “The Google Bubble”.

I have been fascinated about the concept of a bubble created by the very nature of how Google AdSense operates: A keyword starts at a minimal price, you purchase that keyword, then a competitor comes along and starts a bidding war. The pricing on the keyword goes up and up as the competitors battle for control, until, it hits them; the price is too high — it’s not worth it for anyone. The bubble pops as advertisers decide to stop buying the expensive keywords or they simply reduce their pricing on the word, causing contraction. Simply put, the bursting bubble results in advertisers bidding drastically less for the keywords than before.

The bubble theory Is loosely confirmed by reports of advertisers spending less on advertising, The Wall Street Journal writes, “U.S. search advertising spending fell 8% in the fourth quarter of 2008 from the same period in 2007″, which is significantly more than the contraction in GDP. The effect is also seen on TechCrunch’s posting, along with a spattering of other sites, which are reporting far less ad revenue. Granted, ad revenues have fallen due to general economic reasons, but it may have been exacerbated by bubble effects such as the consequences of the bidding war built into AdSense.

The housing market has the exact same bubble: when the economy is good, people bid up houses until there is a breaking point. This natural propensity means that Googlenomics will be difficult to control, but easy to predict.

If I am right, the Google Bubble will expand and contract very quickly, bidding on keywords will increase until something like a bad economy, a competitor, or fraud causes it to tip over. I also think that each time the Google Bubble pops, other ad networks will gain new advertisers and participants because keywords on the alternative ad networks may not have been through a vigorous bidding war.

Enjoy the expansion of the next bubble, but don’t be shocked when it pops.

Tags: , , , ,
Posted in Opinion, Philosophy | 3 Comments »