Archive for the ‘Security’ Category

The Internet is Beta

Tuesday, May 4th, 2010
Beta is an engineering way of saying “almost done” – the product is good enough to use but it’s not quite finished yet. Google often releases their new products with a cute little “BETA” logo. Gmail, the Google email system used by millions, has been in beta for five years.

Like Gmail, the Internet’s core protocol should also have had a Beta tag on it for an extended time – for the past 41 years to be precise. Generally speaking, it works pretty well, but the founding fathers of the Internet could not have anticipated that the software they were building would ever become what it is now: The infrastructure for all of society.

So it appears today that some major features were left out…but not because the people behind the design made a mistake. When MIT first used packet switching in 1965 to communicate with a remote computer in California (confirming that packet switching works), the furthest thing from anyone’s mind was security, network neutrality, network education, privacy, cyber warfare, and the slurry of problems that challenge both business and individual users of the Internet today.

In 1969, with the original workings of the Internet (ARPANET), security was simple: the network was tiny and users on the computers that were connected to it were trusted researchers. It was an open community. As Vint Cerf, one of the most notable developers of the Internet, was quote in Fatal System Error as saying, “My thought at the time, thirty-five years ago, was not to build an ultra-secure system, because I could not tell if even the basic ideas would work…We never got to do the production engineering.” The focus at the time, sensibly, was on fault tolerance, not security.
Vint Cerf – Photo by Charles Haynes

Now, nearly 41 years later, we read about Internet security issues constantly. The lack of security features in IP (Internet Protocol) has spawned entire industries, with vendors and service providers that are happy to sell you the next generation protect-all, whiz-bang software. If one were to ask a roomful of people in the security industry what they think about the security products, including their own, on the market today – if they think there are real solutions to the problems we all face – their answer would be a unified “NO”. No one thinks we are at the point where we can all just stop worrying about security.


Barack Obama
Courtesy The White House
The disturbing fact is that the engine that enables our modern global economy is based on a really cool experiment that was not designed for security. Risks can be reduced, but the naughty truth is that the ‘Net is not a secure place for business or society.

The role that the Internet plays in our economy places it in the category of a critical resource that the government must protect – just as it does our water supply and the national power grid. A threat to Internet security is a threat to national security. In May 2009, President Obama spoke about this issue and the plan his administration has to address it. He stated that the US is “not as prepared as it should be” to defend against cyber threats and he proposed new “digital infrastructure” initiatives to “ensure that these networks are secure, trustworthy and resilient.”

But can the US Government, or any other governing authority, ever adequately protect and defend the Internet? How can that be done if the Internet Protocol itself was not designed to, in Obama’s words, “deter, prevent, detect, and defend against attacks”?

Given the world economy’s substantial dependence on the Internet, wouldn’t it make sense to create a well-funded think-tank with the brightest minds in society to design a new protocol with a new vision? This time when we start the process, we will have the benefit of 41 years of Internet beta testing and we can rethink the vision to also include things such as:

  • Security: Transmitting data safely but easily without special software.
  • Privacy: Balancing anonymity and accountability. Allowing people to communicate freely but ensuring accountability to protect against abuses and criminal activity.
  • Routing Intelligence: Routing data without neutrality issues and allowing the protocol itself to route traffic based on a myriad of metrics, conditions, agreements, and other factors.
  • Enculturation and Education: Bringing new people (children, emerging nations, etc) onto the network with a step approach to ensure that they learn about network culture and functionality before they make mistakes.

I don’t think any of us who are involved with cyber security on a professional level can see the Internet as it is today functioning successfully for the next 50 years. I can envision a world of networking much different than today’s. So why not start turning the ship now?

Is designing a better protocol difficult? Yes. Can it be done? Absolutely!

I will be writing more on this topic in the coming months. Stay tuned.

The Top 10 Things To Do While Under DDoS Attack

Sunday, January 24th, 2010

In my past decade-plus dealing with distributed denial-of-service attacks, I have noticed a few patterns in the way that companies handle these attacks. Usually when an unprepared virgin company is first attacked, all hell breaks loose. The lack of preparedness causes several chain reactions that make the situation worse. Addressing these most common mistakes ahead of time can help a situation tremendously.

When someone calls me for advice, the first few items I go over have nothing to do with fixing the attack. I’m giving advice that I think is common sense, and I’ve been surprised that others don’t find it obvious.

Here are my Top 10 To-do’s for making life less painful during an attack.

1. Don’t Panic

While the network and your services are exploding and bouncing offline, there must be someone that is comfortable enough to make good decisions. I’ve seen managers freak out and threaten everyone with the prospect of the company collapsing. I think they were trying to motivate people to figure out some solution, but they ended up creating more chaos during an already tough situation.

Once I saw employees hastily rip out the network’s firewalls and re-configure the load balancers. They ended up creating more mess than they had before because they were reacting to an angry and stressed manager.

You are going to create a disaster if you approach with a sledgehammer and wishes. Don’t let anyone make quick changes; try to follow your company’s policies. Sit back, analyze the problem, isolate the actual device that’s failing in the chain, and make an informed–and usually small–adjustment.

If you’re in the 10th hour and things don’t seem to be improving, gather everyone, go away from the office, have a beer, relax for 15 minutes, and talk about something positive. The information flow after that beer might just save you and motivate everyone to do a good job – the solution will come!

2. Create a contact list of external email addresses and phone numbers.

This one is sadistically funny. Most companies host their email, VoIP system, IRC, Wiki, databases, primary storage, etc. all in the same colocation behind the same network connection that hosts their web sites and services. This is, for lack of better words, stupid. All of your digital eggs are in one basket, and that basket is also holding a grenade. A DDoS attack ends up crippling the company’s infrastructure, leaving it with no phones, email, or any communications structure whatsoever.

I’ve seen CEOs of massive companies using their hotmail account and cell phone to contact me because it was their only way of communicating from their multi-million dollar offices.

If you insist on being an “eggs in one basket” company, keep a list of vital email accounts and cell phone numbers on a notepad. That way you can at least call your IT person when everything is down.

3. Setup a “War Room”

Convert your conference room into a war room. Get everyone that has influence in the company in that room. This includes marketing, IT, the CEO, etc. It ensures everyone is on the same page, leaders can lead, and everyone can be in sync.

I typically fill the room with a constant flow of healthy snacks, coffee, and other beverages. If you don’t have anything like that handy, order pizza immediately or send someone shopping.

4. Get one of your guys to the colo ASAP

If you are offline due to DDoS attack, chances are your IT staff cannot log in to the remotely hosted hardware in your datacenters. The easy solution is to physically get them there. They can console in to the hardware and actually see what is going wrong. It’s not fun, but it will result in a much faster resolution to the problem (Make sure they have folding chairs, cash for the vending machines, and serial cables).

5. Find an old hub

Yes, I said hub. You know, those old things that cause collisions? If you’re dealing with an attack and yours is like a lot of companies, it may be difficult for you to set up a traffic monitoring port on your main routers. Assuming you’re setup with Ethernet, at least you can bridge a hub in-line and connect a laptop to the hub and sniff or analyze the traffic!

This is key because having eyes into the data stream really helps figure out how to filter it. Pulling random cables and shutting down random services is not the solution. Make an informed call because you were thoughtful enough to have a hub or SPAN/Mirror port pre-configured.

6. Understand the nature of the attack

There’s a reason you are the target for this attack. Obviously there are a lot of reasons for any given attack, yet understanding the attacker’s motivation is key to creating a better defense strategy.

In the field I have observed a very strange phenomenon; the people working at a victim company usually have a gut feeling about why they are being attacked. So far, their gut instinct has been correct.

Some people know they are being extorted and some people feel it’s a competitor trying to shut them down. Others have a customer that has pissed someone off so the attacker takes down the whole company just to silence one customer. Maybe shutting down the attacker’s target for awhile may actually save the entire ship. Go with your gut on this, make a hypothesis and test it.

7. Document everything

Your business was just smacked around by some bad guys, but what proof do you have? If you don’t have any, then what do you think the law enforcement is going to do for you?

During the attack, lock down all your logs and assign someone within the company to be the custodian of the records. Save server logs, web logs, email logs, any packet capture, network graphs, reports – anything – including a timeline of events.

8. Call your ISP

Your ISP can help, however they have a process to follow. The process usually requires a ticket escalation requirement before you can get real help. If you call early in the attack and open a ticket, that can help you when you really need someone.

Your ISP also has hardware that may be capable of filtering or rate-limiting the attack. The more you know about the attack and you can point them in the right direction, the more they can help you.

They may also suggest you to sign up for their DDoS protection system. Don’t do that right away; reserve that until you are out of all other options. If you do sign up, make sure there is a service level agreement. In the meantime, there are a number of free services you can request:

Null routing of the target IP address
Router ACLs of the top attacking source addresses
New IP addresses
Detailed traffic reports

If you can find the guru at the ISP that knows how to fix these problems, that might be time well spent.

9. Setup “We are down” web hosting services

If the attack is running longer than you had anticipated and you don’t have a solution in sight, you could get your site working at least enough to communicate to your customers.

There are web-hosting companies, which as part of what they do, provide DDoS service level agreements. For a small amount of money you could quickly sign up with several of these companies, upload a “Sorry we’re down, but contact us here” page, and flip your DNS to the cluster of hosted servers.

Your customers will have more confidence in your performance and the attackers may get bored because the attack has not completely shut everything down. If this plan doesn’t work, at least you have diverted some of the attack away from your network.

10. Learn from the event

Post attack can be a blur; everyone is exhausted and burnt out. Mostly, everyone just wants the day-to-day atmosphere to return to status quo. Well, if you’ve been attacked and you did not learn and improve your strategy on how to deal with future attacks, then you are not doing your job.

You should start a review the very day after, while everything is fresh, and make sure that everyone is prepared. Go over what worked, what did not work, and how to improve your system’s overall technology.

Spend the money to fix things properly. Don’t just duct-tape it.

RSA 2010

Tuesday, December 15th, 2009

I’m presenting at RSA 2010 with co-speaker Joseph Menn. We will be talking about the soon-to-publish book Fatal System Error, which covers some of my adventures with Cyber Warfare and cyber-extortionists.

What aspects of the adventure would you find most interesting?

Send me an email with suggestions: blyon [at] blyon.com

More Twitter Woes?

Tuesday, August 11th, 2009

Around noon today, Twitter had some stutters and on their status page they wrote:

We’re working to recover from a site outage and will update as we learn more.

Update (12:17p): We’re back up and analyzing the traffic data to determine the nature of this attack.

I am guessing as time goes on, the attacks to Twitter will be much more targeted. Attackers can update their bots to attack specific user accounts or other process-intensive working parts of Twitter which could create processing bottlenecks within the internal Twitter application itself.

It is very difficult to build large scaling databases, judging by Twitter’s current network design, I would assume their database design would be similar in stature — lack luster.

Rather than just basic attacks to Twitter’s hosting partner NTT, attackers will target the Twitter database/application weak spots via their API or via their web interface. A targeted attack on Twitter’s own application weaknesses would bring more bang for the buck and be more difficult to defend against.

As seen in the image on the left, some evidence of application fail is apparent. Twitter’s web site is doing some odd stuff; at times my followers/following stats and the trending topics applet disappear.

The strange behavior myself and others are seeing could be caused by more targeted application layer attacks. Of course this is speculation. However, something is not working right over there and people are attacking them.

Twitter’s Hosting Illustrated: F*ckyeahboobies.com

Thursday, August 6th, 2009
Today was an interesting day, it started with a DDoS attack to Twitter and is ending with boobies.

Richard Stiennon (over at Threat Chaos) and I started looking into the attacks and dug a little deeper into Twitter’s architecture.

We noticed that the status.twitter.com page did not go down. That’s because it’s hosted in RackSpace (far away from their web servers) on some guy’s computer. What’s fun about that (as Stiennon noticed) it also hosts Fuckyeahboobies.com (WARNING: link NSFW). Yes, that’s right, Twitter’s corporate status page is also the same server that hosts Fuckyeahboobies.com.

www.fuckyeahboobies.com. 3600 IN CNAME fuckyeahboobies.com.
fuckyeahboobies.com. 3600 IN A 72.32.231.8

;; ANSWER SECTION:
status.twitter.com. 60 IN A 72.32.231.8

Let’s be fair, hosting the status page away from Twitter’s hosting infrastructure is a very good idea. However, mixing it with 14,476 other hosted sites on the same machine may not be so bright. Those other sites can attract problems. If someone were to hack the group hosted machine and modify the status.twitter.com page, it could be harmful to the value of Twitter. One needs to ask the question: “Does this bring value to Twitter? Does saving a few thousand a year in hosting cost outweigh the risks?” Personally, I would expect a company that has $55 million in funding to strive towards world class design. At this point it’s not bringing value to their shareholders by cutting corners on critical infrastructure.

Anyhoo, over the morning, I constructed this basic diagram of Twitter’s hosting architecture (below) to help people understand what happened during their DDoS attacks. I must admit, it started out well with the boobies site, but when I started looking into their network I was a bit disappointed. They have a rather flat network that appears to be completely managed by NTT/Verio.

The sections in red are the paths that the DDoS would have taken. I would guess something in their load balancing farm was not configured to deal with the attack or this would have just been absorbed without much notice. The upstream routers were doing just fine when I ran tests during the DDoS attack. I get the feeling that their load balancers are doing most of the request validation.

When you look at the TCP handshakes for www.twitter.com, it responds on any port, which indicates they are running some sort of blind syn cookies (weeds out spoofed SYN floods). Again, I am assuming their load balancer or an upstream firewall is doing this. Along with their SYN cookies, they are doing a 302 redirect cookie (web server lingo for, “hey get the page over here instead”):

GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020

The assumption is that a real web browser will follow the new location cookie (”?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed to be a bot and denied access to the site. In some cases I have seen this setup eventually put repeat offenders on a blacklist, this could be determinate to Twitter.

This is a cleaver DDoS defense mechanism, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script, Tweetdeck, Power Twitter, Seesmic, and the list goes on. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks.

It’s pretty clear they are ready for a redesign. They need their own autonomous network, bring in bandwidth from many different providers, and have several layers of security. Building a strong ACL border and a nice mitigation layer would make a lot of sense. I also don’t think abandoning the shared cloud services model is a good idea, but having control over the heart of the operation makes sense.

Facebook has been doing a much better job at this as they were not crippled from the same attacks.

It just goes to show, there needs to be a blend of cloud services (like AWS) along with good in-house design.

UPDATE: 302 redirects were countermeasure fail

Posted on Google Developer Talk, Twitter developer Alex Payne stated:

Alex Payne
Aug 11, 5:00 pm
We’re aware of these issues; sorry.
Our ops team tells me that the countermeasures that are being put in place
should not cause the 302 redirect behavior that impacted OAuth and other
services late last week
. If you’re seeing that behavior, please post here
and we’ll coordinate with them to eliminate it.

Twitter down due to DDoS

Thursday, August 6th, 2009
Twitter Fail, image by hellvetica At a presentation I gave at an International Terrorism and Intelligence conference, I discussed how Twitter is an obvious DDoS target. Well about 30 days later they’re in the thick of it.

Twitter is down and their network has clear signs of massive failure. In the several hundred (if not more) cases of DDoS I have had experience with, this looks like a very clear case of an attack.

Congestion is a very clear sign of a DDoS attack. In this case you will see on a traceroute clean hops up to the last few, where the network starts to get congested. Basically that means each step of the network is clean until things concentrate at the end.

The assumption is the congestion is caused by DDoS and not a system administrator creating a routing loop or something whacky like that.

They also only appear to have ONE network provider (NTT), which is rather insane these days. It also makes targeting Twitter a much less complicated task.

Using very basic tools it is possible to see that the congestion on their network is rather extreme. It’s possible to deduce that the congestion is probably due to a DDoS attack.

DDoS Clue 1: UDP now blocked

6 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 21.497 ms 18.386 ms 19.277 ms
7 128.121.150.245 (128.121.150.245) 19.289 ms 20.950 ms 17.331 ms
8 * * *
9 128.121.150.245 (128.121.150.245) 20.178 ms !X * *
10 128.121.150.245 (128.121.150.245) 20.731 ms !X * *
11 128.121.150.245 (128.121.150.245) 19.777 ms !X * *
12 128.121.150.245 (128.121.150.245) 27.217 ms !X * *
13 * 128.121.150.245 (128.121.150.245) 24.115 ms !X *
14 * * *

The !X in the traceroute tells us that someone has placed an ACL or a filter to block certain types of traffic. In this case the traffic they are blocking is UDP, which is what traceroute generates to test each hop.

DDoS Clue 2: Massive and erratic latency

When you look at a TCP data flow, with a tool like tcptraceroute, it’s possible to get a little deeper into the twitter network. You can see easily that there’s something very wrong at hop 6, where it goes from 10ms to over 700ms.

This is really strong evidence that someone is attacking Twitter:

4 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 5.471 ms 10.941 ms 10.987 ms
5 128.121.150.133 (128.121.150.133) 10.988 ms 10.050 ms 10.988 ms
6 128.121.146.165 (128.121.146.165) 713.595 ms 1927.409 ms 1954.990 ms

One step further you can see the ICMP data is also showing massive struggle with the upstream network:

— twitter.com ping statistics —
248 packets transmitted, 68 packets received, 72.6% packet loss
round-trip min/avg/max/stddev = 1.080/424.280/2216.415/625.497 ms

This shows that the max response time has been 2.2 seconds (should be in milliseconds) and that the average is almost half a second. In my experience, this is very clear evidence of DDoS.

UPDATE 1: Twitter’s status page is reporting DDoS

Apparently they operate segmented networks, thus the www.twitter.com servers and load balancers are different than search.twitter.com which is different from status.twitter.com. Both status.twitter.com and search.twitter.com are up, I would assume also some of their API stuff is up, here’s what they say on status.twitter.com right now:

“Ongoing denial-of-service attack 4 minutes ago
We are defending against a denial-of-service attack, and will update status again shortly.

Site is down 1 hour ago
We are determining the cause and will provide an update shortly.

Update: we are defending against a denial-of-service attack.”

UPDATE 2: Twitter is struggling

The site continues to bounce up and down, it’s pretty clear they are trying to use DDoS mitigation techniques. The technique I see right now is a HTTP redirect with the assumption that Bots do not follow redirects:

GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020

The assumption is that a real web browser will follow the new location cookie (“?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed a bot.

This is a good, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks. There may have been too many cooks in the kitchen.