Archive for the ‘Security’ Category

The Internet is Beta

Tuesday, May 4th, 2010
Beta is an engineering way of saying “almost done” – the product is good enough to use but it’s not quite finished yet. Google often releases their new products with a cute little “BETA” logo. Gmail, the Google email system used by millions, has been in beta for five years.

Like Gmail, the Internet’s core protocol should also have had a Beta tag on it for an extended time – for the past 41 years to be precise. Generally speaking, it works pretty well, but the founding fathers of the Internet could not have anticipated that the software they were building would ever become what it is now: The infrastructure for all of society.

So it appears today that some major features were left out…but not because the people behind the design made a mistake. When MIT first used packet switching in 1965 to communicate with a remote computer in California (confirming that packet switching works), the furthest thing from anyone’s mind was security, network neutrality, network education, privacy, cyber warfare, and the slurry of problems that challenge both business and individual users of the Internet today.

In 1969, with the original workings of the Internet (ARPANET), security was simple: the network was tiny and users on the computers that were connected to it were trusted researchers. It was an open community. As Vint Cerf, one of the most notable developers of the Internet, was quote in Fatal System Error as saying, “My thought at the time, thirty-five years ago, was not to build an ultra-secure system, because I could not tell if even the basic ideas would work…We never got to do the production engineering.” The focus at the time, sensibly, was on fault tolerance, not security.
Vint Cerf – Photo by Charles Haynes

Now, nearly 41 years later, we read about Internet security issues constantly. The lack of security features in IP (Internet Protocol) has spawned entire industries, with vendors and service providers that are happy to sell you the next generation protect-all, whiz-bang software. If one were to ask a roomful of people in the security industry what they think about the security products, including their own, on the market today – if they think there are real solutions to the problems we all face – their answer would be a unified “NO”. No one thinks we are at the point where we can all just stop worrying about security.


Barack Obama
Courtesy The White House
The disturbing fact is that the engine that enables our modern global economy is based on a really cool experiment that was not designed for security. Risks can be reduced, but the naughty truth is that the ‘Net is not a secure place for business or society.

The role that the Internet plays in our economy places it in the category of a critical resource that the government must protect – just as it does our water supply and the national power grid. A threat to Internet security is a threat to national security. In May 2009, President Obama spoke about this issue and the plan his administration has to address it. He stated that the US is “not as prepared as it should be” to defend against cyber threats and he proposed new “digital infrastructure” initiatives to “ensure that these networks are secure, trustworthy and resilient.”

But can the US Government, or any other governing authority, ever adequately protect and defend the Internet? How can that be done if the Internet Protocol itself was not designed to, in Obama’s words, “deter, prevent, detect, and defend against attacks”?

Given the world economy’s substantial dependence on the Internet, wouldn’t it make sense to create a well-funded think-tank with the brightest minds in society to design a new protocol with a new vision? This time when we start the process, we will have the benefit of 41 years of Internet beta testing and we can rethink the vision to also include things such as:

  • Security: Transmitting data safely but easily without special software.
  • Privacy: Balancing anonymity and accountability. Allowing people to communicate freely but ensuring accountability to protect against abuses and criminal activity.
  • Routing Intelligence: Routing data without neutrality issues and allowing the protocol itself to route traffic based on a myriad of metrics, conditions, agreements, and other factors.
  • Enculturation and Education: Bringing new people (children, emerging nations, etc) onto the network with a step approach to ensure that they learn about network culture and functionality before they make mistakes.

I don’t think any of us who are involved with cyber security on a professional level can see the Internet as it is today functioning successfully for the next 50 years. I can envision a world of networking much different than today’s. So why not start turning the ship now?

Is designing a better protocol difficult? Yes. Can it be done? Absolutely!

I will be writing more on this topic in the coming months. Stay tuned.

The Top 10 Things To Do While Under DDoS Attack

Sunday, January 24th, 2010

In my past decade-plus dealing with distributed denial-of-service attacks, I have noticed a few patterns in the way that companies handle these attacks. Usually when an unprepared virgin company is first attacked, all hell breaks loose. The lack of preparedness causes several chain reactions that make the situation worse. Addressing these most common mistakes ahead of time can help a situation tremendously.

When someone calls me for advice, the first few items I go over have nothing to do with fixing the attack. I’m giving advice that I think is common sense, and I’ve been surprised that others don’t find it obvious.

Here are my Top 10 To-do’s for making life less painful during an attack.

1. Don’t Panic

While the network and your services are exploding and bouncing offline, there must be someone that is comfortable enough to make good decisions. I’ve seen managers freak out and threaten everyone with the prospect of the company collapsing. I think they were trying to motivate people to figure out some solution, but they ended up creating more chaos during an already tough situation.

Once I saw employees hastily rip out the network’s firewalls and re-configure the load balancers. They ended up creating more mess than they had before because they were reacting to an angry and stressed manager.

You are going to create a disaster if you approach with a sledgehammer and wishes. Don’t let anyone make quick changes; try to follow your company’s policies. Sit back, analyze the problem, isolate the actual device that’s failing in the chain, and make an informed–and usually small–adjustment.

If you’re in the 10th hour and things don’t seem to be improving, gather everyone, go away from the office, have a beer, relax for 15 minutes, and talk about something positive. The information flow after that beer might just save you and motivate everyone to do a good job – the solution will come!

2. Create a contact list of external email addresses and phone numbers.

This one is sadistically funny. Most companies host their email, VoIP system, IRC, Wiki, databases, primary storage, etc. all in the same colocation behind the same network connection that hosts their web sites and services. This is, for lack of better words, stupid. All of your digital eggs are in one basket, and that basket is also holding a grenade. A DDoS attack ends up crippling the company’s infrastructure, leaving it with no phones, email, or any communications structure whatsoever.

I’ve seen CEOs of massive companies using their hotmail account and cell phone to contact me because it was their only way of communicating from their multi-million dollar offices.

If you insist on being an “eggs in one basket” company, keep a list of vital email accounts and cell phone numbers on a notepad. That way you can at least call your IT person when everything is down.

3. Setup a “War Room”

Convert your conference room into a war room. Get everyone that has influence in the company in that room. This includes marketing, IT, the CEO, etc. It ensures everyone is on the same page, leaders can lead, and everyone can be in sync.

I typically fill the room with a constant flow of healthy snacks, coffee, and other beverages. If you don’t have anything like that handy, order pizza immediately or send someone shopping.

4. Get one of your guys to the colo ASAP

If you are offline due to DDoS attack, chances are your IT staff cannot log in to the remotely hosted hardware in your datacenters. The easy solution is to physically get them there. They can console in to the hardware and actually see what is going wrong. It’s not fun, but it will result in a much faster resolution to the problem (Make sure they have folding chairs, cash for the vending machines, and serial cables).

5. Find an old hub

Yes, I said hub. You know, those old things that cause collisions? If you’re dealing with an attack and yours is like a lot of companies, it may be difficult for you to set up a traffic monitoring port on your main routers. Assuming you’re setup with Ethernet, at least you can bridge a hub in-line and connect a laptop to the hub and sniff or analyze the traffic!

This is key because having eyes into the data stream really helps figure out how to filter it. Pulling random cables and shutting down random services is not the solution. Make an informed call because you were thoughtful enough to have a hub or SPAN/Mirror port pre-configured.

6. Understand the nature of the attack

There’s a reason you are the target for this attack. Obviously there are a lot of reasons for any given attack, yet understanding the attacker’s motivation is key to creating a better defense strategy.

In the field I have observed a very strange phenomenon; the people working at a victim company usually have a gut feeling about why they are being attacked. So far, their gut instinct has been correct.

Some people know they are being extorted and some people feel it’s a competitor trying to shut them down. Others have a customer that has pissed someone off so the attacker takes down the whole company just to silence one customer. Maybe shutting down the attacker’s target for awhile may actually save the entire ship. Go with your gut on this, make a hypothesis and test it.

7. Document everything

Your business was just smacked around by some bad guys, but what proof do you have? If you don’t have any, then what do you think the law enforcement is going to do for you?

During the attack, lock down all your logs and assign someone within the company to be the custodian of the records. Save server logs, web logs, email logs, any packet capture, network graphs, reports – anything – including a timeline of events.

8. Call your ISP

Your ISP can help, however they have a process to follow. The process usually requires a ticket escalation requirement before you can get real help. If you call early in the attack and open a ticket, that can help you when you really need someone.

Your ISP also has hardware that may be capable of filtering or rate-limiting the attack. The more you know about the attack and you can point them in the right direction, the more they can help you.

They may also suggest you to sign up for their DDoS protection system. Don’t do that right away; reserve that until you are out of all other options. If you do sign up, make sure there is a service level agreement. In the meantime, there are a number of free services you can request:

Null routing of the target IP address
Router ACLs of the top attacking source addresses
New IP addresses
Detailed traffic reports

If you can find the guru at the ISP that knows how to fix these problems, that might be time well spent.

9. Setup “We are down” web hosting services

If the attack is running longer than you had anticipated and you don’t have a solution in sight, you could get your site working at least enough to communicate to your customers.

There are web-hosting companies, which as part of what they do, provide DDoS service level agreements. For a small amount of money you could quickly sign up with several of these companies, upload a “Sorry we’re down, but contact us here” page, and flip your DNS to the cluster of hosted servers.

Your customers will have more confidence in your performance and the attackers may get bored because the attack has not completely shut everything down. If this plan doesn’t work, at least you have diverted some of the attack away from your network.

10. Learn from the event

Post attack can be a blur; everyone is exhausted and burnt out. Mostly, everyone just wants the day-to-day atmosphere to return to status quo. Well, if you’ve been attacked and you did not learn and improve your strategy on how to deal with future attacks, then you are not doing your job.

You should start a review the very day after, while everything is fresh, and make sure that everyone is prepared. Go over what worked, what did not work, and how to improve your system’s overall technology.

Spend the money to fix things properly. Don’t just duct-tape it.