At 3Crowd have been working furiously over the last month on the development of our first two services. The whole team has been dedicated to delivering on our mission of helping companies reduce costs, deliver, and scale their network applications through a healthy ecosystem for data delivery.

With our first services – CrowdMonitor and CrowdDirector, we’re giving the world just a tiny view of the foundation we’re building to enable a new type of Content Delivery System. We are using the idea of a crowd or crowd-sourcing to leverage multiple CDNs that can be monitored, managed and deployed as one single service.

CDNs have been an integral part of the vitality of video on the Net, but we think they are starving the next generation of innovation. The natural evolution of this marketplace has favored the content distributors but the market needs to continue its evolution into a more balanced relationship.

CrowdMonitor and CrowdDirector will give content owners the power to know what’s happening with their content and the tools to dynamically manage their assets. They will have significant implications on performance, vendor flexibility and costs and not only impact the business of current content providers, but also re-orient the economics and invite a new generation of providers to participate in the market.

We have been thoughtful about how we are creating this ecosystem and have been working on it for a while. If we do our job right, Internet streaming will be more affordable than traditional broadcast systems and together will have success.

When someone calls me for advice, the first few items I go over have nothing to do with fixing the attack. I’m giving advice that I think is common sense, and I’ve been surprised that others don’t find it obvious.

Here are my Top 10 To-do’s for making life less painful during an attack.

1. Don’t Panic

While the network and your services are exploding and bouncing offline, there must be someone that is comfortable enough to make good decisions. I’ve seen managers freak out and threaten everyone with the prospect of the company collapsing. I think they were trying to motivate people to figure out some solution, but they ended up creating more chaos during an already tough situation.

Once I saw employees hastily rip out the network’s firewalls and re-configure the load balancers. They ended up creating more mess than they had before because they were reacting to an angry and stressed manager.

You are going to create a disaster if you approach with a sledgehammer and wishes. Don’t let anyone make quick changes; try to follow your company’s policies. Sit back, analyze the problem, isolate the actual device that’s failing in the chain, and make an informed–and usually small–adjustment.

If you’re in the 10th hour and things don’t seem to be improving, gather everyone, go away from the office, have a beer, relax for 15 minutes, and talk about something positive. The information flow after that beer might just save you and motivate everyone to do a good job – the solution will come!

2. Create a contact list of external email addresses and phone numbers.

This one is sadistically funny. Most companies host their email, VoIP system, IRC, Wiki, databases, primary storage, etc. all in the same colocation behind the same network connection that hosts their web sites and services. This is, for lack of better words, stupid. All of your digital eggs are in one basket, and that basket is also holding a grenade. A DDoS attack ends up crippling the company’s infrastructure, leaving it with no phones, email, or any communications structure whatsoever.

I’ve seen CEOs of massive companies using their hotmail account and cell phone to contact me because it was their only way of communicating from their multi-million dollar offices.

If you insist on being an “eggs in one basket” company, keep a list of vital email accounts and cell phone numbers on a notepad. That way you can at least call your IT person when everything is down.

3. Setup a “War Room”

Convert your conference room into a war room. Get everyone that has influence in the company in that room. This includes marketing, IT, the CEO, etc. It ensures everyone is on the same page, leaders can lead, and everyone can be in sync.

I typically fill the room with a constant flow of healthy snacks, coffee, and other beverages. If you don’t have anything like that handy, order pizza immediately or send someone shopping.

4. Get one of your guys to the colo ASAP

If you are offline due to DDoS attack, chances are your IT staff cannot log in to the remotely hosted hardware in your datacenters. The easy solution is to physically get them there. They can console in to the hardware and actually see what is going wrong. It’s not fun, but it will result in a much faster resolution to the problem (Make sure they have folding chairs, cash for the vending machines, and serial cables).

5. Find an old hub

Yes, I said hub. You know, those old things that cause collisions? If you’re dealing with an attack and yours is like a lot of companies, it may be difficult for you to set up a traffic monitoring port on your main routers. Assuming you’re setup with Ethernet, at least you can bridge a hub in-line and connect a laptop to the hub and sniff or analyze the traffic!

This is key because having eyes into the data stream really helps figure out how to filter it. Pulling random cables and shutting down random services is not the solution. Make an informed call because you were thoughtful enough to have a hub or SPAN/Mirror port pre-configured.

6. Understand the nature of the attack

There’s a reason you are the target for this attack. Obviously there are a lot of reasons for any given attack, yet understanding the attacker’s motivation is key to creating a better defense strategy. In the field I have observed a very strange phenomenon; the people working at a victim company usually have a gut feeling about why they are being attacked. So far, their gut instinct has been correct. Some people know they are being extorted and some people feel it’s a competitor trying to shut them down. Others have a customer that has pissed someone off so the attacker takes down the whole company just to silence one customer. Maybe shutting down the attacker’s target for awhile may actually save the entire ship. Go with your gut on this, make a hypothesis and test it.

7. Document everything

Your business was just smacked around by some bad guys, but what proof do you have? If you don’t have any, then what do you think the law enforcement is going to do for you?

During the attack, lock down all your logs and assign someone within the company to be the custodian of the records. Save server logs, web logs, email logs, any packet capture, network graphs, reports – anything – including a timeline of events.

8. Call your ISP

Your ISP can help, however they have a process to follow. The process usually requires a ticket escalation requirement before you can get real help. If you call early in the attack and open a ticket, that can help you when you really need someone.

Your ISP also has hardware that may be capable of filtering or rate-limiting the attack. The more you know about the attack and you can point them in the right direction, the more they can help you.

They may also suggest you to sign up for their DDoS protection system. Don’t do that right away; reserve that until you are out of all other options. If you do sign up, make sure there is a service level agreement. In the meantime, there are a number of free services you can request:

Null routing of the target IP address
Router ACLs of the top attacking source addresses
New IP addresses
Detailed traffic reports

If you can find the guru at the ISP that knows how to fix these problems, that might be time well spent.

9. Setup “We are down” web hosting services

If the attack is running longer than you had anticipated and you don’t have a solution in sight, you could get your site working at least enough to communicate to your customers. There are web-hosting companies, which as part of what they do, provide DDoS service level agreements. For a small amount of money you could quickly sign up with several of these companies, upload a “Sorry we’re down, but contact us here” page, and flip your DNS to the cluster of hosted servers.

Your customers will have more confidence in your performance and the attackers may get bored because the attack has not completely shut everything down. If this plan doesn’t work, at least you have diverted some of the attack away from your network.

10. Learn from the event

Post attack can be a blur; everyone is exhausted and burnt out. Mostly, everyone just wants the day-to-day atmosphere to return to status quo. Well, if you’ve been attacked and you did not learn and improve your strategy on how to deal with future attacks, then you are not doing your job.

You should start a review the very day after, while everything is fresh, and make sure that everyone is prepared. Go over what worked, what did not work, and how to improve your system’s overall technology.

Spend the money to fix things properly. Don’t just duct-tape it.

I wrote this blog to protect other people from making the same bad decision I made. Please understand that this experience is not an epidemic with all electric vehicles, it just happens to be my experience with this particular company.

I found a company called EV Innovations (whom just changed their name to Li-Ion Motor Corporation and was formally known as Hybrid Technologies). They convert (or as they would say “build”) standard gas-powered PT Cruisers into fully electric vehicles. This made sense for my needs.

My wife at the time was driving a PT that we bought from my late grandmother, it’s a pretty gutless car which has no sex appeal whatsoever, but it worked for us. It reminded me of my grandmother and is also a car I could drive into San Francisco and not really worry about theft or vandalism.

EV Innovations advertised their converted PT Cruisers as getting a distance of 120+ miles per charge with a charging time of 8 hours. Based on those specifications I could drive to the Sierra Foothills and commute to the Bay Area carbon free.

I did some research and found out EV Innovations at the time was testing their PT conversion as an EV Taxi in New York (which later ended in failure). They sold two cars to a company in Sacramento and the company was also featured on the Discovery Channel’s program Modern Marvels.

I also liked the idea of supporting a startup, I myself enjoy starting companies. Purchasing a car from an American based startup felt right. I know that if something was wrong with one of my customers, I would do anything under my power to make it right. Naively I assumed this was also true for EV Innovations.

I spent time asking EV Innovations a lot of questions about charging, regenerative breaking, cell types, and their engineering philosophy. Their sales people Kelli Cerven and Mike Cerven were very helpful. I decided to move forward and paid the deposit for half of the car, which was used to purchase a new PT with my specification. The car was to be ready in 90 days. Unfortunately, it ended up shipping in 120+ days. But things started to go wrong much before the car was delivered. It turns out the “new car” I paid for was in fact a demo car that already had 4,000 miles on it. In addition, they told me the car was ready to ship a month before it was actually finished. They asked for the balance so they could ship the car. I wired the remaining balance of the car only to find out later the car was not ready.

My Electric Lemon being converted from ICE to EV.

My first moments with the Li-Ion Motors (formally EV Innovations) lemon.

After all was said and done, three months after the promised delivery date, the vehicle was finally shipped, and yes, it was dead on arrival. The truck driver that delivered it didn’t even have a ramp on his truck, so we used the loading dock at a local grocery store where I was visiting family for Christmas in the Sierra Foothills. The truck driver pushed the car off the truck and drove away. I had it towed back to the garage at my family’s home and plugged it in but it would not charge. I immediately contacted EV Innovations and they instructed me to plug it into a 220V outlet and informed me that it cannot charge from empty on a 110V outlet. That seemed obnoxious, but I did as instructed and found a 220V line and plugged it in. It slowly charged to full.

The first drive made it about 50 miles (downhill) before we had to have it towed the rest of the way home to the Bay Area. The car was to do 120+ miles and their sales people suggested the “+” would be impressive.

EV Innovations asked me to do a number of things with the car, which eventually resuted in the car’s charging connector to melt and nearly catching on fire (Later they had to install a relay to prevent some sort of voltage issue which caused the problem).

After I reported the smoking plug to EV Innovations, I received this email::

From: “Luc Pham”
Date: December 29, 2008 3:58:14 PM PST
Subject: Charging
Mr. Lyon,
Please stop charging the car for until we replace the bad cell.
Please open up the emergency switch (EPO).

From there, it sat in the garage inoperable for months while EV Innovations figured out what to do. Eventually they sent people to work on the car; they replaced the lithium cells, re-wired parts of the car, etc. I had them sign work orders to prove that they were there. In all fairness, Luc was a very nice guy and he wanted the car to work. He apologized for the awful things his management has done.

Luc Pham, EV Innovations intern working on the car.

I demanded a refund on the car but they ignored my requests. I followed up by sending their CEO and board of directors a certified letter demanding them to repurchase the car based on the Song Beverly Act (California Lemon Law), again they refused to respond.

Eventually, they asked me to ship the car back to North Carolina so they could attempt to jam more batteries into the vehicle, change the power steering system, and perform various upgrades to improve the issues I have been experiencing.

I contemplated this option but with all the bad experiences I had with the company, I did not fully trust them. So I suggested that if I were to send the car back, perhaps they could provide me with my deposit back as a gesture of good will and integrity on their part. They of course refused saying this was “warranty” work and in no way was the car being sent back due to any fault of the company. I was at a loss.

At this point the car would go about 70 miles on a 10 to 20 hour charge (depending on voltage/amperage).

I was able to drive it between my office and home for a month until the driver side wheel nearly fell off because the bearings were incorrectly packed. That was it for me.

I demanded a refund, I demanded them to take care of this problem the way any decent company would. I called Tom Zgoda their “Plant Manager” and asked them to do this right. He said I was driving the car wrong and that it’s all my fault and hung-up on me. I then pressed him via email and he said said:

“It is our position that we have been diligent in addressing the warranty on your vehicle. I guess since you disagree with that, it will be up to the courts to make a final determination.”

I hired Mark Anderson, a lawyer who specializes in Lemon Law cases and set out to do exactly what he suggested. I filed suit.

My lawyer and I hired a superstar electrical engineering expert named Art MacCarley, Ph.D., PE., who happens to be the Department Chair of the Electrical Engineering department at Cal Poly San Luis Obispo. Dr. MacCarley drove to my house and spent the entire day meticulously going through the car. Later he wrote:

“I determined that the converted vehicle may best be characterized as an engineering prototype or work-in-progress conversion, containing multiple deficiencies and defects introduced by the modification to battery-electric propulsion. These deficiencies and defects affect vehicle function, features and safety, and could be expected to limit reliability. Overall, the subject vehicle fails to meet the manufacturer’s representations and reasonable owner expectations of engineering quality.”

Art put it very politely, but basically he was saying that college students could have built this car better than EV Innovations. Attached to this article is the full report written by Dr. MacCarley.

There were spliced wires exposed to the undercarriage of the vehicle. In some cases it appeared that EV Innovations used the wrong gauge wire for high voltage lines and to top it off, and the power steering was powered by what looks to be a Briggs and Stratton electric lawn mower motor!

Excerpt from Dr. MacCarley’s report.

Excerpt from Dr. MacCarley’s report.

After we sent EV Innovations the report (I ended up paying to help EV Innovations understand why their cars are not safe) they asked to settle for less than I paid for the car. Minus the legal fees, I ended up with a loss. It was not the best outcome but at least the car would be out of my hair and I would never have to talk with the people at EV again. However, the “settlement” ended up being a delay tactic. They changed the terms of the settlement, never paid, never finalized the paperwork, did not return calls or emails as they promised, said we had been “overly aggressive” and said this is going into a “holding pattern”.

In their SEC filings from December 8th, 2009 they wrote:

“Barrett Lyon, an individual, has filed suit against the Company in the Superior Court of California, San Mateo County, for alleged breach of warranty for a vehicle he purchased from the Company seeking $68,222 in damages, plus attorney’s fees estimated in the range of $10,000 to $30,000. The Company has entered into a settlement agreement with Mr. Lyon.”

However, by December 8th, 2009 we did not have a settlement agreement and we were preparing to file a motion to continue the case and re-set a trial date. Prior to December 8th, they stopped responding to us and started to delay and break promises. Their statement in their SEC filings was very misleading to any shareholders of EV Innovations.

Today, the car sits in my garage. While very disappointing, I learned a lot during this year and a half long experience.

EV Innovations/Li-Ion Motors is in my opinion a criminal operation operated by people more interested in taking money from shareholders than building a real electric car.

There are plenty of amazing electric car companies out there such as Tesla, Fisker, and soon the big auto manufactures. Maybe in 3 to 5 years I will take another look at electric cars as a serious mode of transpiration but until then, hopefully morons like EV Innovations do not create a bad name for electric vehicles.

A detailed report on my car and the failure of EV Innovation’s design written by Art MacCarley.

Other EV Innovations a.k.a. Hybrid Technologies related links:

Hybrid Technologies is involved in illegal junk faxes

Hybrid Technologies And The Missing Fritos

Is Hybrid Technologies legitimate?

EV Innovations Stock Holders Screwed!

A hilarious CNET Review of EV Innovations.

UPDATE #1: Two days ago they changed their name from EV Innovations to Li-Ion Motor Corporation. I’m guessing this is to avoid bad press?

UPDATE #2: 1/4/2010 EV Innovations/Li-Ion Motors lawyer responds to my lawyer:

On Mon, Jan 4, 2010 at 11:01 AM, Scott Meehan wrote:
Dear Mr. Anderson:

My client was not amused by the false and defamatory comments made by your client. I hereby demand that your client take down the subject entry and CEASE AND DESIST any further dissemination of the blog entry. We will be filing libel actions against Mr. Lyon and Dr. MacCarley. Will you accept service on behalf of Mr. Lyon and/or Dr. MacCarley?

Scott Meehan

UPDATE #3: 2/16/2010 Li-Ion Motors lawyer responds and demands I take down the blog:

The letter accuses me of staying that EV Innovations/Li-Ion Motors has refused to do the repairs, which is not the case at all. Read the blog and you will see that I am stating the facts of the case and my opinion. I actually highlight the fact that they have offered to take the car back to rebuild it to the standards I was expecting over a year ago. However, adding more batteries so it can achieve the driving distance that it was to reach from day one coupled with my past experience with them, just seems like a losing proposition. Given the absolute nightmare and consistent problems I have experienced, not to mention the fraudulent sales representations, I think they should buy the car back. In my opinion, if they were good honest people, they should buy the car back and learn from this experience.

Here is the letter they just sent:

It is in this vein that I am starting a new business venture – 3Crowd Technologies. It’s in my blood. I can’t stop developing ideas that will reach these goals. I am more than excited to unveil this new project, but before I can fully do so, there needs to be some additional hard work put into it.

Stay tuned. It’s going to be a great ride and I can’t wait to tell everyone about it.

What aspects of the adventure would you find most interesting?

Send me an email with suggestions: blyon [at] blyon.com

CalTrans Bay Bridge Conduit

The conduit (seen on the left) is known as the CalTrans fiber optic cable system or “CalTrans Fibers”. It’s been providing dark fiber services for companies such as Qwest and MCI for over a decade and has wavelengths leased to hundreds of companies. It also carries IP services for some of the transpacific communication lines used by Tata Communications.

Nothing has gone down or is broken, but it is fun for folks to understand that a bridge such as the Bay Bridge carries much more than cars.

CalTrans arranged to disconnect the fiber on the Bay Bridge between 11:00 p.m. Thursday September 3 (Pacific) and 06:00 AM Friday September 4. From what I understand, the work has been completed on schedule without a problem.

Great work CalTrans!

Some people have noticed I have slowed down on my blog a bit. I’m sorry! It’s true, I have put writing on the side burner (for a couple of weeks) while I complete four more patent applications.

I’m an idea factory, without a specific employer to absorb my steady stream of thoughts and energy, I decided to create my own intellectual property. What will I do with it? The answer to that question will be rather interesting but far in the future.

I’ll be back here writing soon! I promise! Meanwhile, here’s a link to my DDoS mitigation patent that was granted recently: US 20060075491A1

These cable systems are important for the communication of the Asia Pacific region, if you live in Taiwan or Hong Kong, the Internet today may not be as snappy as you’re used to. Luckily, so far there are no reports of complete outage.

The cuts so far are on the cable systems: C2C, APCN2, and EAC. I’ve put together a very basic map (accurate enough to get a general feel) of the confirmed cuts:

Background Map Provided by Google Maps

There are other reports of issues on the other major systems SMW2 and SMW3. Both SMW2 and SMW3 were cut December 19th 2008 and were down for over a week. However, current reports about these systems may be due to traffic congestion (load from the other systems failing over).

The remaining systems, TGN and FLAG cable systems also are running well as of now.

So far there are no reports on the cause of the damage to the systems, yet the speculation is typhoon Morakot. I wish the men and women working on routing around them and fixing them good luck!

Update: Suspected cause

Debris flows damage undersea cables off southeastern Taiwan
Taipei, Aug. 12 (CNA) Deepsea debris flows off southeastern Taiwan, which were triggered by Typhoon Morakot, seriously damaged undersea cables and disrupted Internet services between Taiwan, China and Southeast Asia, according to Taiwan’s main telecommunications company.”

Update: Intra-Asia Cable (IAC)

As another important note, The Intra-Asia Cable (IAC) system from Tata Communications (the only system in the region which has planned a route not passing thru the earthquake prone region around Taiwan) was also one of the few systems up and running in the region the entire time.

To learn more, check out these links:

• Images of the World’s Submarine Cable Systems
• Submarine Communications Cables

We’re working to recover from a site outage and will update as we learn more.

Update (12:17p): We’re back up and analyzing the traffic data to determine the nature of this attack.

I am guessing as time goes on, the attacks to Twitter will be much more targeted. Attackers can update their bots to attack specific user accounts or other process-intensive working parts of Twitter which could create processing bottlenecks within the internal Twitter application itself.

It is very difficult to build large scaling databases, judging by Twitter’s current network design, I would assume their database design would be similar in stature — lack luster. Rather than just basic attacks to Twitter’s hosting partner NTT, attackers will target the Twitter database/application weak spots via their API or via their web interface. A targeted attack on Twitter’s own application weaknesses would bring more bang for the buck and be more difficult to defend against. As seen in the image on the left, some evidence of application fail is apparent. Twitter’s web site is doing some odd stuff; at times my followers/following stats and the trending topics applet disappear.

The strange behavior myself and others are seeing could be caused by more targeted application layer attacks. Of course this is speculation. However, something is not working right over there and people are attacking them.

Richard Stiennon (over at Threat Chaos) and I started looking into the attacks and dug a little deeper into Twitter’s architecture.

We noticed that the status.twitter.com page did not go down. That’s because it’s hosted in RackSpace (far away from their web servers) on some guy’s computer. What’s fun about that (as Stiennon noticed) it also hosts Fuckyeahboobies.com (WARNING: link NSFW). Yes, that’s right, Twitter’s corporate status page is also the same server that hosts Fuckyeahboobies.com.

www.fuckyeahboobies.com. 3600 IN CNAME fuckyeahboobies.com.
fuckyeahboobies.com. 3600 IN A 72.32.231.8

;; ANSWER SECTION:
status.twitter.com. 60 IN A 72.32.231.8

Let’s be fair, hosting the status page away from Twitter’s hosting infrastructure is a very good idea. However, mixing it with 14,476 other hosted sites on the same machine may not be so bright. Those other sites can attract problems. If someone were to hack the group hosted machine and modify the status.twitter.com page, it could be harmful to the value of Twitter. One needs to ask the question: “Does this bring value to Twitter? Does saving a few thousand a year in hosting cost outweigh the risks?” Personally, I would expect a company that has $55 million in funding to strive towards world class design. At this point it’s not bringing value to their shareholders by cutting corners on critical infrastructure.

Anyhoo, over the morning, I constructed this basic diagram of Twitter’s hosting architecture (below) to help people understand what happened during their DDoS attacks. I must admit, it started out well with the boobies site, but when I started looking into their network I was a bit disappointed. They have a rather flat network that appears to be completely managed by NTT/Verio.

The sections in red are the paths that the DDoS would have taken. I would guess something in their load balancing farm was not configured to deal with the attack or this would have just been absorbed without much notice. The upstream routers were doing just fine when I ran tests during the DDoS attack. I get the feeling that their load balancers are doing most of the request validation.

When you look at the TCP handshakes for www.twitter.com, it responds on any port, which indicates they are running some sort of blind syn cookies (weeds out spoofed SYN floods). Again, I am assuming their load balancer or an upstream firewall is doing this. Along with their SYN cookies, they are doing a 302 redirect cookie (web server lingo for, “hey get the page over here instead”):

GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020

The assumption is that a real web browser will follow the new location cookie (”?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed to be a bot and denied access to the site. In some cases I have seen this setup eventually put repeat offenders on a blacklist, this could be determinate to Twitter.

This is a cleaver DDoS defense mechanism, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script, Tweetdeck, Power Twitter, Seesmic, and the list goes on. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks.

It’s pretty clear they are ready for a redesign. They need their own autonomous network, bring in bandwidth from many different providers, and have several layers of security. Building a strong ACL border and a nice mitigation layer would make a lot of sense. I also don’t think abandoning the shared cloud services model is a good idea, but having control over the heart of the operation makes sense.

Facebook has been doing a much better job at this as they were not crippled from the same attacks.

It just goes to show, there needs to be a blend of cloud services (like AWS) along with good in-house design.

UPDATE: 302 redirects were countermeasure fail

Posted on Google Developer Talk, Twitter developer Alex Payne stated:

Alex Payne
Aug 11, 5:00 pm
We’re aware of these issues; sorry.
Our ops team tells me that the countermeasures that are being put in place
should not cause the 302 redirect behavior that impacted OAuth and other
services late last week. If you’re seeing that behavior, please post here
and we’ll coordinate with them to eliminate it.