Posts Tagged ‘Cyberwar’

Twitter down due to DDoS

Thursday, August 6th, 2009
Twitter Fail, image by hellvetica At a presentation I gave at an International Terrorism and Intelligence conference, I discussed how Twitter is an obvious DDoS target. Well about 30 days later they’re in the thick of it.

Twitter is down and their network has clear signs of massive failure. In the several hundred (if not more) cases of DDoS I have had experience with, this looks like a very clear case of an attack.

Congestion is a very clear sign of a DDoS attack. In this case you will see on a traceroute clean hops up to the last few, where the network starts to get congested. Basically that means each step of the network is clean until things concentrate at the end.

The assumption is the congestion is caused by DDoS and not a system administrator creating a routing loop or something whacky like that.

They also only appear to have ONE network provider (NTT), which is rather insane these days. It also makes targeting Twitter a much less complicated task.

Using very basic tools it is possible to see that the congestion on their network is rather extreme. It’s possible to deduce that the congestion is probably due to a DDoS attack.

DDoS Clue 1: UDP now blocked

6 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 21.497 ms 18.386 ms 19.277 ms
7 128.121.150.245 (128.121.150.245) 19.289 ms 20.950 ms 17.331 ms
8 * * *
9 128.121.150.245 (128.121.150.245) 20.178 ms !X * *
10 128.121.150.245 (128.121.150.245) 20.731 ms !X * *
11 128.121.150.245 (128.121.150.245) 19.777 ms !X * *
12 128.121.150.245 (128.121.150.245) 27.217 ms !X * *
13 * 128.121.150.245 (128.121.150.245) 24.115 ms !X *
14 * * *

The !X in the traceroute tells us that someone has placed an ACL or a filter to block certain types of traffic. In this case the traffic they are blocking is UDP, which is what traceroute generates to test each hop.

DDoS Clue 2: Massive and erratic latency

When you look at a TCP data flow, with a tool like tcptraceroute, it’s possible to get a little deeper into the twitter network. You can see easily that there’s something very wrong at hop 6, where it goes from 10ms to over 700ms.

This is really strong evidence that someone is attacking Twitter:

4 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 5.471 ms 10.941 ms 10.987 ms
5 128.121.150.133 (128.121.150.133) 10.988 ms 10.050 ms 10.988 ms
6 128.121.146.165 (128.121.146.165) 713.595 ms 1927.409 ms 1954.990 ms

One step further you can see the ICMP data is also showing massive struggle with the upstream network:

— twitter.com ping statistics —
248 packets transmitted, 68 packets received, 72.6% packet loss
round-trip min/avg/max/stddev = 1.080/424.280/2216.415/625.497 ms

This shows that the max response time has been 2.2 seconds (should be in milliseconds) and that the average is almost half a second. In my experience, this is very clear evidence of DDoS.

UPDATE 1: Twitter’s status page is reporting DDoS

Apparently they operate segmented networks, thus the www.twitter.com servers and load balancers are different than search.twitter.com which is different from status.twitter.com. Both status.twitter.com and search.twitter.com are up, I would assume also some of their API stuff is up, here’s what they say on status.twitter.com right now:

“Ongoing denial-of-service attack 4 minutes ago
We are defending against a denial-of-service attack, and will update status again shortly.

Site is down 1 hour ago
We are determining the cause and will provide an update shortly.

Update: we are defending against a denial-of-service attack.”

UPDATE 2: Twitter is struggling

The site continues to bounce up and down, it’s pretty clear they are trying to use DDoS mitigation techniques. The technique I see right now is a HTTP redirect with the assumption that Bots do not follow redirects:

GET /
HTTP/1.1 302 Moved Temporarily
Content-Length: 0
Location: /?c3abf020

The assumption is that a real web browser will follow the new location cookie (“?c3abf020″ in my case). If you don’t follow the cookie, then you’re assumed a bot.

This is a good, however, there are thousands of scripts and tools written around Twitter’s API which don’t understand how to follow a 302. Thus, they are going to lock out lots of non-browser based clients. This includes my front page twitter update PHP script. I’ll fix my script now, but it looks like there was a lack of preparation for these attacks. There may have been too many cooks in the kitchen.

Using Squid Proxy to Fight DDoS

Friday, July 24th, 2009



Complicated web applications are often difficult to scale, as a result they become easy DDoS targets. However, making them scale is easy with front-end proxy servers. The added scale gives an application more resiliency to DDoS attacks.


When setup correctly, the proxy “network” becomes the target of any malicious activity and can be placed globally while still keeping the original web application in same location for content.


This is by no means new, it’s been done all over the Internet and in some cases is the base of a bunch of different companies. This is just a simple tutorial that is meant to help people understand how this works.


Proxy servers can also be used with a dynamic caching function which can provide caching which will help increase the speed and functionality of the web site.

Positives:

  • Scales web server farms
  • Increases reach
  • Can accelerate a web site
  • Can provide additional security layers

    Negatives:

  • Adds an additional layer of debugging
  • Slows down long dynamic pages if they are not cacheable
  • Expensive to operate


    To start, I recommend using Squid Proxy version 2.7, it is available at http://www.squid-cache.org/


    After downloading the package, the vanilla build will suffice for most needs. You can use FreeBSD as the operating system and simply make install on the /usr/ports/www/squid package or build the package with a ./configure –prefix=/usr/local ; make install


    Often the prefix is /usr/local but determine what is appropriate for your OS.


    After the build has finished you will need to configure Squid, attached below is a sample configuration file:

    acl all src 0.0.0.0/0.0.0.0
    acl DO_NOT_CACHE urlpath_regex -i cgi-bin \? asp php css js
    acl manager proto cache_object
    acl purge method PURGE
    #
    refresh_pattern .               0       20%     1440
    #
    http_access allow all
    icp_access allow all
    #
    request_header_max_size 10 KB
    #
    cache_dir ufs /vol1/cachedir 512 16 256
    #
    visible_hostname supersite.com
    pid_filename /var/run/squid.pid
    #
    cache_access_log /var/log/httpd/proxy-a_access.log
    #
    cache_mem 64 MB
    maximum_object_size_in_memory 64 MB
    #
    httpd_accel_host virtual
    httpd_accel_uses_host_header on
    #
    #
    connect_timeout 30 seconds
    #
    emulate_httpd_log on
    hierarchy_stoplist cgi-bin ? asp css js php
    http_port XX.YY.ZZ.AA:80
    http_port XX.YY.ZZ.BB:80
    negative_ttl 60 seconds
    no_cache deny NOCACHE
    

    The configuration options are all explained in the default configuration file, the only major items to change are the http_port list, which should be the IP address it should respond on and the cache configuration. Some sites may have special items that should not cache. Often css and js should cache, but for this example they are dynamic.


    The logs will be written to /var/log/httpd/proxy-a_access.log in a combined Apache style format.


    When starting the squid, you will need to create a /cache directory on the server, simply run:


    mkdir /vol1/cachedir
    chown squid /vol1/cachedir

    You will also need to Create swap directories so Squid can run:


    /usr/local/bin/squid –z


    You will also need to teach squid how to communicate back to your “real” or “backend” web farm, often the DNS for www points to the IP address squid is answering requests for, this can be done using the /etc/hosts file:


    XX.YY.AA.BB www.mysitedns.com


    Replace the example above with the real IP address of the web farm and the host entry you want to be used to reach the IP address.


    Once squid is running and answering requests (/usr/local/bin/squid -k reconfigure /usr/local/etc/squid.conf) and the cache is working, it tends to stay stable until the hardware fails or you become under DDoS attack, which may require some additional ACLs within the squid.conf or SYN cookies configurations on the OS itself.


    Scaling squid is also not very difficult, it’s possible to load balance a farm of Squid servers with any standard load balancer, and have the requests still return to the same web farm, which may or may not work with any given user authentication / sessions setup.


    Blocking a given attack in Squid is trivial, however, if there are hundreds of Squid servers to configure at the same time, this may require some special configuration management that could require some development effort.


    Often most attacks have an empty or mal-formed User-Agent, this simple ACL will block 99% of invalid User-Agent attacks:

    acl OK_BROWSER browser a b c d e f g h i j k l m n o p a r s t u v w x y z 1 2 3 4 5 6 7 8 9 0 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
    acl DO_NOT_CACHE urlpath_regex -i cgi-bin \? asp php css js
    http_access allow OK_BROWSER
    http_access deny all
    http_access deny manager
    http_access deny purge
    icp_access allow all
    


    You can also create a deny filter by creating an ACL that will deny rather than allow, the above ACL requires the user to have an VARCHAR in their User-Agent, which is pretty wide, so denying a specific item can be done like this:


    acl BAD_BROWSER browser Attack-Bot


    Add the deny line as the first line in the http_access ACL:


    http_access deny BAD_BROWSER


    Blocking a specific URL can look like this configuration line (which is designed to block most malicious requests):


    acl BLOCK_URI urlpath_regex -i \.exe \.\./\.\. \.\.\. \.ida \.idq \.IDA \.cnf \.asp \.dll 333-3333 test999 passwd /etc \` boot \.exe cmd \./\./ filenumber \% \* \; SELECT \\\.\.\\ \/\.\.\/


    Configuration of connection rate limiting looks like:


    acl 8conn maxconn 8


    And blocking a specific source address prefix:


    acl ip_addr1 src 192.168.1.0/24


    Just ensure that the ACL that is created is also configured in the http_access deny/allow list properly. Squid also needs to be told to re-read the configuration file, this is done by sending squid a –k reconfigure flag which will simply reload the rules without impacting traffic.


    To enable reverse proxy of SSL with the Squid cache owning the SSL certificate, you can use a pem cert and the following configuration line:


    https_port IP:443 cert=/usr/local/etc/squid/certs/COMPANY/COMPANY.pem key=/usr/local/etc/squid/certs/COMPANY/COMPANY.key


    Good luck and happy calamari

  • OpenSSH owned?

    Monday, July 20th, 2009
    A group called “the anti-sec movement” which credits itself for hacking ImageShack about two weeks ago, and astalavista.com, could have possibly found the holy grail of exploit code: A remote exploit for OpenSSH.

    OpenSSH is an open source client/server protocol that replaced telnet (an unencrypted remote management tool), it’s what system administrators use to login to their Unix computers, phone switches, power management tools, serial consoles, routers, etc. It’s been running so long in the public that people trust it.

    In fact, in many cases it is much like a door to a house, just out there on the street and if you have the right keys, you can come in.

    The anti-sec movement has announced that in 48-hours they will release code which allows you to open any door on the Internet (if it’s visible).

    “Soon, the very foundations of Information Technology and Information Security will be unearthed as millions upon million of systems running ANY version of OpenSSH are compromised by wave after wave of script-kiddie and malicious hacker.”

    This type of hype happens every now and then, especially around the time of DEFCON (a large hacker conference in Las Vegas). It could be, and most likely is — FUD.


    In fact, if it’s not FUD… use your uber cool 0-day sploit to hack my server please! blyon@blyon.com port 22. Prove it!

    SANS also thinks it is FUD and they just put out a release about the “exploit”. Sounds like they feel it’s most likely a brute force user attack, which is pretty basic, old, and boring.

    If it is real, anti-sec said they will also be “unleashing powerful computer worm source code with the ability to automatically find and compromise systems running any and all versions of OpenSSH.”

    Meanwhile: Filter SSH, turn it off if you don’t know what it is, change the default port from 22 to something else, or enable TCP Wrappers.

    Let’s sit back and enjoy the show… if there is one.

    Here are some useful links:

    We are Digital Natives

    Saturday, July 4th, 2009
    A new class of person has emerged in the online world: Digital Natives. While living in San Francisco, I also live on the Internet. The Internet is now a place: a two dimensional world that has transcended the web; there is no government, and the citizens are Digital Natives. As Digital Natives, we are not people that only exist in a physical sense–we are something or someone metaphysically different. We are no longer just citizens of say, the United States; we are also citizens of the Internet.

    The concept of the Digital Native is a paradigm shift. In the past, there were movements, but not full worlds where one can exist and do as one pleases in parallel with their physical being. Some Digital Natives are deeply affiliated with all sorts of interests that bring them together organically: Piracy groups, massively multiplayer online games, open source software development, cracking encryption, etc. Others become deeply interested in movements such as Anonymous, the RBN (Russian Business Network), or even terrorist organizations.

    I’m not trying to say a Digital Native is better than someone unplugged in the Congo, I am trying to say they exist in a different social construct.

    Some Digital Natives may feel like their digital citizenship takes precedence over their physical citizenship. They choose not to define themselves by what country they live in but, rather, by what online movement(s) they are involved in. In these situations, what law does one live by? How are the actions of a Digital Native regulated? Governments don’t know how to react to, control, or assert power over them in these situations.

    Digital Americans are no longer just American citizens–they have a deep affiliation as Internet citizens as well.

    This scares the crap out of Governments all over the world, because they are ill prepared to deal with these situations. To government regimes that are comfortable asserting their control, this concept is terrifying. How do they counteract the changes online and the movements? Do they need to change their politics, defense, propaganda, and warfare?

    Apparently the U.S. Government thinks so. In June of 2009, under an order signed by Defense Secretary Robert Gates, the Pentagon announced it will create a Cyber Command to oversee the U.S. military’s efforts to protect its computer networks and have presence in “cyberspace”.

    Now even the US Military war machine is joining the world of Digital Natives.

    I’m a bit worried, not for us, but for them.

    Barrett Lyon creates fun companies that do all sorts of innovative exciting things with video and security.

    CDN cdn
    BitGravityBitGravity Barrett Lyon
    BitGravityBitGravity
    Barrett Lyon
    LimeLight Networks LimeLight Networks
    EdgeCast EdgeCast
    CDNetworks CDNetworks
    Consulting Consulting
    Speaker Speaking Opportunity
    Speaker Speaking Opportunity
    Content Delivery Network Content Delivery Network
    Content Delivery Content Delivery
    Flash Streaming Flash Streaming
    Interactive Video Interactive Video
    Live Streaming Live Streaming
    Live Video Live Video
    Streaming Audio Streaming Audio
    Streaming Media Streaming Media
    Video Delivery Video Delivery
    Video Hosting Service Video Hosting
    Video Podcasting Video Podcasting
    Video Podcasts Video Podcasts
    Video Services Video Services
    Video Streaming Video Streaming
    Barrett Lyon Barrett Lyon
    Barrett Lyon Barrett Lyon
    Barrett Lyon Barrett Lyon
    Barrett Lyon Barrett Lyon
    Barrett Lyon Barrett Lyon
    Barrett Lyon Barrett Lyon
    Barrett Lyon Barrett Lyon