With Distributed Denial of Service attacks very much in the news, I’m very glad to be hosting this guest article by someone who for now wishes to remain anonymous – Martin.
Until a few months ago I had never seen a DDoS attack and I had no idea what can be done about it, if anything at all. Not because I wasn’t interested; I was. But most of the papers on the net that deal with the subject are academically abstract and hardly useful in a concrete “I’m in deep shit, WTF do I do now?” situation. So here’s a very basic list of what you can reasonably do before you get attacked, assuming that you have no reason to expect an attack:
Preparing before an attack
Choose your hosting provider carefully
If your ISP is the kind that null-routes his customers at the first sign of a DDoS, nothing else of what you could do matters; your site will be killed by your own ISP anyway.
Most ISPs are exactly that kind. Useless. Some ISPs (e.g. Gigenet) capitalize on this by offering DDoS protection to the tune of USD 1.000 per calendar month. For most of us that’s just as useless. You need an ISP with balls and loyalty to his customers who sees DDoS as a technical challenge rather than as a financial problem.
My server is currently hosted by [ISP name removed]. That’s one ISP that I can vouch for; another is xs4all.nl. There are many more around, but hard to find. A simple rule of thumb might say that if an ISP has made a name for resisting attacks on freedom of speech, he will most likely see a DDoS in the same light and resist it as well.
Run on a dedicated unix-like system and have root
If you share system resources with a bunch of other people, every attempt to resist a DDoS against your site will have negative to catastrophic implications for all the other sites on the same server. It is very hard, morally as well as financially, to kill a hundred sites in order to save one, so no ISP will ever do that. You can have a dedicated server for as little as USD 70 per month (including the server itself, rack space, bandwidth, the works) and, if you can’t even afford that, you can still share it with a friend or two who will back you when needed.
If you are on Windoze server, you lack most of the tools that you need to resist a DDoS attack. Not that it matters much; you’ll probably get hacked anyway long before you get DDoS’ed. That crap isn’t made for the internet, so stay away from it.
Most dedicated *nix servers come with root access, but not all of them. If you don’t have root there is pitifully little you can do against a DDoS, so you are completely dependent on the ISP to do it for you. In these cases the ISP will typically charge you for every requested action or every hour of work or fraction thereof, thus converting a network attack to an attack on your wallet. Since it’s far easier to get technical help than to get financial help, you want to have root even if you are totally clueless in matters technical.
Separate static content from dynamic. Do not create dynamic content unnecessarily
It seems to be a fashion to use PHP where plain HTML would work just fine. Don’t. Use server-side scripting where you actually need it and keep the rest static. For various reasons (software requirements, processing time, OS caching), the difference is huge. Illustrative example: right now, [a server] reports
Currently blocked: 21003
Load: 0.22, 0.67, 0.81
but less than two dozen bots were enough to bring [that server's forum] down a month ago, although the forum runs on what is probably the leanest and fastest message board software around.
Have a couple of spare IP addresses, solid DNS and full control of it
You need spare IP addresses to separate static content from dynamic. You might also need them to play tricks with DNS.
DNS is not resource-consuming, so therefore not easy to crash, but if you lose it you lose everything. Protecting DNS is thus a first priority.
Most DDoS bots are pretty stupid, so there is a lot you can do by changing the IP address of your site (perhaps selectively, as I did when I sent [attacker]‘s bots to haunt to his own server).
I use a commercial DNS service, nettica.com, as advertised primary and secondary, while in reality both their servers are slaves to my “ghost” master. This approach combines two advantages: Nettica’s DNS servers can take a zillion times more load than my server could ever cope with, yet I have full control of every single parameter in my DNS zones. Cost: USD 50 per 100 domains per year, i.e. negligible.
If all this is technically above your head, the simple advice is: make sure that your DNS is run by a big boy and see to it that you can change things yourself, e.g. through a web interface or such, without any need to call the helpdesk. If these conditions are true, you can always get help with the rest if and when you need it.
Always have an off-site up-to-date backup of your entire server
All of the above suggestions are general precautions, not actual protections. Therefore, even if you have followed them all correctly, an unexpected DDoS attack is still likely to bring your server down before you can react. If you have misjudged something, e.g. your ISP’s propensity to null-route you or the capacity of your server, it might then be too late to recover. A full backup in a place that can’t be affected by the DDoS, e.g. at home, gives you the option of a fresh start with a new machine at a new ISP. The fact that it also protects you against hardware failures and other mishaps is free extra bonus in this context. Never ever neglect regular backups.
Having some kind of remote monitoring and early warning system will buy you very valuable time in the initial stage of an attack. There are many free monitoring tools out there, all you have to do is use them before you need them. There is no end to what you can do, up to having the system call your cellphone in case of trouble to tell you what’s wrong (check nagios, festival and asterisk), but even the most primitive monitoring, like manually paying attention to website statistics and logwatch mails, is far better than nothing.
Be generous with memory and trim your server
RAM is cheap these days and memory exhaustion can be a serious problem during a DDoS. Invest in a little more RAM than you really need before you get DDoS’ed and you will be happy you did if and when you do get DDoS’ed.
Try to match your server configuration to your hardware as best you can. This is not an easy task and you’ll never get it perfectly right, but some trimming is better than none at all. This applies to everything, from kernel parameters to webserver settings, and has to be done by someone who knows how to do it. Get help to do it before you actually need to.
There are more things you can do if are expecting a DDoS attack, but they come at a cost, either to your CPU or to your wallet, so you might prefer to wait with them until there is an actual need for them.
Rate-limit service requests on your firewall
The other day I gave an example of HTTP request limiting through iptables in the “server meltdown” thread. You can use the same method to limit ping replies, FTP requests, whatever.
Application-level attacks (e.g. excessive HTTP requests) can fairly easily be thwarted on the attacked machine itself. Network-level attacks (e.g. ICMP attacks saturating your uplink) can only be fought higher upstream. That is: if incoming packets saturate your connection, blocking them when they reach your server doesn’t help; you need to block them earlier, at a place where the available bandwidth is greater than what the attack can possibly saturate. Therefore, if you really need to, you might want to pay to place a separate hardware firewall one or two steps upstream from your server.
Instead of one server at one place you could have a hundred servers all over the globe. That would make it much harder to DDoS you but much easier to bankrupt you. The way around this dilemma is scalability on demand.
I mentioned coral as one way of scaling [in previous discussion]. Another way is Amazon’s EC2. They rent out virtual servers by the hour, ten (US) cents an hour plus traffic. I used EC2 for my DNS tricks [against recent attacks] and I am ready to use it for pure website serving anytime, if, when and only while it’s actually needed.
The way EC2 works is that you rent a virtual server, customize it and configure it to taste, then save it as an image on Amazon’s servers and dump the running server. Once this is done, you can start any number of identical virtual servers from the same image within minutes. For less than a dollar per month that it costs to have that base image stored, you have the possibility to have one, ten or three hundred identical servers with unlimited bandwidth up and running in just a few minutes.
In the long run this could be very expensive, but the psychological effect that such a cluster can have on an attacker is comparable to Akamai’s at a negligible fraction of the cost.
Just recently Amazon started offering a similar MySQL cluster service. I haven’t tested it, but if it only lives up to half of what they promise it would allow you to run a dynamic website at network speed and near-zero database overhead. Combining that with a bunch of EC2 cloned front ends, you could match and exceed the serving capacity of CNN and BBC put together for just a few dollars per hour.
Some DDoS attacks are harder to defend against than others because they can spoof the IP address of origin. Typically, TCP attacks with spoofed origin address are useless, while ICMP and UDP attacks with spoofed origin address are fully efficient. Typically, once a packet has arrived at its final destination there is no way to know whether its origin is spoofed or not. The opposite is true if stated origins can be compared to incoming routes at the entry points of a multi-homed network.
If you can rely on the cooperation of your ISP and his upstream ISPs, source-spoofed attacks are a major nuisance but not impossible to deal with. However, sorting out spoofed packets is work-intensive, so usually you can’t count on this degree of upstream cooperation. Having two servers in far-apart locations can help you identify the sources of spoofed packets yourself, thus reduce your work request on your upstream ISPs, thus increase the probability of them helping you out.
Besides, if you anyway have to pay for DDoS traffic, distributing your servers is a way to reduce the overall traffic costs.
Coping with an attack
Once you actually find yourself under attack, there are two almost universal truths: Precisely because a DDoS attack is distributed, it will almost always have more network resources than you, thus be stronger than you. And precisely because a DDoS attack has to be distributed, it will almost always be less intelligent than you. Thus, your best bet is to outsmart it, not to outmuscle it.
Find the attack’s weakness in the form of a pattern
Attack bots are pre-programmed and generic; they have to be, or else they wouldn’t work. Their behaviour can therefore never be identical to that of humans and this is a weakness that can be used to identify the bots and block them. [A specific example is discussed] Other giveaway patterns in HTTP attacks could be spidering without requesting or respecting /robots.txt, non-existent or fake User-Agent strings, too many incomplete requests and other such.
Under all circumstances you have to balance the need to identify the bots so that you can block them against the cost of doing so. There are situations where the identification and blocking process can put more burden on your machine than the DDoS itself, so you might have to resort to blind blocking of networks, countries etc.
Figure the most efficient place and manner to block the attack
You need to block as early as possible in the processing chain. For example, blocking an HTTP attack with .htaccess will do far more damage to your server than not blocking at all. Block with iptables if you must, use ip route if you can, block outside the attacked machine, on a router or external firewall, if possible.
Use serving techniques that the bots can’t deal with
Note that this is not a substitute for network-level blocking, but a complement to it that can help your server survive while you identify and block the bots.
Block unused protocols and ports upstream
If you’re not running a service and are not honouring requests for it, there is no reason why you should receive such requests in the first place. If under attack, ask your ISP to block at his entry points all protocols and ports that you’re not actually using.
Move unnecessary tasks out of the server or suspend them
Every server runs some non-essential applications; anything from website statistics to graphical management interfaces. Kill what you don’t really need and suspend temporarily “good to have but not crucial” applications or move them to some another machine.
If you are running all your services on one machine, as most people do, and have your master DNS, mail etc on the same machine as your attacked website, move the services that are not being attacked elsewhere. This won’t save the attacked machine resource-wise, but it will save those other services from going down because of the DDoS.
At times, depending on the kind of attack, you might be better off if you also move required services to another machine. For example, if a database-driven website is attacked with HTTP requests, moving the database to another machine will boost performance. Then again, if the same machine is attacked with a packet storm that saturates the network, moving the database out of that machine will only make things worse.
Remote logging instead of local can also be a very good idea, provided that your network connection is not threatened.
Try to block accurately and for the shortest time possible
Blocking individual attackers is better than blocking entire networks for the simple reason that if you block legitimate users you end up doing the attacker’s job for him. This will happen if you block entire networks and it will also happen if you block dynamic IP addresses past their recycling time.
Notify the bots’ ISPs
Botmasters hate to lose bots faster than new bots can be added to the botnet. Therefore, by notifying ISPs so that they can get their customers to clean up and recover infected machines, you increase the cost of the attack for whoever is behind it. Of course you can’t expect the same level of interest from every ISP; some take abuse reports very seriously, others delete them unread and you’ll be surprised at how many don’t even get them, but bounce them with “firstname.lastname@example.org: user over quota” errors and other such crap. Still, it’s always better that some ISPs can act than not notifying at all.
Finally, are you sure your home computer is clean, or could it be participating in a DDoS attack against someone while you’re reading this? Making sure to not attack others is in fact the first thing everyone can do to prevent attacks on himself.