Whatever you do against DoS-Attacks, think if what you do may actually increase the the load required to handle malicious or unwanted requests!
If you are using Linux then you should read this article:
Rule-based DoS attacks prevention shell script (from Linux Gazette)
It has the following topics:
- How to detect DoS attacks from
/var/log/secure file
- How to reduce redundant detected IPs
from the temporary file
- How to activate /sbin/iptables
- How to install the proposed shell script
Applying this without properly restricting the number of blocked IPs in iptables may intro a DoS-Vulnerability by increasing the requiered resources to handel unsolicited requests.
To reduces that risk use ipset to match IP-Addresses in iptables.
Also, read about ssh dictionary attack prevention using iptables. (enabling iptables with stateful firewalling as suggested here does not protect against most DoS-Attacks against but may actually ease DoS-Attacks that pollute your RAM with useless state info.)
New to Linux? read the Windows-to-Linux roadmap: Part 5. Linux logging of IBM.
Good Luck!