It’s a common question: “Should I use fail2ban or denyhosts?”. You can search for a more thorough answer, but I say use both and here’s why: while fail2ban is more flexible, denyhost is great for two very specific use-cases.
fail2ban greps through your log files, looking for patterns (user-defined regexes) that match specific behaviour (e.g. failed SSH login attempts, 404 attempts on HTTP URLs that would exist on your local machine if it were compromised). Once a user-defined threshold for such behaviour is exceeded, fail2ban blocks (using iptables) the suspicious (or malicious) host [NOTE: The action is user-configurable, so fail2ban can block using denyhosts, or email an admin, etc].
denyhosts is a simpler tool: denyhosts only looks at failed SSH login attempts and puts suspicious/malicious hosts into the hosts.deny file.
denyhosts however, is great for two very specific use-cases: If you watch your authentication logs (using something like logwatch), you’ll notice that the malicious hosts target specific usernames which usually don’t exist on your machine (e.g. unknown) or aren’t intended to be logged into directly via SSH (e.g. www-data).
denyhosts combats these scenarios using:
1. DENY_THRESHOLD_INVALID : The number of times that a host can attempt to authenticate as a user that doesn’t exist on the system before being blocked. I set this number very low because there’s only one user on most of my machines, and that’s me. So unless it’s a typo, I’m only ever entering one username.
2. DENY_THRESHOLD_RESTRICTED : The number of times that a host can attempt to authenticate as a user that is on your restricted list (/var/lib/denyhosts/restricted-usernames). I set this to one. We all know hackers/crackers like to target common usernames and passwords. So for users that shouldn’t be allowed to directly log in over SSH (e.g. www-data), I put them into the restricted-usernames list.