Ubuntu

Almost every day, I log in to my Ubuntu Server edition distro and one day I noticed something odd with the disk space. I noticed that it shot up from around 50% to 80% of 6.5GB. I’ve had this server since Hardy Heron days (8.04), it is now on Trusty Thar (14.04.3) Bionic Beaver (18.04.1), and never once it consumed more than 60% of the allocated disk space. That said, I was curious why all of a sudden the disk space became so big when I did not even install new packages.

The df -ah command was not very helpful since it only listed that root directory was consuming ~80%. I needed to figure out which directory was actually consuming those ~30% of disk space. After doing a web search, I found a command that allowed me to discover an important log that I should have known from the beginning.

$ sudo du -a / | sort -n -r | head -n 10
3571449	/
2589064	/var
2065580	/var/log
1762760	/var/log/btmp.1
613604	/usr
304008	/lib
274824	/var/cache
239600	/var/lib

Brute Force Attack Discovery

This was when I discovered the /var/log/auth.log file. Yes, I am still a Linux newbie. Every authentication attempt is listed here and also their IP address. That’s when I saw a bunch of SSH connections from different IP addresses I do not recognize and different usernames that the system does not even have or has been disabled. Most of the username I’ve seen are ubnt (Ubiquiti’s username on a lot of their products), pi, admin, etc. There was one IP address that has been brute forcing my box for a month without me knowing! Below is a snippet of my compressed auth.log file.

$ zcat /var/log/auth.log.4.gz | grep -v CRON | tail -n 500
Dec 29 06:43:17 ubuntu sshd[17503]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54 user=root
Dec 29 06:43:18 ubuntu sshd[17501]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 59428 ssh2]
Dec 29 06:43:18 ubuntu sshd[17501]: Received disconnect from 43.229.53.54: 11:  [preauth]
Dec 29 06:43:18 ubuntu sshd[17501]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:18 ubuntu sshd[17521]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:18 ubuntu sshd[17523]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:20 ubuntu sshd[17509]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 11637 ssh2]
Dec 29 06:43:20 ubuntu sshd[17509]: Received disconnect from 43.229.53.54: 11:  [preauth]
Dec 29 06:43:20 ubuntu sshd[17509]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:20 ubuntu sshd[17521]: Failed password for root from 43.229.53.54 port 22024 ssh2
Dec 29 06:43:20 ubuntu sshd[17511]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 11819 ssh2]

Options

There are several options one can implement to mitigate SSH brute force attack. One option is to not allow passwords and just use SSH keys. This is not a good option for me because I want to use this server with any computer and without using any type of keys. That said, password-based authentication is what I need. Another option is to implement two-factor authentication (2FA), which I covered here.

Firewall Option

Initially, I decided to start blocking the IP addresses I’ve seen in auth.log using my PA-200. It worked for a while, but every day I see new IPs popping. I then decided to implement Geo-based IP rule to lower the amount of attack. While it lessens the attacks significantly, I still needed something to help with the attacks that still goes through the firewall.

Enter Fail2ban

This was suggested by a friend of mine, @guerilla7. Thanks, Ron! Fail2ban scans log files (eg. /var/log/auth.log) and bans IPs (using iptables) that show malicious signs – too many password failures, seeking for exploits, etc. By default, Fail2ban monitors the /var/log/auth.log only. Obviously, this can be configured so that it can monitor more log files.

Installation and Configuration

The installation of Fail2ban will vary depending on your distro. Since Ubuntu is Debian based distro, the package manager is apt-get. It is very simple to install in Ubuntu. The command below is how to install the software and its dependencies.

$ sudo apt-get install fail2ban

Once everything is installed, it is time to configure Fail2ban. But, before we edit the configuration file that Fail2ban uses, it is a good idea to use a different file for custom configurations since the original configuration file can be overwritten by updates. That said, we need to create a copy of the configuration file.

$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Once copied, we are now ready to configure the jail.local file.

$ sudo vi /etc/fail2ban/jail.local

The configuration file is around 480 lines (including comments), but don’t be afraid because only a few lines will be changed. Of course, that depends on one’s needs. For my needs, I only touched four lines and are listed below.

ignoreip = 127.0.0.1/8 192.168.1.0/24
bantime = -1
findtime = 31536000
maxretry = 3

The ignoreip is basically the whitelist.

The bantime sets the length of time that a client will be banned for failed authentication attempts. The negative value sets it forever. This feature was added to version 0.6.1 (03/2006).

The maxretry and findtime parameters work together in establishing the conditions under which a client is determined by an unauthorized user. By default, the findtime is set to 600 seconds (10 minutes) and maxretry to 3 attempts. This means that Fail2ban will ban an unauthorized user when it attempts to log in three times within 10-minute window. Above configuration is 365-day window. This should cover slow brute force attacks but the majority of the brute force attacks I have seen are within 2-second window.

The changes on the configuration file will not take effect until the restart so sudo /etc/init.d/fail2ban restart must be issued.

Optional Configurations

Feel free to skip these configurations if you don’t find them useful.

Persistent Ban

While the above changes are good enough, the bans are not persistent. Once the server or Fail2ban service was restarted, the banned IPs will not survive. Some will be fine with that configuration, but that is unacceptable for me. I want to be the one who will unban IP addresses not because the server or service was restarted.

Update: As of Fail2ban 0.9.x, the bans are now persistent, by default, after reboot or restart. It now maintains a database found in /var/lib/fail2ban/fail2ban.sqlite3 file. That said, for new installs, there is no need to modify the iptables-multiport.conf file. However, if you already have existing ip.blacklist file, then you may want to still modify the iptables-multiport.conf file.

If you want to verify, look at the fail2ban.conf file and look for the dbfile section. You’ll also see the dbpurgeage section that has a 1-day setting. I think this setting is ignored so long as the bantime is set to any negative number.

$ cat /etc/fail2ban/fail2ban.conf
<-- Output omitted for brevity -->
# Options: dbfile
# Notes.: Set the file for the fail2ban persistent data to be stored.
#         A value of ":memory:" means database is only stored in memory
#         and data is lost when fail2ban is stopped.
#         A value of "None" disables the database.
# Values: [ None :memory: FILE ] Default: /var/lib/fail2ban/fail2ban.sqlite3
dbfile = /var/lib/fail2ban/fail2ban.sqlite3
# Options: dbpurgeage
# Notes.: Sets age at which bans should be purged from the database
# Values: [ SECONDS ] Default: 86400 (24hours)
dbpurgeage = 1d

First Step

The first step in making the ban persistent is to create a file where the list of banned IPs will be added. So that when the service gets restarted, for whatever reason, the software will load the file and issue the proper commands to re-ban the IPs. To create the file, issue the sudo touch /etc/fail2ban/ip.blacklist command. This will create a blank file called ip.blacklist. Feel free to call it different than mine but make sure to use the same file name on the configuration on the next step.

Second Step

The second step is to verify that we are actually going to edit the right configuration file. This is done by viewing the /etc/fail2ban/jail.local file and look for the first banaction = iptables-multiport configuration. At this time of writing, Fail2ban 0.8.11-1 on Ubuntu 14.04.3 is using iptables-multiport, which points to the iptables-multiport.conf file. It’s always a good idea to save a backup configuration, so issue the sudo cp /etc/fail2ban/action.d/iptables-multiport.conf /etc/fail2ban/action.d/iptables-multiport.conf.bak command.

Once completed, we can now move on to the final step. Edit the configuration iptables-multiport.conf file. To edit the iptables-multiport configuration file, issue the sudo vi /etc/fail2ban/action.d/iptables-multiport.conf command. The configuration file has around 70 configuration lines including the comments. There are only two configuration sections that we need to edit. These two are the actionstart and actionban section. The configuration will look as below.

actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> -m multiport --dports <port> -j fail2ban-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I fail2ban-<name> 1 -s $IP -j DROP; done; fi
actionban = if ! iptables -C fail2ban-<name> -s <ip> -j DROP; then iptables -I fail2ban-<name> 1 -s <ip> -j DROP; fi
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Update: Jim commented that the Fail2ban 0.9.3 on Ubuntu 16.04 changed from fail2ban to f2b. Thanks for the comment Jim!

actionstart = iptables -N f2b-<name>
              iptables -A f2b-<name> -j RETURN
              iptables -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done; fi
actionban = if ! iptables -C f2b-<name> -s <ip> -j DROP; then iptables -I f2b-<name> 1 -s <ip> -j DROP; fi
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Update: With Fail2ban 0.10.2 on Ubuntu 18.04, the default config was slightly changed. However, the line that we added on previous versions remains the same.

actionstart = <iptables> -N f2b-<name>
              <iptables> -A f2b-<name> -j <returntype>
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done; fi
actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Since we’ve made a modification to a configuration file, we need to restart the service by issuing sudo /etc/init.d/fail2ban restart or sudo service fail2ban restart command.

Note: I haven’t figured out why the IPs in the ip.blacklist file do not load after reboot or service restart. It will only add the list once a new failed SSH attempt has been made.

DROP vs REJECT

At the time this post was written, fail2ban used DROP as the default block type. Now, they changed the behavior to REJECT with an ICMP message of unreachable.

The biggest difference between the two is that DROP won’t send anything, while REJECT will send a message back to the source.

If you want to change the block type to DROP, then edit the /etc/fail2ban/action.d/iptables-common.conf file. The configuration below shows that I commented out the default behavior and changed it to DROP instead.

$ sudo more /etc/fail2ban/action.d/iptables-common.conf | grep "blocktype = "
# blocktype = REJECT --reject-with icmp-port-unreachable
blocktype = DROP
# blocktype = REJECT --reject-with icmp6-port-unreachable
blocktype = DROP

Verification

Depending on how often the attack occurs, check the iptables after several hours or a day. To check the iptables, issue the command below.

$ sudo iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
fail2ban-ssh  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 22
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Chain fail2ban-ssh (1 references)
target     prot opt source               destination
DROP       all  --  43.229.53.54         0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Manually unbanning an IP address

To unban an IP address, issue the command below.

$ sudo fail2ban-client set <jail_name> unbanip <ip_address>
# Example using Fail2ban 0.8.11
$ sudo fail2ban-client set ssh unbanip 43.229.53.54
43.229.53.54
$ sudo fail2ban-client reload
# Example using Fail2ban 0.9.3 and 0.10.2
$ sudo fail2ban-client set sshd unbanip 43.229.53.54
43.229.53.54
$ sudo fail2ban-client reload

While the command above is enough without the optional configuration (discussed above), this command is not the only thing needed with the optional configuration since the ip.blacklist file still contains the IP address that we’re trying to unban. If the server or service was restarted, then the IP address will be banned again. That said, it is necessary to take it out from the ip.blacklist file. To do this, issue the command below.

$ sudo sed --in-place '/<ip>,<name>/d' /etc/fail2ban/ip.blacklist
# Example
$ sudo sed --in-place '/43.229.53.54,ssh/d' /etc/fail2ban/ip.blacklist
$ sudo fail2ban-client reload

When unbanning fails

When someone issues the sudo fail2ban-client reload command then there is a very high chance that the user will encounter an error message similar to the one below.

$ sudo fail2ban-client set ssh unbanip 58.218.211.198
ERROR  NOK: ('IP 58.218.211.198 is not banned',)
IP 58.218.211.198 is not banned
# Example using Fail2ban 0.10.2
$ sudo fail2ban-client set sshd unbanip 58.218.211.198
ERROR  NOK: ('IP 58.218.211.198 is not banned',)
IP 58.218.211.198 is not banned

The IP can be still unbanned by the following:

$ sudo sed --in-place '/58.218.211.198,ssh/d' /etc/fail2ban/ip.blacklist
$ sudo fail2ban-client reload
$ sudo iptables -L -n | grep 58.218.211.198

Let’s say deleting the IP address from the blacklist file and reloading fail2ban didn’t work like what I experienced recently. The IP address that I was trying to unban kept coming back. I had to find another way to unban it using the iptables command. Here’s what I did to unban the IP address.

$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
655    DROP       all  --  58.218.211.198       0.0.0.0/0
$ sudo iptables -D fail2ban-ssh 655
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
$ sudo fail2ban-client reload
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
# Example using 0.10.2
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
655    DROP       all  --  58.218.211.198       0.0.0.0/0
$ sudo iptables -D f2b-sshd 655
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
$ sudo fail2ban-client reload
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198

Though, this seems to be a very rare occasion since I tried unbanning another IP address using the method in the manually unbanning section and it worked just fine.

Thoughts

This is exactly the software I was looking for. It is automated which means I no longer need to check auth.log and block it on my Ubiquiti EdgeRouter Lite. I did transfer the rules from my PA-200 to my new router/firewall, however. Though, I am still getting used to the creation of firewall rules because it is not as intuitive as creating rules on Cisco ASA or Palo Alto Networks firewall. While this software automatically blocks failed attempts, it does not protect from weak passwords. It is still recommended to use strong passwords.

UPDATE: I no longer use the EdgeRouter to protect my DMZ – I now use pfSense. The PA-200 will eventually be used on some user traffic but will be limited. Mostly, to learn more about how to configure it.

Want to learn more about Linux System Administration?

UNIX and Linux System Administration Handbook, 4th Edition

References

List based permanent bans with fail2ban
How To Protect SSH with fail2ban on Debian 7
How to unban an IP properly with fail2ban

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.