NetworkJutsu

Network Security Consulting | San Francisco Bay Area

Security

Adding Two-Factor Authentication to FreeRADIUS

By 17 Comments

In my previous post, I talked about enabling two-factor authentication (2FA) for my public facing Linux host. In today’s post, I will talk about integrating Google Authenticator PAM to FreeRADIUS. As a result, any hosts that are pointed to my RADIUS server will have the 2FA functionality.

Update: Migrated FreeRADIUS with Google Authenticator to a Docker container
Update: FreeRADIUS 3.0 with Two-Factor Authentication (2FA)

Installing FreeRADIUS and Google Authenticator PAM

While there are several RADIUS software out there, FreeRADIUS is one of the most popular RADIUS software of choice in Linux. Since it has PAM library, this is also perfect for integrating it with Google Authenticator PAM. If you want to know more about FreeRADIUS, you might want to check this book out. I have not read it so read through the reviews to see if that will work for your needs.

Installing FreeRADIUS and Google Authenticator on Ubuntu 16.04 is very easy. All we need is to issue one line command. I added NTP package here since my Google Authenticator configuration is TOTP based. If one went through the Ubuntu installation properly, there might not be a need for this so long as the system is syncing to the time correctly.

$ sudo apt-get install freeradius libpam-google-authenticator -y

Configuring FreeRADIUS

After the package installation, the next step is to set up FreeRADIUS by editing configuration files. There are four config files we need to edit to complete this setup. By no means, one needs to follow the order.

First config file

The first config file that we need to edit is the /etc/freeradius/radiusd.conf file. There are two ways in configuring this and it seems that the most popular option is the one with FreeRADIUS running as root. For some people, this is not acceptable so I included instructions below where we’ll leave it as the default configuration.

Option 1 – Run as root

According to my limited research, the need to change the user and group to root is because of how both FreeRADIUS and Google Authenticator PAM works. My observation seems to indicate that FreeRADIUS will also need to access the secret key (.google_authenticator) in each user’s home directory – I could be totally wrong with this. My Linux boxes have encrypted home directories so only the owner and root can access these. That said, letting FreeRADIUS run as root will have access to the necessary files.

$ sudo vi /etc/freeradius/radiusd.conf

We’ll now need to find the lines user = and group =. The default configuration is set to freerad. Change both of them to root.

user = root
group = root

Option 2 – Use default configuration

As mentioned, we can just leave the file as default. I will explain more about this once we get to the section where we need to edit the /etc/pam.d/radiusd file.

Second config file

The the next config file that we need to edit is the /etc/freeradius/users file. This file will instruct FreeRADIUS to use PAM libraries to authenticate users as the default.

$ sudo vi /etc/freeradius/users

Add the lines found below. I usually like to add lines at the end of the file. Add the line after all the commented text of the file, just before the DEFAULT Framed Protocol == PPP line. This will ensure that this line will take precedence. I found out the hard way when I was troubleshooting an issue with L2TP over IPsec authentication.

# Instruct FreeRADIUS to use PAM to authenticate users
DEFAULT Auth-Type := PAM

Third config file

The second to the last config file on our list to be edited is the /etc/freeradius/sites-enabled/default file. This file tells FreeRADIUS to enable PAM authentication. We just need to edit one line here.

$ sudo vi /etc/freeradius/sites-enabled/default

Once the file is open, look for the following lines:

        #  Pluggable Authentication Modules.
#        pam

We now need to uncomment the pam line to enable it. It should look like this now:

        #  Pluggable Authentication Modules.
        pam

Fourth config file

Finally, the last FreeRADIUS config file that we need to change is the /etc/freeradius/clients.conf. This is where we can set up our secret key that is used by the clients to connect to the RADIUS server. Please change the default secret key to random alphanumeric characters. Use a key generator to generate the secret to make things life a little easier. For demo purposes, I will be using the default secret. To change the secret, look for secret = testing123 line.

$ sudo vi /etc/freeradius/clients.conf
<-- Output omitted for brevity -->
secret          = my_super_awesome_strong_secret
client rtr {
	ipaddr = 192.168.200.1
	secret = my_super_awesome_strong_secret 
}

As usual in Linux, when a configuration file has been changed, then the service needs to be restarted for the changes to take effect. To restart FreeRADIUS daemon, issue the sudo service freeradius restart command.

Configuring FreeRADIUS PAM

Since we instructed FreeRADIUS to use PAM to authenticate users, we need to configure the /etc/pam.d/radiusd file and instruct it to integrate Google Authenticator PAM. By default, the file will look something like this:

<-- Output omitted for brevity -->
@include common-auth
@include common-account
@include common-password
@include common-session

Option 1

If you picked the first option in the FreeRADIUS configuration section, then you need to comment those four lines above and add two lines. The file should look like this:

#@include common-auth
#@include common-account
#@include common-password
#@include common-session
auth requisite pam_google_authenticator.so forward_pass
auth required pam_unix.so use_first_pass

Option 2

If you left the /etc/freeradius/radiusd.conf file alone, then it becomes a little bit more complicated setup. Also, you will notice that my instructions are what I will consider a workaround to AppArmor (I am guessing this is the real issue). You will see why later in the next section, after the generating Google Authenticator secret key. Anyway, the /etc/pam.d/radiusd file should look like this:

#@include common-auth
#@include common-account
#@include common-password
#@include common-session
auth requisite pam_google_authenticator.so forward_pass secret=/etc/freeradius/${USER}/.google_authenticator user=freerad
auth required pam_unix.so use_first_pass

Google Authenticator Secret Key

I’ve already covered the generation of the secret key in my previous post, so look for the generating Google Authenticator secret key section. Once you are done generating secret keys, come back to this page. If you picked the first option throughout this tutorial, then skip this section and go to the verification section.

If you picked the second option, then we’ll need to do additional steps to make this work. Again, you do not have to follow the order in which they are listed here.

We first need to create a directory equal to the user account that we’re working on. In this scenario, we’ll use user account named test.

$ sudo mkdir /etc/freeradius/test

Then, we need to change the owner of the directory that we just created.

$ sudo chown freerad:freerad /etc/freeradius/test

The second to the last step is to copy the secret key to the directory that we just created.

$ sudo cp .google_authenticator /etc/freeradius/test/.google_authenticator

Finally, we need to change the owner of the file.

$ sudo chown freerad:freerad /etc/freeradius/test/.google_authenticator

If I ever learn more about AppArmor, then I will update this blog post because I think this is the real issue why it’s failing. I did try creating an AppArmor profile, but testing shows that I was still failing. When I looked at the /var/log/auth.log file, I saw an error message that looked like this:

Oct 10 21:24:53 radius radiusd(pam_google_authenticator)[18433]: Failed to update secret file "/etc/freeradius/test/.google_authenticator"

Verification

We now need to test to make sure that we can successfully authenticate. FreeRADIUS software package includes a simple tool that we can use to directly query the daemon with requests. The command format is radtest test <password+google authenticator token> localhost 18120 <RADIUS secret key>. The password and Google Authenticator token should not have space in between. Below shows the syntax that I used to test my configuration and the test result.

$ radtest test testing1234803732 localhost 18120 testing123
Sending Access-Request of id 79 to 127.0.0.1 port 1812
	User-Name = "test"
	User-Password = "testing1234803732"
	NAS-IP-Address = 127.0.1.1
	NAS-Port = 18120
	Message-Authenticator = 0x00000000000000000000000000000000
rad_recv: Access-Accept packet from host 127.0.0.1 port 1812, id=79, length=20

Final Words

I’d like to speculate that a lot of people would pick the first option since the second option involves a lot more steps. Also, this isn’t a scalable solution so I think it will turn off quite a number of people. Imagine doing the steps above for multiple accounts. It’s going to be a lot of administrative work. For my purpose, this is a perfectly acceptable solution because I only have one account in my RADIUS server. It is probably an overkill to be creating a FreeRADIUS server instance for home use, but some may argue that nothing is overkill in security. The more secure you are, the better.

You might also like to read

FreeRADIUS 3.0 with Two-Factor Authentication
Securing SSH with Google Authenticator
Migrated FreeRADIUS with Google Authenticator to a Docker container
Adding Two-Factor Authentication to TACACS+

References

Integrating Google Authenticator PAM module with FreeRADIUS Server
Freeradius and Google Authenticator
GitHub Google Authenticator

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

Securing SSH with Google Authenticator

By 2 Comments

In my old blog post, I talked about how to mitigate from persistent SSH brute force attack. While there are several options in mitigating SSH brute force attack, I opted to use the Fail2Ban option at the time. Today, I’ve decided to add another security layer to the host since this is a public facing server. This addition of security layer is based on defense in depth, which is an information assurance concept. As the title says, I will be using Google Authenticator to generate a time-based one-time password (TOTP) for two-step verification.

Related: What is multi-factor authentication?

It seems like two-factor authentication (2FA) is becoming a norm these days. More and more security professionals are pushing organizations to use 2FA for every sensitive systems and application. Understandably so, because the consensus is that password is no longer enough to protect accounts in this day and age. As a result, I’ve also decided to start implementing 2FA in my home devices.

Installing Google Authenticator PAM

Ubuntu has been my distro of choice for several years now, so all of my Linux-related tutorials have been on that. I will keep that going, mostly because I do not have time to learn another distro, like CentOS.

Installing Google Authenticator on Ubuntu 16.04 is a piece of cake. All you need is one command.

$ sudo apt-get install libpam-google-authenticator -y

Configuring SSH PAM

The Google Authenticator 2FA is accomplished by integrating into Linux’s Pluggable Authentication Modules (PAM) library. PAM is a way for programs to use an underlying authentication mechanism. With that said, we’ll need to configure PAM configuration to pass it to Google Authenticator. To do this, we need to edit the PAM configuration file for SSH.

$ sudo vi /etc/pam.d/sshd

At the bottom of the file, I added the following lines below. The first line is optional, but it’s always best to add comment lines in my opinion.

# Enable MFA using Google Authenticator PAM
auth required pam_google_authenticator.so nullok

If you want to understand the syntax, please check this site. Though, the article did not include what nullok argument means. The argument simply means that if a user hasn’t created secret key yet, then they’re still allowed to log in. My recommendation is to enable 2FA for all users so make sure to have all the users generate the secret key. Once all secret keys are generated, take out the nullok argument.

Configuring OpenSSH Daemon

The next step is to actually configure SSH to check for backend system (e.g. PAM) to use the challenge-response authentication method. To do this, we need to edit the OpenSSH configuration file.

$ sudo vi /etc/ssh/sshd_config

Once the file is open, look for the line with ChallengeResponseAuthentication no and change it to yes. Save the file and exit.

ChallengeResponseAuthentication yes

In Linux, changes to Linux configuration files require a service restart to take effect, for the most part. With that said, we need to restart SSH daemon (sshd). To restart SSH service, issue sudo service ssh restart command. Once sshd is back up, test to make sure that user can still log in without two-step verification.

Generating Google Authenticator Secret Key

The last step to make SSH 2FA work is to generate a secret key for the two-step verification. In this example, I created a test user account for demo purposes.

generating-secret-key

Use the Google Authenticator app on your mobile device and add the QR code.

Account with Google Authenticator secret key

In this example, SSH daemon is asking the user to enter Google Authenticator OTP.

$ ssh test@radius
Password: testing1234
Verification code: 664449
<-- Output omitted for brevity -->
test@radius:~$
google authenticator token

Account with no Google Authenticator secret key

Here the admin account does not have the Google Authenticator secret key yet. If one forgets to add the nullok argument, then the system will not allow user accounts without the secret key.

$ ssh admin@radius
Password:
<-- Output omitted for brevity -->
admin@radius:~$

Optional Configuration

It is probably a good idea to enable 2FA to the local login (console) too. So this way, anyone who has access to the physical machine will be subjected to two-step verification. To do this, we need to edit the login configuration file.

$ sudo vi /etc/pam.d/login

Once the file is open, add the lines below at the end of the file. Save the file and exit out.

# Enable MFA using Google Authenticator PAM
auth required pam_google_authenticator.so nullok

From now on, all user accounts with Google Authenticator secret key will need to enter a verification code.

Thoughts

Balancing security and convenience is one of the biggest dilemmas that information security professionals face. In a nutshell, making it convenient for users to access their account means it is less secure. In a perfect world (no bad guys) people would probably pick one password across all of their accounts. Since we’re not living in a perfect world, this leaves the accounts to be very insecure. If one account is compromised, then all of the other accounts could potentially be accessed by the bad guys.

The use of complex passwords and password managers are good first steps toward securing accounts. However, in today’s world, some of the information security professionals view password alone as an antiquated technique in securing accounts. As a result, a lot of online services have had multi-factor authentication feature. However, companies that provide these online services do not force accounts to use it. As a result, accounts are still at risk by relying on passwords alone.

You might also like to read

Adding Two-Factor Authentication to FreeRADIUS

Reference

How to Secure SSH with Google Authenticator’s Two-Factor Authentication

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

SSH Brute Force Attack

By 2 Comments

Almost every day, I log in to my Ubuntu Server edition distro and one day I noticed something odd with the disk space. I noticed that it shot up from around 50% to 80% of 6.5GB. I’ve had this server since Hardy Heron days (8.04), it is now on Trusty Thar (14.04.3) Bionic Beaver (18.04.1), and never once it consumed more than 60% of the allocated disk space. That said, I was curious why all of a sudden the disk space became so big when I did not even install new packages.

The df -ah command was not very helpful since it only listed that root directory was consuming ~80%. I needed to figure out which directory was actually consuming those ~30% of disk space. After doing a web search, I found a command that allowed me to discover an important log that I should have known from the beginning.

$ sudo du -a / | sort -n -r | head -n 10
3571449	/
2589064	/var
2065580	/var/log
1762760	/var/log/btmp.1
613604	/usr
304008	/lib
274824	/var/cache
239600	/var/lib

Brute Force Attack Discovery

This was when I discovered the /var/log/auth.log file. Yes, I am still a Linux newbie. Every authentication attempt is listed here and also their IP address. That’s when I saw a bunch of SSH connections from different IP addresses I do not recognize and different usernames that the system does not even have or has been disabled. Most of the username I’ve seen are ubnt (Ubiquiti’s username on a lot of their products), pi, admin, etc. There was one IP address that has been brute forcing my box for a month without me knowing! Below is a snippet of my compressed auth.log file.

$ zcat /var/log/auth.log.4.gz | grep -v CRON | tail -n 500
Dec 29 06:43:17 ubuntu sshd[17503]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54 user=root
Dec 29 06:43:18 ubuntu sshd[17501]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 59428 ssh2]
Dec 29 06:43:18 ubuntu sshd[17501]: Received disconnect from 43.229.53.54: 11:  [preauth]
Dec 29 06:43:18 ubuntu sshd[17501]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:18 ubuntu sshd[17521]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:18 ubuntu sshd[17523]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:20 ubuntu sshd[17509]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 11637 ssh2]
Dec 29 06:43:20 ubuntu sshd[17509]: Received disconnect from 43.229.53.54: 11:  [preauth]
Dec 29 06:43:20 ubuntu sshd[17509]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:20 ubuntu sshd[17521]: Failed password for root from 43.229.53.54 port 22024 ssh2
Dec 29 06:43:20 ubuntu sshd[17511]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 11819 ssh2]

Options

There are several options one can implement to mitigate SSH brute force attack. One option is to not allow passwords and just use SSH keys. This is not a good option for me because I want to use this server with any computer and without using any type of keys. That said, password-based authentication is what I need. Another option is to implement two-factor authentication (2FA), which I covered here.

Firewall Option

Initially, I decided to start blocking the IP addresses I’ve seen in auth.log using my PA-200. It worked for a while, but every day I see new IPs popping. I then decided to implement Geo-based IP rule to lower the amount of attack. While it lessens the attacks significantly, I still needed something to help with the attacks that still goes through the firewall.

Enter Fail2ban

This was suggested by a friend of mine, @guerilla7. Thanks, Ron! Fail2ban scans log files (eg. /var/log/auth.log) and bans IPs (using iptables) that show malicious signs – too many password failures, seeking for exploits, etc. By default, Fail2ban monitors the /var/log/auth.log only. Obviously, this can be configured so that it can monitor more log files.

Installation and Configuration

The installation of Fail2ban will vary depending on your distro. Since Ubuntu is Debian based distro, the package manager is apt-get. It is very simple to install in Ubuntu. The command below is how to install the software and its dependencies.

$ sudo apt-get install fail2ban

Once everything is installed, it is time to configure Fail2ban. But, before we edit the configuration file that Fail2ban uses, it is a good idea to use a different file for custom configurations since the original configuration file can be overwritten by updates. That said, we need to create a copy of the configuration file.

$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Once copied, we are now ready to configure the jail.local file.

$ sudo vi /etc/fail2ban/jail.local

The configuration file is around 480 lines (including comments), but don’t be afraid because only a few lines will be changed. Of course, that depends on one’s needs. For my needs, I only touched four lines and are listed below.

ignoreip = 127.0.0.1/8 192.168.1.0/24
bantime = -1
findtime = 31536000
maxretry = 3

The ignoreip is basically the whitelist.

The bantime sets the length of time that a client will be banned for failed authentication attempts. The negative value sets it forever. This feature was added to version 0.6.1 (03/2006).

The maxretry and findtime parameters work together in establishing the conditions under which a client is determined by an unauthorized user. By default, the findtime is set to 600 seconds (10 minutes) and maxretry to 3 attempts. This means that Fail2ban will ban an unauthorized user when it attempts to log in three times within 10-minute window. Above configuration is 365-day window. This should cover slow brute force attacks but the majority of the brute force attacks I have seen are within 2-second window.

The changes on the configuration file will not take effect until the restart so sudo /etc/init.d/fail2ban restart must be issued.

Optional Configurations

Feel free to skip these configurations if you don’t find them useful.

Persistent Ban

While the above changes are good enough, the bans are not persistent. Once the server or Fail2ban service was restarted, the banned IPs will not survive. Some will be fine with that configuration, but that is unacceptable for me. I want to be the one who will unban IP addresses not because the server or service was restarted.

Update: As of Fail2ban 0.9.x, the bans are now persistent, by default, after reboot or restart. It now maintains a database found in /var/lib/fail2ban/fail2ban.sqlite3 file. That said, for new installs, there is no need to modify the iptables-multiport.conf file. However, if you already have existing ip.blacklist file, then you may want to still modify the iptables-multiport.conf file.

If you want to verify, look at the fail2ban.conf file and look for the dbfile section. You’ll also see the dbpurgeage section that has a 1-day setting. I think this setting is ignored so long as the bantime is set to any negative number.

$ cat /etc/fail2ban/fail2ban.conf
<-- Output omitted for brevity -->
# Options: dbfile
# Notes.: Set the file for the fail2ban persistent data to be stored.
#         A value of ":memory:" means database is only stored in memory
#         and data is lost when fail2ban is stopped.
#         A value of "None" disables the database.
# Values: [ None :memory: FILE ] Default: /var/lib/fail2ban/fail2ban.sqlite3
dbfile = /var/lib/fail2ban/fail2ban.sqlite3
# Options: dbpurgeage
# Notes.: Sets age at which bans should be purged from the database
# Values: [ SECONDS ] Default: 86400 (24hours)
dbpurgeage = 1d

First Step

The first step in making the ban persistent is to create a file where the list of banned IPs will be added. So that when the service gets restarted, for whatever reason, the software will load the file and issue the proper commands to re-ban the IPs. To create the file, issue the sudo touch /etc/fail2ban/ip.blacklist command. This will create a blank file called ip.blacklist. Feel free to call it different than mine but make sure to use the same file name on the configuration on the next step.

Second Step

The second step is to verify that we are actually going to edit the right configuration file. This is done by viewing the /etc/fail2ban/jail.local file and look for the first banaction = iptables-multiport configuration. At this time of writing, Fail2ban 0.8.11-1 on Ubuntu 14.04.3 is using iptables-multiport, which points to the iptables-multiport.conf file. It’s always a good idea to save a backup configuration, so issue the sudo cp /etc/fail2ban/action.d/iptables-multiport.conf /etc/fail2ban/action.d/iptables-multiport.conf.bak command.

Once completed, we can now move on to the final step. Edit the configuration iptables-multiport.conf file. To edit the iptables-multiport configuration file, issue the sudo vi /etc/fail2ban/action.d/iptables-multiport.conf command. The configuration file has around 70 configuration lines including the comments. There are only two configuration sections that we need to edit. These two are the actionstart and actionban section. The configuration will look as below.

actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> -m multiport --dports <port> -j fail2ban-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I fail2ban-<name> 1 -s $IP -j DROP; done; fi
actionban = if ! iptables -C fail2ban-<name> -s <ip> -j DROP; then iptables -I fail2ban-<name> 1 -s <ip> -j DROP; fi
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Update: Jim commented that the Fail2ban 0.9.3 on Ubuntu 16.04 changed from fail2ban to f2b. Thanks for the comment Jim!

actionstart = iptables -N f2b-<name>
              iptables -A f2b-<name> -j RETURN
              iptables -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done; fi
actionban = if ! iptables -C f2b-<name> -s <ip> -j DROP; then iptables -I f2b-<name> 1 -s <ip> -j DROP; fi
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Update: With Fail2ban 0.10.2 on Ubuntu 18.04, the default config was slightly changed. However, the line that we added on previous versions remains the same.

actionstart = <iptables> -N f2b-<name>
              <iptables> -A f2b-<name> -j <returntype>
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done; fi
actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Since we’ve made a modification to a configuration file, we need to restart the service by issuing sudo /etc/init.d/fail2ban restart or sudo service fail2ban restart command.

Note: I haven’t figured out why the IPs in the ip.blacklist file do not load after reboot or service restart. It will only add the list once a new failed SSH attempt has been made.

DROP vs REJECT

At the time this post was written, fail2ban used DROP as the default block type. Now, they changed the behavior to REJECT with an ICMP message of unreachable.

The biggest difference between the two is that DROP won’t send anything, while REJECT will send a message back to the source.

If you want to change the block type to DROP, then edit the /etc/fail2ban/action.d/iptables-common.conf file. The configuration below shows that I commented out the default behavior and changed it to DROP instead.

$ sudo more /etc/fail2ban/action.d/iptables-common.conf | grep "blocktype = "
# blocktype = REJECT --reject-with icmp-port-unreachable
blocktype = DROP
# blocktype = REJECT --reject-with icmp6-port-unreachable
blocktype = DROP

Verification

Depending on how often the attack occurs, check the iptables after several hours or a day. To check the iptables, issue the command below.

$ sudo iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
fail2ban-ssh  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 22
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Chain fail2ban-ssh (1 references)
target     prot opt source               destination
DROP       all  --  43.229.53.54         0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Manually unbanning an IP address

To unban an IP address, issue the command below.

$ sudo fail2ban-client set <jail_name> unbanip <ip_address>
# Example using Fail2ban 0.8.11
$ sudo fail2ban-client set ssh unbanip 43.229.53.54
43.229.53.54
$ sudo fail2ban-client reload
# Example using Fail2ban 0.9.3 and 0.10.2
$ sudo fail2ban-client set sshd unbanip 43.229.53.54
43.229.53.54
$ sudo fail2ban-client reload

While the command above is enough without the optional configuration (discussed above), this command is not the only thing needed with the optional configuration since the ip.blacklist file still contains the IP address that we’re trying to unban. If the server or service was restarted, then the IP address will be banned again. That said, it is necessary to take it out from the ip.blacklist file. To do this, issue the command below.

$ sudo sed --in-place '/<ip>,<name>/d' /etc/fail2ban/ip.blacklist
# Example
$ sudo sed --in-place '/43.229.53.54,ssh/d' /etc/fail2ban/ip.blacklist
$ sudo fail2ban-client reload

When unbanning fails

When someone issues the sudo fail2ban-client reload command then there is a very high chance that the user will encounter an error message similar to the one below.

$ sudo fail2ban-client set ssh unbanip 58.218.211.198
ERROR  NOK: ('IP 58.218.211.198 is not banned',)
IP 58.218.211.198 is not banned
# Example using Fail2ban 0.10.2
$ sudo fail2ban-client set sshd unbanip 58.218.211.198
ERROR  NOK: ('IP 58.218.211.198 is not banned',)
IP 58.218.211.198 is not banned

The IP can be still unbanned by the following:

$ sudo sed --in-place '/58.218.211.198,ssh/d' /etc/fail2ban/ip.blacklist
$ sudo fail2ban-client reload
$ sudo iptables -L -n | grep 58.218.211.198

Let’s say deleting the IP address from the blacklist file and reloading fail2ban didn’t work like what I experienced recently. The IP address that I was trying to unban kept coming back. I had to find another way to unban it using the iptables command. Here’s what I did to unban the IP address.

$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
655    DROP       all  --  58.218.211.198       0.0.0.0/0
$ sudo iptables -D fail2ban-ssh 655
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
$ sudo fail2ban-client reload
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
# Example using 0.10.2
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
655    DROP       all  --  58.218.211.198       0.0.0.0/0
$ sudo iptables -D f2b-sshd 655
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
$ sudo fail2ban-client reload
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198

Though, this seems to be a very rare occasion since I tried unbanning another IP address using the method in the manually unbanning section and it worked just fine.

Thoughts

This is exactly the software I was looking for. It is automated which means I no longer need to check auth.log and block it on my Ubiquiti EdgeRouter Lite. I did transfer the rules from my PA-200 to my new router/firewall, however. Though, I am still getting used to the creation of firewall rules because it is not as intuitive as creating rules on Cisco ASA or Palo Alto Networks firewall. While this software automatically blocks failed attempts, it does not protect from weak passwords. It is still recommended to use strong passwords.

UPDATE: I no longer use the EdgeRouter to protect my DMZ – I now use pfSense. The PA-200 will eventually be used on some user traffic but will be limited. Mostly, to learn more about how to configure it.

Want to learn more about Linux System Administration?

UNIX and Linux System Administration Handbook, 4th Edition

References

List based permanent bans with fail2ban
How To Protect SSH with fail2ban on Debian 7
How to unban an IP properly with fail2ban

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

Implementing Wired 802.1X

By 2 Comments

I first learned about 802.1X when I was studying for one of the CCNP exams, BCMSN exam (SWITCH equivalent), at Ohlone College. At the time, I assumed that the short material covered in the book was all of it. Of course, that was a bad assumption in my part. That’s probably a normal assumption of someone who at the time just finished Cisco Network Academy Program CCNA 1 to 4 and newly minted CCNA with no professional experience.

What is 802.1X?

Essentially, 802.1X is a security feature that provides a mechanism to authenticate devices before it can access network resources. While it’s a good idea to have this security feature implemented, I’ve worked for companies who didn’t have this feature or similar implemented or it’s on their roadmap. It’s a shame that it wasn’t on their roadmap a long time ago since it was ratified in 2001. Then again, implementing technologies have its challenges.

How it works?

802.1X_wired_protocols
Image from Wikipedia

While there are other sources that will explain this in detail, this post includes a very short description on how it works. Basically, when a device connects to the wired network, the authenticator (switch) will send an EAP message to the supplicant (computer). If the computer has a supplicant, it will send an EAP response to the authenticator. The authenticator will then send a RADIUS message to the authentication server (RADIUS server). The authentication server will then challenge the supplicant to verify its identity. Once verified, the device will then be able to connect to organization’s network resources.

Environment

Every organization has their own unique implementation of technologies, so gather what you can and go from there. For example, in this scenario the requirements were to have two sets of RADIUS servers: one for switch-based authentication and the other for port-based authentication. This seems to be an uncommon setup so it required some research to split the two sets of RADIUS servers. My initial assumption was that it wasn’t possible. That assumption is only correct in older code, but with IOS 15.x the feature is supported.

This is a multivendor organization so LLDP is used instead of CDP, which is disabled globally by default due to the switch template configuration.

A lot of users are using Apple notebooks and/or desktops and most of these users are running VMware Fusion to run Windows and/or Linux.

IP phones are ubiquitous so this requires a great deal of attention. If my memory serves me right, the 802.1X topic in BCMSN didn’t cover how to implement it with IP phones so Cisco’s documentation and Google were my friend during my research.

A lot of devices do not have supplicant and there are instances where PXE boot is needed.

In addition, there were some locations that need WoL (Wake on LAN) feature so that needs an attention as well.

Configuration

As mentioned earlier, the requirement is to have two separate RADIUS servers for both switch-based and port-based authentication. That said, let’s take a look on how to do this. But first, let me show you how it was done prior to IOS 15.x code. This command still works in 15.0(2), but you’ll receive a warning saying that it will soon be deprecated.

Old format

radius-server host 192.168.1.1 auth-port 1812 acct-port 1813
radius-server host 192.168.1.2 auth-port 1812 acct-port 1813
radius-server retransmit 1
radius-server timeout 2
radius-server key 7 hashkeyhere

Since the requirement is to split the RADIUS servers, we need to use the new format of specifying the RADIUS servers which will be needed when we create the AAA groups.

New format

radius server switch-auth1
 address ipv4 192.168.1.1 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server switch-auth2
 address ipv4 192.168.1.2 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server dot1x-auth1
 address ipv4 192.168.1.3 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server dot1x-auth2
 address ipv4 192.168.1.4 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere

Enable AAA

Once enabled, authentication method for 802.1X needs to be defined. I included the one for the switch-based authentication with the port-based authentication for completeness sake. RADIUS accounting is turned on as well since it is listed as best practice in Cisco’s deployment guide.

aaa new-model
aaa group server radius switch-auth
 server name switch-auth1
 server name switch-auth2
aaa group server radius dot1x-auth
 server name dot1x-auth1
 server name dot1x-auth2
aaa authentication login default group switch-auth enable
aaa authentication dot1x default group dot1x-auth
aaa accounting dot1x default start-stop group dot1x-auth
aaa authorization network default group dot1x-auth

Enable 802.1X

Issue the command below.

dot1x system-auth-control

Configure switch ports

Next step is to configure each switch port that will use 802.1X. This command will automatically include dot1x pae authenticator in the running configuration so don’t be alarmed if you see it there. This is to ensure that dot1x authentication still works on legacy configurations without manual intervention. NOTE: It seems to be that the IOS that I was using automatically included the dot1x pae authenticator command. That said, please make sure to add the command if you do not see it.

interface range g1/0/1 - 48
 ! Make sure that the ports should at least have switchport mode access or it won't take the commands.
 authentication port-control auto
 dot1x pae authenticator

In the IOS 12.x, this would’ve been a different command. The command in the old world is dot1x port-control auto.

Technically, the commands above are all we need to configure for the 802.1X to work. However, the environment in this scenario requires more things from us that we still need to address.

VoIP phones

Let’s address the IP phones first since it’s ubiquitous within the enterprise environment. By default, the interfaces are set to be single-host mode. This means only one MAC is allowed in the data VLAN. This mode technically allows another MAC address but on the voice VLAN and only if CDP is supported. Since CDP is disabled on all of the switches deployed in this scenario, this needs to be enabled. I included the single-host mode command below since it won’t show up in the running configuration because it is the default configuration.

cdp run
interface g1/0/1 - 48
 authentication host-mode single-host

While this configuration works, there are few things that we need to keep in mind. The single-host mode means only single MAC can be authenticated on a switch port. If a different MAC address is detected on a port after an endpoint has authenticated then a security violation is triggered on the port. This will cause the port to be in errdisabled state and will require a manual intervention unless errdisable recovery is configured.

Since the computers are daisy chained on the back of the Cisco phones, there are technically two MAC addresses that will be seen on the port. As mentioned earlier, the single-host mode ignores the MAC address seen in the voice VLAN so this should work. It does work, however, once you shut the port down and enable it again, and phone or switch reboots, the switch port will see two MAC addresses on the data VLAN.

Now, you’re probably wondering why would the switch see two MAC addresses in the data VLAN when the IP Phone should only show up in the voice VLAN especially when the boot process is described in books like this. But, I’ve seen this happened in all three organizations I’ve worked for where the phone’s MAC address shows up in both data and voice VLAN, as shown below. If you do a quick search, you’ll see more people are seeing the same thing so it appears that this is the default behavior.

switch#sh mac add int g1/0/1
          Mac Address Table
-------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    0004.f2f0.4d98    DYNAMIC     Gi1/0/1
  20    0004.f2f0.4d98    DYNAMIC     Gi1/0/1
Total Mac Addresses for this criterion: 2

As you can imagine, this could turn to an operational nightmare especially when you have facilities people going in and out of the closet to do some work and they occasionally bump into the power cord of the switch by accident. The solution is to change the host mode to something that will not cause a security violation. One option is to use the Multi-domain authentication (MDA), which is shown below.

interface range g1/0/1- 48
 authentication host-mode multi-domain

MDA vs Multi-Auth

Multi-domain authentication (MDA) allows one MAC address on both data and voice VLAN. It is kind of similar with the single-host mode but this mode requires the device in the voice VLAN to authenticate. Initial testing looks like it’s working as expected. I didn’t see the same behavior where the phone’s MAC address shows up on both VLANs when I bounced the port.

MDA does not address the fact that the environment in our scenario will have users running VMware Fusion on their computer(s). When the user configures the VM with a network type of bridged mode, which means the switch will see two MAC addresses, then that will result in a security violation. This needs to be addressed so there has to be another mode that we could use. Fortunately, there is and it is called multiple authentication.

interface range g1/0/1 - 48
 authentication host-mode multi-auth

The difference between MDA and multiple authentication is that it allows multiple MAC addresses in the data VLAN, however, all devices must be authenticated to access the network resources.

As mentioned, there is a way to automatically recover from a security violation, by default it is set to five minutes. Before I show you the command for it, let’s think about the fact that the port will be in the errdisabled state once a security violation occurs. That means, the phones will be out of commission too. This is going to be frustrating for the users so we need to find a solution that only errdisable the VLAN where the security violation occurred. Fortunately, the switch has the voice aware 802.1X security feature and is shown below with the errdisable recovery.

errdisable detect cause security-violation shutdown vlan
errdisable recovery cause security-violation

MAC Authentication Bypass

The devices that do not support 802.1X feature still needs to access network resources so we need to find a way to let them in without disabling the port-based authentication where these devices are connected to. Cisco supports fallback mechanisms when a device fails to authenticate using 802.1X. A great option for devices that do not support 802.1X is the MAC Authentication Bypass (MAB).

With MAB, the MAC address is entered to the RADIUS server and when the device fails to authenticate using the 802.1X then the switch will fallback to MAB. The switch will then forward a message, with the MAC address of the device, to the RADIUS server. RADIUS server will then check its database to see if the MAC address is in its list. If it is, then the RADIUS server will signal the switch to allow access to the network. To enable MAB, issue the command below.

interface range g1/0/1 - 48
 mab

Another version of this command is shown below. If this command is used, the IOS will change it to mab in the running and startup config.

interface range g1/0/1 - 48
 dot1x mac-auth-bypass

While this fallback mechanism works, Cisco Catalyst switches have default values which delays the transition of a non-802.1X compliant from unauthorized to authenticated for 90 seconds. This might cause some issues with DHCP or PXE clients so it is recommended to tweak the default values to make it faster for the non-802.1X compliant devices to access network resources.

The 90 seconds is the combination of the dot1x max-reauth-req and dot1x timeout tx-period values. The default value for the former is two and the latter is 30 seconds. Multiply both values and the result is 60 seconds. You’re probably thinking where’s the other 30 out of the 90 seconds? Well, that was the initial request for the device to authenticate and when it fails the switch will then send a request. It would keep sending up to the configured max-reauth-req values when there’s no response from the device. It is recommended to test what’s best for your network since there are really no recommended values. For our scenario, let’s configure them with a value of one and 10 seconds.

interface range g1/0/1 - 48
 dot1x max-reauth-req 1
 dot1x timeout tx-period timer 10

The last thing that we need to address is the WoL feature that some people use in the environment. By default, traffic through the unauthorized port is blocked in both directions and the magic packet, WoL packet sent by the server, never gets to the sleeping computer.

To support the WoL feature in 802.1X environment, we’ll need to configure the switch to allow outbound traffic to the unauthorized port but still control the incoming traffic. The command to do this is shown below.

interface range g1/0/1 - 48
 authentication control-direction in

Other considerations

Not every scenario is covered here so I recommend you to read Cisco’s configuration and deployment guide about 802.1X. For example, what if all RADIUS servers that handles the port-based authentication are unreachable? That would mean, unauthorized ports trying to move to authenticated ports will not work. Configuring critical VLAN both for data and voice may be necessary for this environment.

For partners’ devices, how would you like to handle their access to network resources? Would you allow them by implementing a Guest VLAN feature?

If you opt for using EAP-TLS, how would you manage the deployment of the certificates to all devices including mobile? This might frustrate users and may also overwhelm the desktop support staff if not handled properly.

What if your organization use non-Cisco phones? What will happen to the devices behind the phones once it gets authenticated and gets removed from the port? Does it support EAPoL Logoff/Proxy EAPoL Logoff? This is not an issue with Cisco phones with CDP since it supports CDP Enhancement for Second Port Disconnect. With this feature, when the user disconnects from the phone’s port, the phone will signal the Catalyst switch to move the data VLAN from authenticated to unauthorized state.

How do you want to authenticate the phones? Do you want to use EAP-MD5, MIC (Manufacturer Installed Certificate), or LSC (Locally Significant Certificate)?

If you do allow MAB fallback mechanism, how do you combat the possibility of unauthorized users spoofing MAC addresses that are in your RADIUS’s MAC address database? If the organization is big enough, how do you manage adding MAC addresses to the database? How do you maintain the database properly without leaving temporary entries?

Thoughts

Deploying 802.1X definitely has its challenges. This could be the reason why some organizations choose to not have some type of port-based authentication because of it may affect the availability of network resources. When it comes to deployment, I believe proper planning and testing is needed to make it a smooth deployment. Few things that could be used to make it a smooth deployment are the following: monitor mode, low impact mode, and closed mode, which is covered in this Cisco Live! presentation. Some might just opt for the lab testing then move to pilot phase, which is doable in my opinion.

References

CCNP SWITCH
Wired 802.1X Deployment Guide
Catalyst 2960X Configuration Guide

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

TACACS+ (tac_plus daemon) ACL

By 15 Comments

I covered how to install and configure TACACS+ (tac_plus) on Ubuntu here. In this blog post, I am going to cover how to deny a group in accessing a network device or group of network devices. Yes, you can certainly restrict IPs using ACL on routers, switches, or ASA firewalls. However, if your security policy is to use a jump server to connect to network devices, then that jump server’s IP will not be blocked and users are still allowed. With this guide, you’ll be able to restrict access to network devices by user or group using your tac_plus server.

Configuration

This guide assumes you know how to configure TACACS+ (tac_plus) daemon. If you haven’t had a chance to work on this then please feel free to visit it here. Without further ado, here’s the configuration on how to restrict a group of users on a particular network device.

group = Test {
        default service = deny
        acl = test_acl
        service = exec {
        priv-lvl = 2
        }
}
acl = test_acl {
        deny = 192.168.12.100
        permit = .*
}

The above configuration restricts the group called Test in accessing a network device with an IP address of 192.168.12.100. The IP address that is needed in the tac_plus.conf needs to be the IP address of the source interface (ip tacacs source-interface interface_type_here) that you configured on your network device. If the TACACS+ source interface IP address doesn’t match the one in the tac_plus.conf, then any other IP address of that network device can be used to connect and the group will be allowed to access it.

Below is an example of a user that is member of the Test group trying to access the network device and was denied.

networkjutsu@tacacs:~$ ssh -l test 192.168.12.100
test@192.168.12.100's password:
Permission denied, please try again.
test@192.168.12.100's password:
Permission denied, please try again.
test@192.168.12.100's password:
Permission denied (password).

The example config below is how to restrict group(s) using regex (regular expressions). If you’re not a scripter/programmer like me then please feel free to use cheat sheet found here. For more detailed info, regex is covered in CCIE Routing and Switching Certification Guide book by Wendell Odom.

group = Test {
        default service = deny
        acl = test_acl
        service = exec {
        priv-lvl = 2
        }
}
acl = test_acl {
        deny = 172.17.99.[0-9]
        deny = 172.24.64\.50$
        permit = .*
}

Here’s an attempt of a member of Test group being denied.

networkjutsu@tacacs1:~$ ssh -l test 172.17.99.50
Password:
Password:
Password:
test@172.17.99.50's password:
Permission denied, please try again.
test@172.17.99.50's password:
Permission denied, please try again.
test@172.17.99.50's password:
Permission denied (keyboard-interactive,password).
networkjutsu@tacacs1:~$ ssh -l test 172.17.99.17
test@172.17.99.17's password:
Permission denied, please try again.
test@172.17.99.17's password:
Permission denied, please try again.
test@172.17.99.17's password:
Permission denied (password).
networkjutsu@tacacs1:~$ ssh -l test 172.17.99.3
test@172.17.99.3's password:
Permission denied, please try again.
test@172.17.99.3's password:
Permission denied, please try again.
test@172.17.99.3's password:
Permission denied (password).
networkjutsu@tacacs1:~$ ssh -l test 172.24.64.50
Password:
Password:
Password:
test@172.24.64.50's password:
Connection closed by 172.24.64.50

If you noticed, the first attempt (172.17.99.50) had more prompts than others. This might be a bug on an IOS, due to the device type, or just how this particular IOS/device behaves with the tac_plus. Unfortunately, I didn’t dig deeper enough. In any case, tac_plus denied access to the device.

With this config, it allows an organization to give out privilege access to certain users or groups but still be able to deny access to certain devices. For example, an organization may allow the NOC employees to make changes on access layer routers/switches but are not allowed to connect to distribution and core routers/switches and ASA firewalls.

Hope this has been helpful and thank you for reading!

You might also like to read

How to build and configure tac_plus
Adding two-factor authentication to TACACS+

Reference

TACACS+ daemon

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.