NetworkJutsu

Network Security Consulting | San Francisco Bay Area

Hardening

Securing SSH with Google Authenticator

By 2 Comments

In my old blog post, I talked about how to mitigate from persistent SSH brute force attack. While there are several options in mitigating SSH brute force attack, I opted to use the Fail2Ban option at the time. Today, I’ve decided to add another security layer to the host since this is a public facing server. This addition of security layer is based on defense in depth, which is an information assurance concept. As the title says, I will be using Google Authenticator to generate a time-based one-time password (TOTP) for two-step verification.

Related: What is multi-factor authentication?

It seems like two-factor authentication (2FA) is becoming a norm these days. More and more security professionals are pushing organizations to use 2FA for every sensitive systems and application. Understandably so, because the consensus is that password is no longer enough to protect accounts in this day and age. As a result, I’ve also decided to start implementing 2FA in my home devices.

Installing Google Authenticator PAM

Ubuntu has been my distro of choice for several years now, so all of my Linux-related tutorials have been on that. I will keep that going, mostly because I do not have time to learn another distro, like CentOS.

Installing Google Authenticator on Ubuntu 16.04 is a piece of cake. All you need is one command.

$ sudo apt-get install libpam-google-authenticator -y

Configuring SSH PAM

The Google Authenticator 2FA is accomplished by integrating into Linux’s Pluggable Authentication Modules (PAM) library. PAM is a way for programs to use an underlying authentication mechanism. With that said, we’ll need to configure PAM configuration to pass it to Google Authenticator. To do this, we need to edit the PAM configuration file for SSH.

$ sudo vi /etc/pam.d/sshd

At the bottom of the file, I added the following lines below. The first line is optional, but it’s always best to add comment lines in my opinion.

# Enable MFA using Google Authenticator PAM
auth required pam_google_authenticator.so nullok

If you want to understand the syntax, please check this site. Though, the article did not include what nullok argument means. The argument simply means that if a user hasn’t created secret key yet, then they’re still allowed to log in. My recommendation is to enable 2FA for all users so make sure to have all the users generate the secret key. Once all secret keys are generated, take out the nullok argument.

Configuring OpenSSH Daemon

The next step is to actually configure SSH to check for backend system (e.g. PAM) to use the challenge-response authentication method. To do this, we need to edit the OpenSSH configuration file.

$ sudo vi /etc/ssh/sshd_config

Once the file is open, look for the line with ChallengeResponseAuthentication no and change it to yes. Save the file and exit.

ChallengeResponseAuthentication yes

In Linux, changes to Linux configuration files require a service restart to take effect, for the most part. With that said, we need to restart SSH daemon (sshd). To restart SSH service, issue sudo service ssh restart command. Once sshd is back up, test to make sure that user can still log in without two-step verification.

Generating Google Authenticator Secret Key

The last step to make SSH 2FA work is to generate a secret key for the two-step verification. In this example, I created a test user account for demo purposes.

generating-secret-key

Use the Google Authenticator app on your mobile device and add the QR code.

Account with Google Authenticator secret key

In this example, SSH daemon is asking the user to enter Google Authenticator OTP.

$ ssh test@radius
Password: testing1234
Verification code: 664449
<-- Output omitted for brevity -->
test@radius:~$
google authenticator token

Account with no Google Authenticator secret key

Here the admin account does not have the Google Authenticator secret key yet. If one forgets to add the nullok argument, then the system will not allow user accounts without the secret key.

$ ssh admin@radius
Password:
<-- Output omitted for brevity -->
admin@radius:~$

Optional Configuration

It is probably a good idea to enable 2FA to the local login (console) too. So this way, anyone who has access to the physical machine will be subjected to two-step verification. To do this, we need to edit the login configuration file.

$ sudo vi /etc/pam.d/login

Once the file is open, add the lines below at the end of the file. Save the file and exit out.

# Enable MFA using Google Authenticator PAM
auth required pam_google_authenticator.so nullok

From now on, all user accounts with Google Authenticator secret key will need to enter a verification code.

Thoughts

Balancing security and convenience is one of the biggest dilemmas that information security professionals face. In a nutshell, making it convenient for users to access their account means it is less secure. In a perfect world (no bad guys) people would probably pick one password across all of their accounts. Since we’re not living in a perfect world, this leaves the accounts to be very insecure. If one account is compromised, then all of the other accounts could potentially be accessed by the bad guys.

The use of complex passwords and password managers are good first steps toward securing accounts. However, in today’s world, some of the information security professionals view password alone as an antiquated technique in securing accounts. As a result, a lot of online services have had multi-factor authentication feature. However, companies that provide these online services do not force accounts to use it. As a result, accounts are still at risk by relying on passwords alone.

You might also like to read

Adding Two-Factor Authentication to FreeRADIUS

Reference

How to Secure SSH with Google Authenticator’s Two-Factor Authentication

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

SSH Brute Force Attack

By 2 Comments

Almost every day, I log in to my Ubuntu Server edition distro and one day I noticed something odd with the disk space. I noticed that it shot up from around 50% to 80% of 6.5GB. I’ve had this server since Hardy Heron days (8.04), it is now on Trusty Thar (14.04.3) Bionic Beaver (18.04.1), and never once it consumed more than 60% of the allocated disk space. That said, I was curious why all of a sudden the disk space became so big when I did not even install new packages.

The df -ah command was not very helpful since it only listed that root directory was consuming ~80%. I needed to figure out which directory was actually consuming those ~30% of disk space. After doing a web search, I found a command that allowed me to discover an important log that I should have known from the beginning.

$ sudo du -a / | sort -n -r | head -n 10
3571449	/
2589064	/var
2065580	/var/log
1762760	/var/log/btmp.1
613604	/usr
304008	/lib
274824	/var/cache
239600	/var/lib

Brute Force Attack Discovery

This was when I discovered the /var/log/auth.log file. Yes, I am still a Linux newbie. Every authentication attempt is listed here and also their IP address. That’s when I saw a bunch of SSH connections from different IP addresses I do not recognize and different usernames that the system does not even have or has been disabled. Most of the username I’ve seen are ubnt (Ubiquiti’s username on a lot of their products), pi, admin, etc. There was one IP address that has been brute forcing my box for a month without me knowing! Below is a snippet of my compressed auth.log file.

$ zcat /var/log/auth.log.4.gz | grep -v CRON | tail -n 500
Dec 29 06:43:17 ubuntu sshd[17503]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54 user=root
Dec 29 06:43:18 ubuntu sshd[17501]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 59428 ssh2]
Dec 29 06:43:18 ubuntu sshd[17501]: Received disconnect from 43.229.53.54: 11:  [preauth]
Dec 29 06:43:18 ubuntu sshd[17501]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:18 ubuntu sshd[17521]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:18 ubuntu sshd[17523]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:20 ubuntu sshd[17509]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 11637 ssh2]
Dec 29 06:43:20 ubuntu sshd[17509]: Received disconnect from 43.229.53.54: 11:  [preauth]
Dec 29 06:43:20 ubuntu sshd[17509]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=43.229.53.54  user=root
Dec 29 06:43:20 ubuntu sshd[17521]: Failed password for root from 43.229.53.54 port 22024 ssh2
Dec 29 06:43:20 ubuntu sshd[17511]: message repeated 2 times: [ Failed password for root from 43.229.53.54 port 11819 ssh2]

Options

There are several options one can implement to mitigate SSH brute force attack. One option is to not allow passwords and just use SSH keys. This is not a good option for me because I want to use this server with any computer and without using any type of keys. That said, password-based authentication is what I need. Another option is to implement two-factor authentication (2FA), which I covered here.

Firewall Option

Initially, I decided to start blocking the IP addresses I’ve seen in auth.log using my PA-200. It worked for a while, but every day I see new IPs popping. I then decided to implement Geo-based IP rule to lower the amount of attack. While it lessens the attacks significantly, I still needed something to help with the attacks that still goes through the firewall.

Enter Fail2ban

This was suggested by a friend of mine, @guerilla7. Thanks, Ron! Fail2ban scans log files (eg. /var/log/auth.log) and bans IPs (using iptables) that show malicious signs – too many password failures, seeking for exploits, etc. By default, Fail2ban monitors the /var/log/auth.log only. Obviously, this can be configured so that it can monitor more log files.

Installation and Configuration

The installation of Fail2ban will vary depending on your distro. Since Ubuntu is Debian based distro, the package manager is apt-get. It is very simple to install in Ubuntu. The command below is how to install the software and its dependencies.

$ sudo apt-get install fail2ban

Once everything is installed, it is time to configure Fail2ban. But, before we edit the configuration file that Fail2ban uses, it is a good idea to use a different file for custom configurations since the original configuration file can be overwritten by updates. That said, we need to create a copy of the configuration file.

$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Once copied, we are now ready to configure the jail.local file.

$ sudo vi /etc/fail2ban/jail.local

The configuration file is around 480 lines (including comments), but don’t be afraid because only a few lines will be changed. Of course, that depends on one’s needs. For my needs, I only touched four lines and are listed below.

ignoreip = 127.0.0.1/8 192.168.1.0/24
bantime = -1
findtime = 31536000
maxretry = 3

The ignoreip is basically the whitelist.

The bantime sets the length of time that a client will be banned for failed authentication attempts. The negative value sets it forever. This feature was added to version 0.6.1 (03/2006).

The maxretry and findtime parameters work together in establishing the conditions under which a client is determined by an unauthorized user. By default, the findtime is set to 600 seconds (10 minutes) and maxretry to 3 attempts. This means that Fail2ban will ban an unauthorized user when it attempts to log in three times within 10-minute window. Above configuration is 365-day window. This should cover slow brute force attacks but the majority of the brute force attacks I have seen are within 2-second window.

The changes on the configuration file will not take effect until the restart so sudo /etc/init.d/fail2ban restart must be issued.

Optional Configurations

Feel free to skip these configurations if you don’t find them useful.

Persistent Ban

While the above changes are good enough, the bans are not persistent. Once the server or Fail2ban service was restarted, the banned IPs will not survive. Some will be fine with that configuration, but that is unacceptable for me. I want to be the one who will unban IP addresses not because the server or service was restarted.

Update: As of Fail2ban 0.9.x, the bans are now persistent, by default, after reboot or restart. It now maintains a database found in /var/lib/fail2ban/fail2ban.sqlite3 file. That said, for new installs, there is no need to modify the iptables-multiport.conf file. However, if you already have existing ip.blacklist file, then you may want to still modify the iptables-multiport.conf file.

If you want to verify, look at the fail2ban.conf file and look for the dbfile section. You’ll also see the dbpurgeage section that has a 1-day setting. I think this setting is ignored so long as the bantime is set to any negative number.

$ cat /etc/fail2ban/fail2ban.conf
<-- Output omitted for brevity -->
# Options: dbfile
# Notes.: Set the file for the fail2ban persistent data to be stored.
#         A value of ":memory:" means database is only stored in memory
#         and data is lost when fail2ban is stopped.
#         A value of "None" disables the database.
# Values: [ None :memory: FILE ] Default: /var/lib/fail2ban/fail2ban.sqlite3
dbfile = /var/lib/fail2ban/fail2ban.sqlite3
# Options: dbpurgeage
# Notes.: Sets age at which bans should be purged from the database
# Values: [ SECONDS ] Default: 86400 (24hours)
dbpurgeage = 1d

First Step

The first step in making the ban persistent is to create a file where the list of banned IPs will be added. So that when the service gets restarted, for whatever reason, the software will load the file and issue the proper commands to re-ban the IPs. To create the file, issue the sudo touch /etc/fail2ban/ip.blacklist command. This will create a blank file called ip.blacklist. Feel free to call it different than mine but make sure to use the same file name on the configuration on the next step.

Second Step

The second step is to verify that we are actually going to edit the right configuration file. This is done by viewing the /etc/fail2ban/jail.local file and look for the first banaction = iptables-multiport configuration. At this time of writing, Fail2ban 0.8.11-1 on Ubuntu 14.04.3 is using iptables-multiport, which points to the iptables-multiport.conf file. It’s always a good idea to save a backup configuration, so issue the sudo cp /etc/fail2ban/action.d/iptables-multiport.conf /etc/fail2ban/action.d/iptables-multiport.conf.bak command.

Once completed, we can now move on to the final step. Edit the configuration iptables-multiport.conf file. To edit the iptables-multiport configuration file, issue the sudo vi /etc/fail2ban/action.d/iptables-multiport.conf command. The configuration file has around 70 configuration lines including the comments. There are only two configuration sections that we need to edit. These two are the actionstart and actionban section. The configuration will look as below.

actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> -m multiport --dports <port> -j fail2ban-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I fail2ban-<name> 1 -s $IP -j DROP; done; fi
actionban = if ! iptables -C fail2ban-<name> -s <ip> -j DROP; then iptables -I fail2ban-<name> 1 -s <ip> -j DROP; fi
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Update: Jim commented that the Fail2ban 0.9.3 on Ubuntu 16.04 changed from fail2ban to f2b. Thanks for the comment Jim!

actionstart = iptables -N f2b-<name>
              iptables -A f2b-<name> -j RETURN
              iptables -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done; fi
actionban = if ! iptables -C f2b-<name> -s <ip> -j DROP; then iptables -I f2b-<name> 1 -s <ip> -j DROP; fi
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Update: With Fail2ban 0.10.2 on Ubuntu 18.04, the default config was slightly changed. However, the line that we added on previous versions remains the same.

actionstart = <iptables> -N f2b-<name>
              <iptables> -A f2b-<name> -j <returntype>
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j f2b-<name>
              # This configuration loads the ip.blacklist file every time Fail2ban service is started.
              if [ -f /etc/fail2ban/ip.blacklist ]; then cat /etc/fail2ban/ip.blacklist | grep -e <name>$ | cut -d "," -s -f 1 | while read IP; do iptables -I f2b-<name> 1 -s $IP -j DROP; done; fi
actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>
            # Add offenders to ip.blacklist file, if it is not already there yet.
            if ! grep -Fxq '<ip>,<name>' /etc/fail2ban/ip.blacklist; then echo '<ip>,<name>' >> /etc/fail2ban/ip.blacklist; fi

Since we’ve made a modification to a configuration file, we need to restart the service by issuing sudo /etc/init.d/fail2ban restart or sudo service fail2ban restart command.

Note: I haven’t figured out why the IPs in the ip.blacklist file do not load after reboot or service restart. It will only add the list once a new failed SSH attempt has been made.

DROP vs REJECT

At the time this post was written, fail2ban used DROP as the default block type. Now, they changed the behavior to REJECT with an ICMP message of unreachable.

The biggest difference between the two is that DROP won’t send anything, while REJECT will send a message back to the source.

If you want to change the block type to DROP, then edit the /etc/fail2ban/action.d/iptables-common.conf file. The configuration below shows that I commented out the default behavior and changed it to DROP instead.

$ sudo more /etc/fail2ban/action.d/iptables-common.conf | grep "blocktype = "
# blocktype = REJECT --reject-with icmp-port-unreachable
blocktype = DROP
# blocktype = REJECT --reject-with icmp6-port-unreachable
blocktype = DROP

Verification

Depending on how often the attack occurs, check the iptables after several hours or a day. To check the iptables, issue the command below.

$ sudo iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
fail2ban-ssh  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 22
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Chain fail2ban-ssh (1 references)
target     prot opt source               destination
DROP       all  --  43.229.53.54         0.0.0.0/0
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

Manually unbanning an IP address

To unban an IP address, issue the command below.

$ sudo fail2ban-client set <jail_name> unbanip <ip_address>
# Example using Fail2ban 0.8.11
$ sudo fail2ban-client set ssh unbanip 43.229.53.54
43.229.53.54
$ sudo fail2ban-client reload
# Example using Fail2ban 0.9.3 and 0.10.2
$ sudo fail2ban-client set sshd unbanip 43.229.53.54
43.229.53.54
$ sudo fail2ban-client reload

While the command above is enough without the optional configuration (discussed above), this command is not the only thing needed with the optional configuration since the ip.blacklist file still contains the IP address that we’re trying to unban. If the server or service was restarted, then the IP address will be banned again. That said, it is necessary to take it out from the ip.blacklist file. To do this, issue the command below.

$ sudo sed --in-place '/<ip>,<name>/d' /etc/fail2ban/ip.blacklist
# Example
$ sudo sed --in-place '/43.229.53.54,ssh/d' /etc/fail2ban/ip.blacklist
$ sudo fail2ban-client reload

When unbanning fails

When someone issues the sudo fail2ban-client reload command then there is a very high chance that the user will encounter an error message similar to the one below.

$ sudo fail2ban-client set ssh unbanip 58.218.211.198
ERROR  NOK: ('IP 58.218.211.198 is not banned',)
IP 58.218.211.198 is not banned
# Example using Fail2ban 0.10.2
$ sudo fail2ban-client set sshd unbanip 58.218.211.198
ERROR  NOK: ('IP 58.218.211.198 is not banned',)
IP 58.218.211.198 is not banned

The IP can be still unbanned by the following:

$ sudo sed --in-place '/58.218.211.198,ssh/d' /etc/fail2ban/ip.blacklist
$ sudo fail2ban-client reload
$ sudo iptables -L -n | grep 58.218.211.198

Let’s say deleting the IP address from the blacklist file and reloading fail2ban didn’t work like what I experienced recently. The IP address that I was trying to unban kept coming back. I had to find another way to unban it using the iptables command. Here’s what I did to unban the IP address.

$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
655    DROP       all  --  58.218.211.198       0.0.0.0/0
$ sudo iptables -D fail2ban-ssh 655
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
$ sudo fail2ban-client reload
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
# Example using 0.10.2
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
655    DROP       all  --  58.218.211.198       0.0.0.0/0
$ sudo iptables -D f2b-sshd 655
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198
$ sudo fail2ban-client reload
$ sudo iptables -L -n --line-numbers | grep 58.218.211.198

Though, this seems to be a very rare occasion since I tried unbanning another IP address using the method in the manually unbanning section and it worked just fine.

Thoughts

This is exactly the software I was looking for. It is automated which means I no longer need to check auth.log and block it on my Ubiquiti EdgeRouter Lite. I did transfer the rules from my PA-200 to my new router/firewall, however. Though, I am still getting used to the creation of firewall rules because it is not as intuitive as creating rules on Cisco ASA or Palo Alto Networks firewall. While this software automatically blocks failed attempts, it does not protect from weak passwords. It is still recommended to use strong passwords.

UPDATE: I no longer use the EdgeRouter to protect my DMZ – I now use pfSense. The PA-200 will eventually be used on some user traffic but will be limited. Mostly, to learn more about how to configure it.

Want to learn more about Linux System Administration?

UNIX and Linux System Administration Handbook, 4th Edition

References

List based permanent bans with fail2ban
How To Protect SSH with fail2ban on Debian 7
How to unban an IP properly with fail2ban

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

Implementing Wired 802.1X

By 2 Comments

I first learned about 802.1X when I was studying for one of the CCNP exams, BCMSN exam (SWITCH equivalent), at Ohlone College. At the time, I assumed that the short material covered in the book was all of it. Of course, that was a bad assumption in my part. That’s probably a normal assumption of someone who at the time just finished Cisco Network Academy Program CCNA 1 to 4 and newly minted CCNA with no professional experience.

What is 802.1X?

Essentially, 802.1X is a security feature that provides a mechanism to authenticate devices before it can access network resources. While it’s a good idea to have this security feature implemented, I’ve worked for companies who didn’t have this feature or similar implemented or it’s on their roadmap. It’s a shame that it wasn’t on their roadmap a long time ago since it was ratified in 2001. Then again, implementing technologies have its challenges.

How it works?

802.1X_wired_protocols
Image from Wikipedia

While there are other sources that will explain this in detail, this post includes a very short description on how it works. Basically, when a device connects to the wired network, the authenticator (switch) will send an EAP message to the supplicant (computer). If the computer has a supplicant, it will send an EAP response to the authenticator. The authenticator will then send a RADIUS message to the authentication server (RADIUS server). The authentication server will then challenge the supplicant to verify its identity. Once verified, the device will then be able to connect to organization’s network resources.

Environment

Every organization has their own unique implementation of technologies, so gather what you can and go from there. For example, in this scenario the requirements were to have two sets of RADIUS servers: one for switch-based authentication and the other for port-based authentication. This seems to be an uncommon setup so it required some research to split the two sets of RADIUS servers. My initial assumption was that it wasn’t possible. That assumption is only correct in older code, but with IOS 15.x the feature is supported.

This is a multivendor organization so LLDP is used instead of CDP, which is disabled globally by default due to the switch template configuration.

A lot of users are using Apple notebooks and/or desktops and most of these users are running VMware Fusion to run Windows and/or Linux.

IP phones are ubiquitous so this requires a great deal of attention. If my memory serves me right, the 802.1X topic in BCMSN didn’t cover how to implement it with IP phones so Cisco’s documentation and Google were my friend during my research.

A lot of devices do not have supplicant and there are instances where PXE boot is needed.

In addition, there were some locations that need WoL (Wake on LAN) feature so that needs an attention as well.

Configuration

As mentioned earlier, the requirement is to have two separate RADIUS servers for both switch-based and port-based authentication. That said, let’s take a look on how to do this. But first, let me show you how it was done prior to IOS 15.x code. This command still works in 15.0(2), but you’ll receive a warning saying that it will soon be deprecated.

Old format

radius-server host 192.168.1.1 auth-port 1812 acct-port 1813
radius-server host 192.168.1.2 auth-port 1812 acct-port 1813
radius-server retransmit 1
radius-server timeout 2
radius-server key 7 hashkeyhere

Since the requirement is to split the RADIUS servers, we need to use the new format of specifying the RADIUS servers which will be needed when we create the AAA groups.

New format

radius server switch-auth1
 address ipv4 192.168.1.1 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server switch-auth2
 address ipv4 192.168.1.2 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server dot1x-auth1
 address ipv4 192.168.1.3 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server dot1x-auth2
 address ipv4 192.168.1.4 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere

Enable AAA

Once enabled, authentication method for 802.1X needs to be defined. I included the one for the switch-based authentication with the port-based authentication for completeness sake. RADIUS accounting is turned on as well since it is listed as best practice in Cisco’s deployment guide.

aaa new-model
aaa group server radius switch-auth
 server name switch-auth1
 server name switch-auth2
aaa group server radius dot1x-auth
 server name dot1x-auth1
 server name dot1x-auth2
aaa authentication login default group switch-auth enable
aaa authentication dot1x default group dot1x-auth
aaa accounting dot1x default start-stop group dot1x-auth
aaa authorization network default group dot1x-auth

Enable 802.1X

Issue the command below.

dot1x system-auth-control

Configure switch ports

Next step is to configure each switch port that will use 802.1X. This command will automatically include dot1x pae authenticator in the running configuration so don’t be alarmed if you see it there. This is to ensure that dot1x authentication still works on legacy configurations without manual intervention. NOTE: It seems to be that the IOS that I was using automatically included the dot1x pae authenticator command. That said, please make sure to add the command if you do not see it.

interface range g1/0/1 - 48
 ! Make sure that the ports should at least have switchport mode access or it won't take the commands.
 authentication port-control auto
 dot1x pae authenticator

In the IOS 12.x, this would’ve been a different command. The command in the old world is dot1x port-control auto.

Technically, the commands above are all we need to configure for the 802.1X to work. However, the environment in this scenario requires more things from us that we still need to address.

VoIP phones

Let’s address the IP phones first since it’s ubiquitous within the enterprise environment. By default, the interfaces are set to be single-host mode. This means only one MAC is allowed in the data VLAN. This mode technically allows another MAC address but on the voice VLAN and only if CDP is supported. Since CDP is disabled on all of the switches deployed in this scenario, this needs to be enabled. I included the single-host mode command below since it won’t show up in the running configuration because it is the default configuration.

cdp run
interface g1/0/1 - 48
 authentication host-mode single-host

While this configuration works, there are few things that we need to keep in mind. The single-host mode means only single MAC can be authenticated on a switch port. If a different MAC address is detected on a port after an endpoint has authenticated then a security violation is triggered on the port. This will cause the port to be in errdisabled state and will require a manual intervention unless errdisable recovery is configured.

Since the computers are daisy chained on the back of the Cisco phones, there are technically two MAC addresses that will be seen on the port. As mentioned earlier, the single-host mode ignores the MAC address seen in the voice VLAN so this should work. It does work, however, once you shut the port down and enable it again, and phone or switch reboots, the switch port will see two MAC addresses on the data VLAN.

Now, you’re probably wondering why would the switch see two MAC addresses in the data VLAN when the IP Phone should only show up in the voice VLAN especially when the boot process is described in books like this. But, I’ve seen this happened in all three organizations I’ve worked for where the phone’s MAC address shows up in both data and voice VLAN, as shown below. If you do a quick search, you’ll see more people are seeing the same thing so it appears that this is the default behavior.

switch#sh mac add int g1/0/1
          Mac Address Table
-------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    0004.f2f0.4d98    DYNAMIC     Gi1/0/1
  20    0004.f2f0.4d98    DYNAMIC     Gi1/0/1
Total Mac Addresses for this criterion: 2

As you can imagine, this could turn to an operational nightmare especially when you have facilities people going in and out of the closet to do some work and they occasionally bump into the power cord of the switch by accident. The solution is to change the host mode to something that will not cause a security violation. One option is to use the Multi-domain authentication (MDA), which is shown below.

interface range g1/0/1- 48
 authentication host-mode multi-domain

MDA vs Multi-Auth

Multi-domain authentication (MDA) allows one MAC address on both data and voice VLAN. It is kind of similar with the single-host mode but this mode requires the device in the voice VLAN to authenticate. Initial testing looks like it’s working as expected. I didn’t see the same behavior where the phone’s MAC address shows up on both VLANs when I bounced the port.

MDA does not address the fact that the environment in our scenario will have users running VMware Fusion on their computer(s). When the user configures the VM with a network type of bridged mode, which means the switch will see two MAC addresses, then that will result in a security violation. This needs to be addressed so there has to be another mode that we could use. Fortunately, there is and it is called multiple authentication.

interface range g1/0/1 - 48
 authentication host-mode multi-auth

The difference between MDA and multiple authentication is that it allows multiple MAC addresses in the data VLAN, however, all devices must be authenticated to access the network resources.

As mentioned, there is a way to automatically recover from a security violation, by default it is set to five minutes. Before I show you the command for it, let’s think about the fact that the port will be in the errdisabled state once a security violation occurs. That means, the phones will be out of commission too. This is going to be frustrating for the users so we need to find a solution that only errdisable the VLAN where the security violation occurred. Fortunately, the switch has the voice aware 802.1X security feature and is shown below with the errdisable recovery.

errdisable detect cause security-violation shutdown vlan
errdisable recovery cause security-violation

MAC Authentication Bypass

The devices that do not support 802.1X feature still needs to access network resources so we need to find a way to let them in without disabling the port-based authentication where these devices are connected to. Cisco supports fallback mechanisms when a device fails to authenticate using 802.1X. A great option for devices that do not support 802.1X is the MAC Authentication Bypass (MAB).

With MAB, the MAC address is entered to the RADIUS server and when the device fails to authenticate using the 802.1X then the switch will fallback to MAB. The switch will then forward a message, with the MAC address of the device, to the RADIUS server. RADIUS server will then check its database to see if the MAC address is in its list. If it is, then the RADIUS server will signal the switch to allow access to the network. To enable MAB, issue the command below.

interface range g1/0/1 - 48
 mab

Another version of this command is shown below. If this command is used, the IOS will change it to mab in the running and startup config.

interface range g1/0/1 - 48
 dot1x mac-auth-bypass

While this fallback mechanism works, Cisco Catalyst switches have default values which delays the transition of a non-802.1X compliant from unauthorized to authenticated for 90 seconds. This might cause some issues with DHCP or PXE clients so it is recommended to tweak the default values to make it faster for the non-802.1X compliant devices to access network resources.

The 90 seconds is the combination of the dot1x max-reauth-req and dot1x timeout tx-period values. The default value for the former is two and the latter is 30 seconds. Multiply both values and the result is 60 seconds. You’re probably thinking where’s the other 30 out of the 90 seconds? Well, that was the initial request for the device to authenticate and when it fails the switch will then send a request. It would keep sending up to the configured max-reauth-req values when there’s no response from the device. It is recommended to test what’s best for your network since there are really no recommended values. For our scenario, let’s configure them with a value of one and 10 seconds.

interface range g1/0/1 - 48
 dot1x max-reauth-req 1
 dot1x timeout tx-period timer 10

The last thing that we need to address is the WoL feature that some people use in the environment. By default, traffic through the unauthorized port is blocked in both directions and the magic packet, WoL packet sent by the server, never gets to the sleeping computer.

To support the WoL feature in 802.1X environment, we’ll need to configure the switch to allow outbound traffic to the unauthorized port but still control the incoming traffic. The command to do this is shown below.

interface range g1/0/1 - 48
 authentication control-direction in

Other considerations

Not every scenario is covered here so I recommend you to read Cisco’s configuration and deployment guide about 802.1X. For example, what if all RADIUS servers that handles the port-based authentication are unreachable? That would mean, unauthorized ports trying to move to authenticated ports will not work. Configuring critical VLAN both for data and voice may be necessary for this environment.

For partners’ devices, how would you like to handle their access to network resources? Would you allow them by implementing a Guest VLAN feature?

If you opt for using EAP-TLS, how would you manage the deployment of the certificates to all devices including mobile? This might frustrate users and may also overwhelm the desktop support staff if not handled properly.

What if your organization use non-Cisco phones? What will happen to the devices behind the phones once it gets authenticated and gets removed from the port? Does it support EAPoL Logoff/Proxy EAPoL Logoff? This is not an issue with Cisco phones with CDP since it supports CDP Enhancement for Second Port Disconnect. With this feature, when the user disconnects from the phone’s port, the phone will signal the Catalyst switch to move the data VLAN from authenticated to unauthorized state.

How do you want to authenticate the phones? Do you want to use EAP-MD5, MIC (Manufacturer Installed Certificate), or LSC (Locally Significant Certificate)?

If you do allow MAB fallback mechanism, how do you combat the possibility of unauthorized users spoofing MAC addresses that are in your RADIUS’s MAC address database? If the organization is big enough, how do you manage adding MAC addresses to the database? How do you maintain the database properly without leaving temporary entries?

Thoughts

Deploying 802.1X definitely has its challenges. This could be the reason why some organizations choose to not have some type of port-based authentication because of it may affect the availability of network resources. When it comes to deployment, I believe proper planning and testing is needed to make it a smooth deployment. Few things that could be used to make it a smooth deployment are the following: monitor mode, low impact mode, and closed mode, which is covered in this Cisco Live! presentation. Some might just opt for the lab testing then move to pilot phase, which is doable in my opinion.

References

CCNP SWITCH
Wired 802.1X Deployment Guide
Catalyst 2960X Configuration Guide

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.