• Skip to main content
  • Skip to footer

NetworkJutsu

Network Security Consulting | San Francisco Bay Area

  • Blog
  • Services
  • Testimonials
  • About
    • About Us
    • Terms of Use
    • Privacy Policy
  • Contact Us

Blog

How to configure Quanta T3040-LY3

07/17/2015 By Andrew Roderos Leave a Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

I was looking at Nexus 3064-T for one of the projects that I was involved in that required 10GBASE-T but while I was waiting for a demo unit I was tasked to play with the Quanta T3040-LY3. I’ve worked with them a little bit since some of our clients have them deployed and sometimes I get phone calls about helping them to get it working. Technically, I didn’t have to but I didn’t want to be the guy who doesn’t try help.

The Quanta’s syntax is almost identical to IOS which is not a surprise since there are vendors out there copy the CLI commands. Only one I’ve encountered so far that is completely different from IOS is Junos OS. Having said that, it was quite easy to convert my template to Quanta equivalent commands. However, I did have to read the docs when I had questions on how to do a specific command on their OS.

Configuration

The configuration shown here are pretty basic so if you’re looking at advanced stuff then you might want to move on to the next site. If you work with IOS, then there’s really no need to explain line by line since it’s pretty self-explanatory, for the most part. Here’s a sample configuration:

hostname networkjutsu-switch
!
no username guest
enable password passwordhere
username admin password passwordhere
!
ip domain-name networkjutsu.com
ip name-server 192.168.200.100
ip name-server 192.168.201.100
!
port-channel load-balance src-dst-ip all
vtp
vtp mode transparent
lldp med all
no cdp run all
!
vlan database
!Do you remember vlan database in Cisco IOS?
 vlan 99
 vlan name 99 MGMT_192.168.1.0/24
 vlan 10
 vlan name 10 DATA
 exit
!
interface vlan 1
 no ip address
 shutdown
interface vlan99
 ip address 192.168.1.100 255.255.255.0
 no shutdown
 no ip redirects
 no ip unreachables
!
serviceport protocol none
!This is to disable DHCP on the management port of the switch.
!To statically assign an IP address to the management port then
!use serviceport ip ipaddresshere subnetmaskhere gatewayhere command.
!
ip dhcp snooping vlan 1-4093
!
no ip dhcp snooping information option
no ip dhcp snooping verify mac-address
ip dhcp snooping
!
errdisable recovery cause bpdu
spanning-tree edgeport bpduguard
!
sflow rate 2000
sflow receiver 1 ip 192.168.100.200
sflow source-interface vlan 99
!
interface range 0/1 - 0/40
 no shutdown
 switchport access vlan 10
 switchport mode access
 storm-control broadcast
 spanning-tree edgeport
!
interface range 0/41 - 46
 no shutdown
 switchport tagging 1,10,99
 switchport allowed vlan add 10,99
 ip dhcp snooping trust
 exit
!
interface range 0/47 - 48
 no shutdown
 channel-group 1 mode active
interface port-channel 1
 no shutdown
 switchport tagging 1,10,99
 switchport allowed vlan add 10,99
 ip dhcp snooping trust
!
ip default-gateway 192.168.1.1
no ip http server
no ip http secure-server
!
logging traps debug
logging host 192.168.202.100 ipv4
!
line console
line vty 
 no sessions
line ssh
!
sntp clock timezone CA 8 0 after-utc
sntp server 172.16.100.50 ipv4
sntp server 172.16.100.60 ipv4
sntp source-interface vlan 99
!
aaa authentication login default radius enable
aaa authentication enable default enable
!
radius source-interface vlan 17
radius server timeout 2
radius server retransmit 1
radius server host auth 192.168.210.100 name radius01 port 1812
radius server host acct 192.168.210.100 name radius01 port 1813
radius server key auth 172.16.100.100 keyusedbytheradiusserverhere
!RADIUS key cannot be more than 16 characters.
radius server host auth 172.16.100.101 name radius02 port 1812
radius server host acct 172.16.100.101 name radius02 port 1813
radius server key auth 172.16.100.101 keyusedbytheradiusserverhere
!
exit
copy run start

Thoughts

As you can see, the configuration is almost identical as the IOS. There are some differences but for the most part almost identical. The OS does allow you to configure the switch via web, but it was quite painful to use. It’s still better to just use the CLI than the Web GUI.

The switch is pretty inexpensive compared to its competitors. However, there are few things that I didn’t quite like about it. First, the switch didn’t have QSFP. If I am not mistaken, at the time I played with this switch they weren’t selling a 10GBASE-T with QSFP uplinks in them like the Cisco Nexus 3064-T, 3172TQ, or Juniper QFX5100. Second, the RADIUS key was limited to 16 characters. Not such a big deal but it is quite a hassle to involve the RADIUS person/department to generate a shorter key for the switch. Third, I was using a demo unit and it had a trial OS license expired that didn’t warn me that it was expired and ports are unusable. It was quite annoying that the OS didn’t warn me about it instead of me wasting time in figuring out why the ports would automatically be in disabled state when I literary admin up the ports. Last, I was not able to configure SNMP ACL. I looked at the docs and I didn’t find a way to configure it. I believe this should come standard with the OS.

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

Filed Under: Switching Tagged With: Switch

Implementing Wired 802.1X

06/29/2015 By Andrew Roderos 2 Comments

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

I first learned about 802.1X when I was studying for one of the CCNP exams, BCMSN exam (SWITCH equivalent), at Ohlone College. At the time, I assumed that the short material covered in the book was all of it. Of course, that was a bad assumption in my part. That’s probably a normal assumption of someone who at the time just finished Cisco Network Academy Program CCNA 1 to 4 and newly minted CCNA with no professional experience.

What is 802.1X?

Essentially, 802.1X is a security feature that provides a mechanism to authenticate devices before it can access network resources. While it’s a good idea to have this security feature implemented, I’ve worked for companies who didn’t have this feature or similar implemented or it’s on their roadmap. It’s a shame that it wasn’t on their roadmap a long time ago since it was ratified in 2001. Then again, implementing technologies have its challenges.

How it works?

802.1X_wired_protocols
Image from Wikipedia

While there are other sources that will explain this in detail, this post includes a very short description on how it works. Basically, when a device connects to the wired network, the authenticator (switch) will send an EAP message to the supplicant (computer). If the computer has a supplicant, it will send an EAP response to the authenticator. The authenticator will then send a RADIUS message to the authentication server (RADIUS server). The authentication server will then challenge the supplicant to verify its identity. Once verified, the device will then be able to connect to organization’s network resources.

Environment

Every organization has their own unique implementation of technologies, so gather what you can and go from there. For example, in this scenario the requirements were to have two sets of RADIUS servers: one for switch-based authentication and the other for port-based authentication. This seems to be an uncommon setup so it required some research to split the two sets of RADIUS servers. My initial assumption was that it wasn’t possible. That assumption is only correct in older code, but with IOS 15.x the feature is supported.

This is a multivendor organization so LLDP is used instead of CDP, which is disabled globally by default due to the switch template configuration.

A lot of users are using Apple notebooks and/or desktops and most of these users are running VMware Fusion to run Windows and/or Linux.

IP phones are ubiquitous so this requires a great deal of attention. If my memory serves me right, the 802.1X topic in BCMSN didn’t cover how to implement it with IP phones so Cisco’s documentation and Google were my friend during my research.

A lot of devices do not have supplicant and there are instances where PXE boot is needed.

In addition, there were some locations that need WoL (Wake on LAN) feature so that needs an attention as well.

Configuration

As mentioned earlier, the requirement is to have two separate RADIUS servers for both switch-based and port-based authentication. That said, let’s take a look on how to do this. But first, let me show you how it was done prior to IOS 15.x code. This command still works in 15.0(2), but you’ll receive a warning saying that it will soon be deprecated.

Old format

radius-server host 192.168.1.1 auth-port 1812 acct-port 1813
radius-server host 192.168.1.2 auth-port 1812 acct-port 1813
radius-server retransmit 1
radius-server timeout 2
radius-server key 7 hashkeyhere

Since the requirement is to split the RADIUS servers, we need to use the new format of specifying the RADIUS servers which will be needed when we create the AAA groups.

New format

radius server switch-auth1
 address ipv4 192.168.1.1 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server switch-auth2
 address ipv4 192.168.1.2 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server dot1x-auth1
 address ipv4 192.168.1.3 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere
radius server dot1x-auth2
 address ipv4 192.168.1.4 auth-port 1812 acct-port 1813
 timeout 2
 retransmit 1
 key 7 hashkeyhere

Enable AAA

Once enabled, authentication method for 802.1X needs to be defined. I included the one for the switch-based authentication with the port-based authentication for completeness sake. RADIUS accounting is turned on as well since it is listed as best practice in Cisco’s deployment guide.

aaa new-model
aaa group server radius switch-auth
 server name switch-auth1
 server name switch-auth2
aaa group server radius dot1x-auth
 server name dot1x-auth1
 server name dot1x-auth2
aaa authentication login default group switch-auth enable
aaa authentication dot1x default group dot1x-auth
aaa accounting dot1x default start-stop group dot1x-auth
aaa authorization network default group dot1x-auth

Enable 802.1X

Issue the command below.

dot1x system-auth-control

Configure switch ports

Next step is to configure each switch port that will use 802.1X. This command will automatically include dot1x pae authenticator in the running configuration so don’t be alarmed if you see it there. This is to ensure that dot1x authentication still works on legacy configurations without manual intervention. NOTE: It seems to be that the IOS that I was using automatically included the dot1x pae authenticator command. That said, please make sure to add the command if you do not see it.

interface range g1/0/1 - 48
 ! Make sure that the ports should at least have switchport mode access or it won't take the commands.
 authentication port-control auto
 dot1x pae authenticator

In the IOS 12.x, this would’ve been a different command. The command in the old world is dot1x port-control auto.

Technically, the commands above are all we need to configure for the 802.1X to work. However, the environment in this scenario requires more things from us that we still need to address.

VoIP phones

Let’s address the IP phones first since it’s ubiquitous within the enterprise environment. By default, the interfaces are set to be single-host mode. This means only one MAC is allowed in the data VLAN. This mode technically allows another MAC address but on the voice VLAN and only if CDP is supported. Since CDP is disabled on all of the switches deployed in this scenario, this needs to be enabled. I included the single-host mode command below since it won’t show up in the running configuration because it is the default configuration.

cdp run
interface g1/0/1 - 48
 authentication host-mode single-host

While this configuration works, there are few things that we need to keep in mind. The single-host mode means only single MAC can be authenticated on a switch port. If a different MAC address is detected on a port after an endpoint has authenticated then a security violation is triggered on the port. This will cause the port to be in errdisabled state and will require a manual intervention unless errdisable recovery is configured.

Since the computers are daisy chained on the back of the Cisco phones, there are technically two MAC addresses that will be seen on the port. As mentioned earlier, the single-host mode ignores the MAC address seen in the voice VLAN so this should work. It does work, however, once you shut the port down and enable it again, and phone or switch reboots, the switch port will see two MAC addresses on the data VLAN.

Now, you’re probably wondering why would the switch see two MAC addresses in the data VLAN when the IP Phone should only show up in the voice VLAN especially when the boot process is described in books like this. But, I’ve seen this happened in all three organizations I’ve worked for where the phone’s MAC address shows up in both data and voice VLAN, as shown below. If you do a quick search, you’ll see more people are seeing the same thing so it appears that this is the default behavior.

switch#sh mac add int g1/0/1
          Mac Address Table
-------------------------------------------
Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    0004.f2f0.4d98    DYNAMIC     Gi1/0/1
  20    0004.f2f0.4d98    DYNAMIC     Gi1/0/1
Total Mac Addresses for this criterion: 2

As you can imagine, this could turn to an operational nightmare especially when you have facilities people going in and out of the closet to do some work and they occasionally bump into the power cord of the switch by accident. The solution is to change the host mode to something that will not cause a security violation. One option is to use the Multi-domain authentication (MDA), which is shown below.

interface range g1/0/1- 48
 authentication host-mode multi-domain

MDA vs Multi-Auth

Multi-domain authentication (MDA) allows one MAC address on both data and voice VLAN. It is kind of similar with the single-host mode but this mode requires the device in the voice VLAN to authenticate. Initial testing looks like it’s working as expected. I didn’t see the same behavior where the phone’s MAC address shows up on both VLANs when I bounced the port.

MDA does not address the fact that the environment in our scenario will have users running VMware Fusion on their computer(s). When the user configures the VM with a network type of bridged mode, which means the switch will see two MAC addresses, then that will result in a security violation. This needs to be addressed so there has to be another mode that we could use. Fortunately, there is and it is called multiple authentication.

interface range g1/0/1 - 48
 authentication host-mode multi-auth

The difference between MDA and multiple authentication is that it allows multiple MAC addresses in the data VLAN, however, all devices must be authenticated to access the network resources.

As mentioned, there is a way to automatically recover from a security violation, by default it is set to five minutes. Before I show you the command for it, let’s think about the fact that the port will be in the errdisabled state once a security violation occurs. That means, the phones will be out of commission too. This is going to be frustrating for the users so we need to find a solution that only errdisable the VLAN where the security violation occurred. Fortunately, the switch has the voice aware 802.1X security feature and is shown below with the errdisable recovery.

errdisable detect cause security-violation shutdown vlan
errdisable recovery cause security-violation

MAC Authentication Bypass

The devices that do not support 802.1X feature still needs to access network resources so we need to find a way to let them in without disabling the port-based authentication where these devices are connected to. Cisco supports fallback mechanisms when a device fails to authenticate using 802.1X. A great option for devices that do not support 802.1X is the MAC Authentication Bypass (MAB).

With MAB, the MAC address is entered to the RADIUS server and when the device fails to authenticate using the 802.1X then the switch will fallback to MAB. The switch will then forward a message, with the MAC address of the device, to the RADIUS server. RADIUS server will then check its database to see if the MAC address is in its list. If it is, then the RADIUS server will signal the switch to allow access to the network. To enable MAB, issue the command below.

interface range g1/0/1 - 48
 mab

Another version of this command is shown below. If this command is used, the IOS will change it to mab in the running and startup config.

interface range g1/0/1 - 48
 dot1x mac-auth-bypass

While this fallback mechanism works, Cisco Catalyst switches have default values which delays the transition of a non-802.1X compliant from unauthorized to authenticated for 90 seconds. This might cause some issues with DHCP or PXE clients so it is recommended to tweak the default values to make it faster for the non-802.1X compliant devices to access network resources.

The 90 seconds is the combination of the dot1x max-reauth-req and dot1x timeout tx-period values. The default value for the former is two and the latter is 30 seconds. Multiply both values and the result is 60 seconds. You’re probably thinking where’s the other 30 out of the 90 seconds? Well, that was the initial request for the device to authenticate and when it fails the switch will then send a request. It would keep sending up to the configured max-reauth-req values when there’s no response from the device. It is recommended to test what’s best for your network since there are really no recommended values. For our scenario, let’s configure them with a value of one and 10 seconds.

interface range g1/0/1 - 48
 dot1x max-reauth-req 1
 dot1x timeout tx-period timer 10

The last thing that we need to address is the WoL feature that some people use in the environment. By default, traffic through the unauthorized port is blocked in both directions and the magic packet, WoL packet sent by the server, never gets to the sleeping computer.

To support the WoL feature in 802.1X environment, we’ll need to configure the switch to allow outbound traffic to the unauthorized port but still control the incoming traffic. The command to do this is shown below.

interface range g1/0/1 - 48
 authentication control-direction in

Other considerations

Not every scenario is covered here so I recommend you to read Cisco’s configuration and deployment guide about 802.1X. For example, what if all RADIUS servers that handles the port-based authentication are unreachable? That would mean, unauthorized ports trying to move to authenticated ports will not work. Configuring critical VLAN both for data and voice may be necessary for this environment.

For partners’ devices, how would you like to handle their access to network resources? Would you allow them by implementing a Guest VLAN feature?

If you opt for using EAP-TLS, how would you manage the deployment of the certificates to all devices including mobile? This might frustrate users and may also overwhelm the desktop support staff if not handled properly.

What if your organization use non-Cisco phones? What will happen to the devices behind the phones once it gets authenticated and gets removed from the port? Does it support EAPoL Logoff/Proxy EAPoL Logoff? This is not an issue with Cisco phones with CDP since it supports CDP Enhancement for Second Port Disconnect. With this feature, when the user disconnects from the phone’s port, the phone will signal the Catalyst switch to move the data VLAN from authenticated to unauthorized state.

How do you want to authenticate the phones? Do you want to use EAP-MD5, MIC (Manufacturer Installed Certificate), or LSC (Locally Significant Certificate)?

If you do allow MAB fallback mechanism, how do you combat the possibility of unauthorized users spoofing MAC addresses that are in your RADIUS’s MAC address database? If the organization is big enough, how do you manage adding MAC addresses to the database? How do you maintain the database properly without leaving temporary entries?

Thoughts

Deploying 802.1X definitely has its challenges. This could be the reason why some organizations choose to not have some type of port-based authentication because of it may affect the availability of network resources. When it comes to deployment, I believe proper planning and testing is needed to make it a smooth deployment. Few things that could be used to make it a smooth deployment are the following: monitor mode, low impact mode, and closed mode, which is covered in this Cisco Live! presentation. Some might just opt for the lab testing then move to pilot phase, which is doable in my opinion.

References

CCNP SWITCH
Wired 802.1X Deployment Guide
Catalyst 2960X Configuration Guide

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Filed Under: Security, Switching Tagged With: Cisco, Hardening, IOS, Network Security, Switch

NetFlow-Lite on Catalyst 2960-X

04/19/2015 By Andrew Roderos Leave a Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Several months ago, sFlow became instrumental in figuring out the issue with HP switches that we inherited. Just to give you an idea of what the issue was, the HP switches would sporadically drop off the network but the user data traffic was still flowing. Good thing it was only the management traffic that was dropping and not user traffic. With the help of sFlow collector, I was able to correlate the timestamps of when several HP switches went down and I found out that MLD (Multicast Listener Discovery) was the culprit. Tried to search the web for some answers but no luck. I upgraded the code of the switches but still no luck. Finally, I decided to contact HP Tech Support since they offer a lifetime warranty on hardware and software. When the tech support asked for the config, he saw that igmp querier was turned on and when we turned it off the problem never came back. Since we’ve been replacing the HP switches with Cisco Catalyst switches, I wanted to replicate some level of the sFlow functionality. Luckily, the Catalyst 2960-X supports NetFlow-Lite.

What is NetFlow-Lite?

Cisco defines it as shown below. If you want to read more about NetFlow-Lite, please read this. To me, it’s a way for a network professional to see some visibility of what’s on the wire and gather statistics.

NetFlow-Lite collects packets randomly, classifies them into flows, and measures flow statistics as they pass through the switch. It is a true flow-based traffic-monitoring mechanism that conserves valuable forwarding bandwidth when exporting flow-based data for analysis and reporting.

Prior to sFlow and NetFlow-Lite, I was somewhat exposed with NetFlow but it was very limited implementation. That NetFlow implementation was good enough for what we used it for. Besides, the traffic generated by devices and/or computers on the network were very specific to the business applications and the computers were locked down tight so it was not needed at all. The places where we needed application visibility had protocol analyzers deployed so there was not a whole lot of push to deploy NetFlow.

NetFlow-Lite is not available in all Catalyst switches, I believe it was first supported on Catalyst 4948 platform and now being supported on newer Catalyst switches. The NetFlow-Lite requires the FPGA (Field-Programmable Gate Array) that contains the logic to implement NetFlow engine. Without it, then there won’t be support of NetFlow-Lite. Hence, no support on older platforms.

NetFlow-Lite Configuration

If you want to know what the commands do, please visit the configuration guide here.

flow record netflow
 match datalink mac source address input
 match datalink mac destination address input
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match ipv6 protocol
 match ipv6 source address
 match ipv6 destination address
 match transport source-port
 match transport destination-port
 collect transport tcp flags
 collect interface input
 collect flow sampler
 collect counter bytes long
 collect counter packets long
 collect timestamp sys-uptime first
 collect timestamp sys-uptime last
!
flow exporter collector
 description To NetFlow Collector
 destination 192.168.1.100
 source Vlan100
 transport udp 9985
 template data timeout 60
 option interface-table
!
flow monitor netflow
 record netflow
 exporter collector
 cache timeout active 30
!
sampler netflow
 mode random 1 out-of 32
!
!
interface range Gi1/0/1 - 48
 ip flow monitor netflow sampler netflow input
!
interface range Te1/0/1, TeX/0/1
 ip flow monitor netflow sampler netflow input

NetFlow/sFlow Collector

There are many vendors out there that sell flow collector software. Vendors out there like inMon (sFlow creator), Plixer, ntop, SolarWinds, etc. Make sure that they support NetFlow v9 or IPFIX since that’s the format that NetFlow-Lite can export to. Most of these vendors have trial software that you could use to give you a demo of their product. I am sure they’ll be happy to do a webinar so that they could introduce you to their product before starting to play with their software.

Thoughts

While NetFlow-Lite gave us some visibility, I noticed that sFlow provided more information so it is still better than not having any visibility at all. If your switches are capable of doing NetFlow-Lite, I suggest you do some trial to see if it’s going to be helpful for your environment. For us, it’s definitely helpful to have visibility so it is still being used. Another pretty cool feature that I find it very convenient is the fact that it could tell you the switch and port number of the device you’re looking for. While it’s not quite of a big deal to just log in to routers and switches to trace the device you’re looking for, it’s rather inconvenient to do so, especially if you implement two-factor for your switch-based authentication.

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

Filed Under: Switching Tagged With: Cisco, IOS, Switch

Same Data and Voice VLAN

12/24/2014 By Andrew Roderos Leave a Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Yes, it’s not the best practice to put both data and voice traffic in the same VLAN and subnet, but I’ve recently encountered it in production and caused us some head scratching scenario. While it was probably best to redesign the whole thing since it doesn’t follow best practices, it was not going to fly in this case.

Configuration

The stack’s configuration was copied from an existing Catalyst 3750-X production switch. For the most part, it’s a basic configuration that you will encounter in a lot of production switches. Some of the configurations are the following: AAA, BPDU Guard, PortFast, Storm Control, Access and Voice VLAN (same VLAN number), etc. The 3750-X stack’s configuration was not changed for more than two years. There were no issues or complaints from the clients so it was pretty much safe to copy the configuration to the new Catalyst 2960-X stack. This stack will be the new location of the same users of the 3750-X stack – just moving to a new location.

Issue

The desktop support guy started hooking up Avaya phones and desktops to the switch and noticed that all desktops and phones connected to switch 1 (two in a stack) were not communicating to the network. The desktops and phones weren’t getting IP addresses, but once he connects to switch 2 everything started to work. The interface configurations on both switches were configured the same but for whatever reason it wasn’t working. I rebooted switch 1 and all the desktops and phones connected to it started communicating to the network.

Several hours later, the desktop support guy said that everything connected to switch 2 stopped working. I decided to reboot the whole stack at this point. Before I rebooted the stack, I noticed that one of the switches was in EX4 and the other one was in EX5. Normally, there would be a version mismatch error when there are two different IOS installed and shouldn’t join the stack, but on this particular instance both switches were in the stack. I decided to upgrade the EX4 switch to EX5 since that is the latest code anyway and we’re making that as a standard IOS.

After upgrading and rebooting the stack, I still noticed that computers and phones on switch 2 were not communicating. At that point, our client was now panicking because they were not compliant with their contract to their client. That said, we decided to replace the whole stack with new switches. The switch replacement worked for a while but the issue came back which we kind of suspected anyway – we were not fully convinced that it would resolve the issue. Since we’ve been using the Catalyst 2960-X with EX5 code for several months now, we know it works just fine with our standard configuration. The only thing different with this stack and others was the same voice and data VLAN. While the switch is solid, we’ve hit bugs starting with EX1 all the way to EX4. That said, I begun to suspect that we might be hitting a bug in EX5, so I decided to open a TAC case while it was on the working state.

Troubleshooting

A Cisco TAC engineer contacted via e-mail me to troubleshoot the issue since it was working at the time I opened the case. As usual, the first thing I had to give them was show tech output. I still do not know why I do not include that by default when opening a case.  Anyway, as mentioned, the issue came back and I engaged the TAC engineer as soon as I was informed it stopped working.

The TAC engineer started checking some stuff up to try diagnose the issue. The first thing he mentioned is the voice and data VLAN being the same and it’s not a recommended practice. I agreed with him but I also reminded him that we have this deployed in two locations and were working fine using Catalyst 3750-X. He even asked me to connect to those switches so he could see it in his own eyes.

Upon issuing few show commands, we finally found some clue on what the problem was. There was no spanning tree instance on the interfaces where the phones and desktops were connected to. It acted as if there was nothing connected to the port even though the switch sees something connected to those ports. He decided to take out the switchport voice vlan vlan-id  and the desktop started pinging and spanning tree instance on the port showed forwarding.

While the configuration worked for desktops, I still needed the phones to work. Having only switchport access vlan vlan-id on the port won’t work with the phones even if I separated phones and desktop connections. That said, we needed a way to configure the port for the phones to also work. At the time, I only knew two ways of configuring a port with phones and data connected to the switch: configuring access vlan and voice vlan commands on the interface and configuring the interface as a trunk with native VLAN set for data. Essentially, they will act the same thing so we didn’t want to do that.

Upon him talking to his colleague(s) and probably done his research, he came back and said we’re going to try the switchport voice vlan dot1p command. That solved the issue! While he explained what the command did, I was not quite happy with his explanation because I felt it was incomplete, so I decided to read the configuration guide and was quite disappointed it didn’t have more information about it.

If you do a search, you will find plenty of information about the command, which I hoped the configuration guide would have. From what I’ve gathered both online and from the TAC engineer, if configured, the switch will instruct the phone to use VLAN 0 for its voice traffic and also mark its voice traffic with their proper CoS values, 5 or 3 depending on what traffic it is. Once the frame is received, the voice traffic with VLAN 0 will be accepted and drop the voice and data traffic in the same VLAN configured on the port – access vlan command.

Thoughts

While the dot1p configuration worked, I still would’ve preferred to have two different VLANs for both data and voice which is the standard configuration anyway in our environment. But, for whatever reason, they decided to design it this way and it’s going to stay that way. Maybe when it breaks that’s when they are going to decide to separate it, but at this point we had to choose the battle we were going to fight.

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Filed Under: Switching, VoIP Tagged With: Cisco, IOS, Switch

CCIE R&S v5 Home Lab

08/16/2014 By Andrew Roderos Leave a Comment

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Cisco announced CCIE Routing & Switching Version 5 exam update back on December 3, 2013. Even before the announcement, there were already speculations on what will or will not be on the exam. There were two things we knew for sure that will not be in the v5 exam, Frame Relay and IOS 12 – we all turned out to be right. That being said, my old CCIE R&S v4 home lab needed an upgrade.

UPDATED: If you are running ESXi 6.x, please look at the updated section below.

Before the CCIE R&S v5 exam has been released, I’ve been studying for the written exam by using the two very well known books for Cisco routing and switching – it does not matter if one is studying for CCNA R&S, CCNP R&S, or CCIE R&S. These two books are very awesome resources to have on their bookshelf at work or at home. The two books are Routing TCP/IP, Vol 1 and Routing TCP/IP, Vol 2. While these two books are old, they are still very helpful! According to Jeff Doyle, he is working on updating at least one of the books. You can now order Routing TCP/IP, Volume II Second Edition!

The other two books that a CCIE aspirant must have are the following: CCIE R&S v5.0 Official Cert Guide, Vol 1 and CCIE R&S v5.0 Official Cert Guide, Vol 2. The first two books are also available in a bundled format, CCIE R&S v5.0 Official Cert Guide Library, which costs less than buying them separately. I do want to point out that these books are not a replacement for Routing TCP/IP Vol 1 and 2.

Cisco’s Recommended Gears

The recommended gears to prepare for the exam are ISR G2 2900s and Catalyst 3560X, as stated in this document. I can’t afford all of those gears, especially with INE’s hardware topology. As you can see, the topology has 20 routers for full-scale labs but the advanced technology labs require only 10 routers. Even with 10 routers, I still can’t afford to buy them. On top of all the routers, you still need the 3560X which costs an arm and leg just like the 2900s. Fortunately, there are other options in building a CCIE home lab that will mimic INE’s workbook topology. I opted for the CSR 1000v which requires a hypervisor, like VMware’s ESXi, Microsoft’s Hyper-V, Xen, and KVM.

The CSR 1000v takes care of the routing section of the lab, but I still need something for the switching section. Fortunately, I can reuse my two Catalyst 3560s loaded with IOS 15. Yes, you read that right. There are some versions of Catalyst 3560v1 that are capable of loading IOS 15 as long as you have the 32MB flash version. I have two flavors of 3560: WS-3560-24TS-S and WS-3560-48TS-S. If I try to log into Cisco’s IOS download page with your CCO account and start looking for an IOS 15 image for that model, you won’t be able to find one. The last version for these models is 12.x, but if you look under 3560G then you have the option to download IOS 15 and the images do work on these two models.

With two on hand, I needed to add two more to complete my CCIE home lab. Fortunately, we have tons of them at work that aren’t being used at all. I wanted to buy them from my employer but was told that I didn’t have to. I was given a permission to borrow two of them for a long period so I took that opportunity and borrowed two more of the WS-3560-24TS-S. With four 3560s, I now have a complete CCIE R&S v5 home lab! While it won’t do 100% of the topics covered in the lab exam, at least I can still do the majority of the topics with my home lab. For the topics that won’t be able to do with my home lab, I can always do rack rental or probably use VIRL (Virtual Internet Routing Lab).

ESXi Server

Since I already have a VMware ESXi home lab, it was an easy decision on which hypervisor to use and it’s what INE is using anyway for their rack rental. The ESXi server build that I have is a couple of years old so if you’re looking for a new one, then check out my new build to get some idea.

Related: VMware ESXi Home Lab – Intel NUC 10 (Frost Canyon)

My new build is underpowered compared to my first ESXi host, but the Intel NUC Skull Canyon is a great candidate for your home lab. If the build is still pricey for you, eBay is there for old servers. I almost bought the Dell T5500 but glad I decided to hold off on the purchase. Also, I didn’t like the fact that it was too big since I do not have space for it unless I clear out my rack. Anyone wants to buy my old routers and switches that were part of my CCIE R&S v4 home lab? The decision to hold off turned out to be a great decision since I was able to squeeze all 20 CSR 1000v with 2.5GB RAM configured on each VM, more on that later.

Installation

I am not going to do a tutorial on how to install it since INE has one already. However, I do want to show some tips that I’ve learned from the INE’s forum members on both CCIE R&Sv5 Equipment Build and Building INE’s RSv5 topology on CSR1000v threads.

INE’s blog post about how to install CSR 1000v is a bit old, but you can still follow the wizard since it’s pretty self-explanatory. One of the differences would be after the Name and Location section, which is new with the OVA file that I downloaded. The Deployment Configuration section will ask you what type of deployment you would like to use – Small, Medium, Large, and Large + DRAM Upgrade. For the lab environment, a small hardware profile is sufficient.

If you still decided to use ESXi 5.5, then deploying the OVF template install is different from ESXi 5.1. Different in terms of how it creates the VM. The ESXi 5.1 used hardware version 9 versus hardware version 10 of ESXi 5.5. You’re probably asking, so what if ESXi 5.5 uses hardware version 10. Well, you cannot edit the VM using the vSphere client. You would need to use vSphere Web Client to edit the VM, as seen below. That means you would need to run a vCenter Server instance to just run vSphere Web Client which complicates your home lab. Then again, if you are studying for VCP or will study for it, then it doesn’t really matter. However, for people who just want to get their CCIE home lab running, then it will be annoying to go through the process of building a vCenter Server instance just to edit the VM.

editing the vm
Editing the VM

Tips

If you still decided to use ESXi 5.5, then there are ways around the error that you get when trying to edit a virtual machine that is version 10 or higher. Some people decided to copy the VMX file from the ESXi server to their desktop and edit the file and re-upload it. I decided to use CLI of the ESXi server since I am familiar with it. To access the CLI, you’ll need to enable SSH, it’s mentioned in one of my blog posts. Next, you will need to find the VMX file of your CSR 1000v. In my case, my baseline VM is called CSR1000v and normally what you call your VM will be also the name of the folder within your datastore.

~ # vi /vmfs/volumes/nfs01/CSR1000V/CSR1000V.vmx
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "10"
nvram = "CSR1000V.nvram"
! Output omitted for brevity !

The line that we’re looking for here is the virtualHw.version = “10”. We need to change it from 10 to 9 and save the file. We’re still not quite done yet, even though we’ve changed the hardware version. The vSphere Client will still give you an error if you try to edit it since it still thinks that the VM is version 10. We need to remove the VM from the inventory and add it back in. Once added, the VM can now be edited. My suggestion is not to edit this and make this as your baseline VM for CSR 1000v needs. Since we need 20 CSR 1000vs for INE’s full-scale labs, we need to start cloning the VM. Once you’ve created all 20 VMs, you’re now ready to edit them.

In INE’s blog post, it tells you to issue the command platform hardware throughput level 50000 but this one doesn’t prompt you to activate the evaluation license with a newer CSR 1000v image, at least on the one I downloaded – csr1000v-universalk9.03.12.00.S.154-2.S-std.ova.

R1(config)#platform hardware throughput level 50000
R1(config)#
*Aug  6 04:36:20.067: %VXE_THROUGHPUT-3-CONF_FAILED: Configuration failed. Installed license does not support the throughput level. Please install the valid license

To activate the evaluation premium license, you need to issue the command below and accept the EULA. Before activating the license, you may want to do a snapshot or configure your VM’s Virtual Disk to non-persistent. This way, you can still take advantage of the higher throughput level compared to a measly 2.5Mbps once the evaluation license expires. For the lab environment, though, the basic throughput is enough so it’s really optional. Another option is to activate the license and put all the initial configs and then convert the VM’s Virtual Disk to non-persistent. This will give you fresh evaluation license every time you the shutdown and power up the VM.

R1(config)#license boot level premium 
         Feature Name:prem_eval
! Output omitted for brevity !
ACCEPT? (yes/[no]):yes
*Aug  6 04:41:05.560: %LICENSE-6-EULA_ACCEPTED: EULA for feature prem_eval 1.0 has been accepted. UDI=CSR1000V:9L5WSSXMWKP; StoreIndex=0:Built-In License Storage% use 'write' command to make license boot config take effect on next boot
*Aug  6 04:41:09.344: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = csr1000v Next reboot level = premium and License = prem_eval
R1#show license
! Output omitted for brevity !                
Index 30 Feature: prem_eval                      
        Period left: 8  weeks 3  days 
        Period Used: 0  minute  3  seconds 
        License Type: Evaluation
        License State: Active, Not in Use, EULA accepted
        License Count: Non-Counted
        License Priority: Low
! Output omitted for brevity !

Once the license is activated, then you can now change the throughput to 50Mbps.

R1(config)#platform hardware throughput level 50000
R1(config)#
*Aug 21 01:34:36.573: %VXE_THROUGHPUT-6-LEVEL: Throughput level has been set to 50000 kbps

What happens when the evaluation license expires? Well, as stated earlier the throughput goes back down to 2.5Mbps but you still get all the premium license features. You also get an annoying message about the license being expired. If you’re doing debugs and what have you then it gets annoying to see one of the lines is about the license. That said, the non-persistent disk or snapshot becomes handy in this situation.

R1#show platform hardware throughput level
The current throughput level is 2500 kb/s
R1#             
%LICENSE-1-EXPIRED: License for feature prem_eval 1.0 has expired 7 hours and 30 minutes ago. UDI=CSR1000V:905F9PPAKYB

Since my ESXi server only has 32GB of RAM (maxed out), the amount of CSR 1000v VMs that I can do is probably around 12 excluding my other VMs. That means I will be short of 8 routers for the full-scale labs. My original plan was to do memory overcommitment and take advantage of the performance that you get from SSDs. That said, I bought a 128GB Samsung SSD to be used as a host SSD cache. In theory, the swapping to SSD shouldn’t be noticeable for a lab environment. I was ready to install the SSD until I saw a guy posted in the IEOC thread about disabling the Transparent Page Sharing (PDF) Large Page support feature. By default, some ESXi versions support large pages and small pages. By disabling the large page support for OS, VMware ESXi kernel will force the guest OS to use small page support and will increase the chances of TPS being able to keep one copy of the page and share it among the VMs. Since we’re running the same OS twenty times over, there are some contents that will be the same for each and every VM. That said, it would make sense to just keep one and share it among the VMs, which reduces the amount of physical memory being used by the VMs and allows a higher level of memory overcommitment. That’s how I was able to run 23 VMs concurrently and use only about 16GB of physical memory.

To disable the large page behavior and instead force small pages, go to Configuration tab > Advanced Settings (under Software) > Mem > Mem.AllocGuestLargePage > Change the value from 1 to 0.

Forcing small page support
Forcing small page support

Once disabled, it would take some time to scan the VMs and reduce the amount of RAM used. That being said, just try to run 10 or so CSR VMs and wait several minutes for the ESXi server to scan them for duplicate memory pages. Once ESXi is able to scan fully, you should start to see your memory consumption to drop significantly.

UPDATE: For those people who are running certain versions of ESXi 5.x and now 6.0, the TPS is disabled by default. My ESXi hosts are now on 6.0U2 6.5 and noticed that Mem.AllocGuestLargePage setting changed back to the value of 1. Please check your settings and change it back to 0. In addition, there is a new setting in certain versions of 5.x and version 6.0 that needs to be changed. This is to revert back to the traditional behavior of TPS. The setting is the Mem.ShareForceSalting and needs to be changed to 0. Please be aware that there is a VMware KB that talks about security concerns of TPS. In a CCIE home lab environment, I do believe that it is perfectly OK to revert back to the traditional behavior.

Inter-VM Transparent Page Sharing

Previously, once the Mem.AllocGuestLargePage has been changed to 0 the inter-VM TPS will kick in after several minutes. However, I noticed that the behavior of ESXi 6.0U2 is quite different. The inter-VM TPS did not kick in until the VMs were powered down or migrated (vMotion) to a different host. Once done, RAM consumption will start to decrease but may take 15 – 30 minutes before it goes down to around 14 GB. Ignore the other VMs on the left since those are on a different host. The RAM consumption of 20 CSR1000v should be around the same as mine.

TPS Enabled
TPS Enabled

Thoughts

If you haven’t been living under a rock, then you definitely know that there are several options out there. Some people decided to use Web IOU or Unified Networking Lab by @adainese, GNS3 v1.x, IOSv that is extracted from the OnePK, or a combination of hardware and virtual. If you are not Cisco employee, then you shouldn’t be running IOU/IOL since it is against Cisco’s EULA. That said, run it at your own risk.

The GNS3, Web IOU, and Unified Networking Lab options are great since you can take them on the go because the system requirements are not very high so any decent notebook can run it. The problem with IOU/IOL, they weren’t designed to be used for learning so some features may not work properly and can be frustrating at times. The IOSv’s, which is the software that will be used in VIRL, system requirements are not very high as well so if you have a not so powerful desktop/notebook, then this is definitely something you can consider.

Since I happened to have an ESXi server, I made a decision to utilize it since I was only using it for playing with some OS and a few VMs that I use for everyday use like FTP, proxy, Plex, etc. Another reason why I went with the CSR 1000v route is because the INE’s labs were written with their rack rental in mind. That said, the convenience that you get with following INE’s setup is priceless. I am quite happy with my setup and my ESXi server can handle the load that I am currently throwing at it. Then again, I haven’t finished the whole advanced technology labs so take it with a grain of salt.

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

  • Share on Twitter Share on Twitter
  • Share on Facebook Share on Facebook
  • Share on LinkedIn Share on LinkedIn
  • Share on Reddit Share on Reddit
  • Share via Email Share via Email

Filed Under: Certifications Tagged With: CCIE, Cisco, Home Lab

  • « Go to Previous Page
  • Page 1
  • Interim pages omitted …
  • Page 5
  • Page 6
  • Page 7
  • Page 8
  • Page 9
  • Interim pages omitted …
  • Page 18
  • Go to Next Page »

Footer

WORK WITH US

Schedule a free consultation now!

LET’S TALK

Copyright © 2011–2026 · NetworkJutsu · All Rights Reserved · Privacy Policy · Terms of Use