NetworkJutsu

Network Security Consulting | San Francisco Bay Area

Nexus

DHCP Issues (CoPP Profile)

By Leave a Comment

There are several ways to troubleshoot DHCP issues. One could verify that DHCP scope is good, verify if the DHCP relay configuration is configured correctly, etc. Today, this blog post will cover a possible solution for DHCP issues that one might encounter. Though, this might be unlikely to happen in some networks. Of course, that depends on the environment.

Symptom

There were several IP phones spread throughout the enterprise that was working before but then stopped getting IP addresses. Other phones on the same VLAN were getting IP addresses just fine though. Several buildings were affected by this issue and when the phones were replaced, they were still not getting IP addresses. If they were brought back to the tech’s office, the phones did get IP addresses but the topology is not the same, so the only thing that test tells us was that the phones were not the issue.

Troubleshooting

Few steps were already taken so either one would verify everything again or save some time and continue where it was left off before the escalation. For example, DHCP scope was checked already to make sure they were correct and there were available IP addresses left to give out, voice VLAN was set, VLAN is on the switch’s VLAN database, DHCP relay configurations were correct, etc. One interesting piece of information was that the DHCP server never saw DHCP Discover and/or DHCP Request from the phones that were having issues. Majority of the time (at least in my experience), the issue was the relay configuration. This time, it was not. This makes sense since only a few phones were affected and not the whole voice VLAN.

So far, this tells us that somewhere in the network is dropping packets. Before busting out the Wireshark or Ethanalyzer, if available, to track where the packet is being dropped, one possible location where it is being dropped is the gateway. In this scenario, the gateway is a pair of Nexus 7000 running HSRP. With Nexus switches, CoPP (Control Plane Policing) is one of the configurations that will be set during the initial configuration.

In this scenario, the CoPP is set to strict. This can be verified by several ways and one way is by using show run | i “copp profile” command. To verify if the CoPP is being violated, one has to issue the command below.

N7K-enable# show policy-map interface control-plane class copp-system-p-class-normal-dhcp
Control Plane
  service-policy  input: copp-system-p-policy-strict
    class-map copp-system-p-class-normal-dhcp (match-any)
      match access-group name copp-system-p-acl-dhcp
      match redirect dhcp-snoop
      set cos 1
      police cir 680 kbps , bc 250 ms
      module 3 :
        conformed 3430912611 bytes; action: transmit
        violated 1331596081 bytes; action: drop
      module 4 :
        conformed 6375239885 bytes; action: transmit
        violated 2201866339 bytes; action: drop

As shown above, the CoPP for DHCP is being violated and packets are being dropped. There are some instances that the nodes will recover since it will send DHCP Discover again, but there might be times where a custom CoPP is needed. To verify that this policy is the right one for DHCP, issue the commands below.

N7K-enable# sh run copp all | sec normal-dhcp
class-map type control-plane match-any copp-system-p-class-normal-dhcp
  match access-group name copp-system-p-acl-dhcp
  match access-group name copp-system-p-acl-dhcp
  match redirect dhcp-snoop
N7K-enable# show ip access-lists copp-system-p-acl-dhcp
IP access list copp-system-p-acl-dhcp
        10 permit udp any eq bootpc any
        20 permit udp any neq bootps any eq bootps

Custom CoPP

To customize the CoPP to fit in one’s environment, the recommended is to double the rate and monitor for issues. If problem persists, continue to modify the rate until the issue is gone. Make sure to monitor the CPU as well because this might be impacted with the changes on the CoPP since we’re allowing more packets to go through the control plane.

The first step in customizing the CoPP is to copy the profile.

N7K-enable# copp copy profile strict suffix custom-copp

Once the CoPP profile has been copied, the value(s) that need(s) modification can now be done.

N7K-enable(config)# policy-map type control-plane copp-policy-strict-custom-copp
N7K-enable(config-pmap)# class copp-class-normal-dhcp-custom-copp
N7K-enable(config-pmap-c)# police cir 2048 kbps bc 1250 ms conform transmit violate drop

The configuration is well above the recommended rate, but to eliminate the possibility of the CoPP being the issue, it might be a good idea to increase it high enough so that no packets will be dropped.

The last step is to now assign the policy to the control plane and some verification.

N7K-enable(config)# control-plane
N7K-enable(config-cp)# service-policy input copp-policy-strict-custom-copp
N7K-enable(config-cp)# exit
N7K-enable (config)# exit
N7K-enable# copy run start
N7K-enable# show copp status
Last Config Operation: service-policy input copp-policy-strict-custom-copp
Last Config Operation Timestamp: 06:04:03 UTC Jan 1 2015
Last Config Operation Status: Success
Policy-map attached to the control-plane: copp-policy-strict-custom-copp
N7K-enable# show policy-map interface control-plane class copp-class-normal-dhcp-custom-copp
Control Plane
  service-policy  input: copp-policy-strict-custom-copp
    class-map copp-class-normal-dhcp-custom-copp (match-any)
      match access-group name copp-acl-dhcp-custom-copp
      match redirect dhcp-snoop
      set cos 1
      police cir 2048 kbps , bc 1250 ms
      module 3 :
        conformed 980075838 bytes; action: transmit
        violated 0 bytes; action: drop
      module 4 :
        conformed 275544063 bytes; action: transmit
        violated 0 bytes; action: drop

Thoughts

This may be an uncommon solution for DHCP issues but once a similar symptom is being experienced, then it does not hurt to do some verification that the CoPP is being hit. There are still a lot things I have to learn in Nexus platform since it is quite different from the Catalyst platform. This is one of those things that is different between Nexus and Catalyst.

Want to learn more about NX-OS?

NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.

How to configure sFlow on Nexus 3000

By Leave a Comment

I mentioned about sFlow in this blog post. Today, I want to show how to configure it on Nexus 3000 series. The configuration lines covered here were tested on both Nexus 3172 and 3048. Nexus 3524/3548 is not sFlow capable so please check the Cisco’s documentation to see if the switch supports it.

What is sFlow?

sFlow, short for sampled flow, is an industry standard sampling technology for monitoring traffic in data networks. It was originally developed by InMon Corporation, maker of Traffic Sentinel – a flow collector, and it is now defined by RFC 3176, for the most part.

Configuration

If you are familiar with NX-OS, then you know that the feature command is pretty much needed to enable the features you want on Nexus switches. This is no different from the other features such as BGP, OSPF, etc.

feature sflow

The command above enables the sFlow feature.

The next two settings are important and one needs to consider how it should be configured on his/her network devices. These two settings are sampling rate and polling interval. The sampling rate is the number of samples of packets that traverses the interface in a specified period of time (polling interval).

Let’s take a look at a scenario where the sampling rate is set to sample 1 of 5,000 and the polling interval is set to 60 seconds. Let’s say that in 60 seconds there were 50,000 packets traversed the interface. With the configured sampling rate, sFlow collected 10 out of 50K packets. This is quite a small number of packets collected and doesn’t really reveal the overall network traffic. Let’s say that sFlow collected five packets of HTTPS traffic, one of NFS, one of SSH, one of FTP, and one of ARP. One could speculate that there were more HTTPS traffic but one could never be 100% sure because 10 is a very small number.

Make sure to adjust the sampling rate and polling interval for your needs. The configuration below is just a sample based on the scenario above. Switches that support sFlow have an ASIC that is capable of handling more frequent collection. That said, feel free to collect as more often as you want since it will consume less resources than NetFlow does. Though, newer Cisco switch might have the necessary ASIC(s) that can handle NetFlow as good as the sFlow.

sflow sampling-rate 5000
! Sampling rate is configurable from 4096 to 1000000000. 0 disables the sampling. 4096 is the default value.
!
sflow counter-poll-interval 60
! Polling interval is configurable from 0 to 2147483647. 20 seconds is the default.

The next important command is the data source. With no data source, there is nothing to report to the collector.

sflow data-source interface e1/1 - 50
sflow data-source interface port-channel 1

If all the interfaces on the switch is defined as the data source, there will be an error if in the future one decides to use the interface for an EtherChannel. That said, they need to be taken out of the data source and added back in, if one needs to collect sFlow data. If the interface is a member of an EtherChannel, one has to add the logical interface as the data source and not the physical interface.

Now, we need to point the switch to the flow collector. Check with your flow collector vendor if it supports sFlow. I know that InMon’s Traffic Sentinel supports multiple flows like sFlow (obviously, they originally developed the protocol!), NetFlow/NetFlow-Lite/Flexible NetFlow, IPFIX, etc.

sflow collector-port 6343
! 6343 is the default and the official sFlow port.
!
sflow collector-ip 192.168.100.100 vrf default (or management)
sflow agent-ip 192.168.1.10

If it’s not obvious enough, the collector-ip is the IP address of the flow collector. The vrf option depends on how everything is set up. The management would be used if one is using the management port and the collector is reachable via that interface. If not, use the default option. The agent-ip is the source IP address that one wants to use. Use the management interface’s IP address if that is the one chosen in the vrf option.

The last two configurations are basically optional. They have default values but may want to configure depending on one’s environment.

sflow max-sampled-size 128
! This configure the maximum number of bytes that should be copied from a sampled packet.
!
sflow max-datagram-size 1400
! This configure the maximum number of data bytes that can be sent in a single sample datagram.

Thoughts

If you have a flow collector and Nexus switches that support sFlow, consider enabling it. It allows you to have some type of network visibility that may help you in the future. For example, the one I mentioned here, finding out who the target of a DoS/DDoS attack, top talkers of the network, etc.

Reference

Configuring sFlow

Want to learn more about NX-OS?

NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures

Disclosure

NetworkJutsu.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.