© 2012 Cisco and/or its affiliates. All rights reserved. Nexus 7000 and Nexus 3000 "TAC Time" Mike Pavlovich Yogesh Ramdoss
© 2012 Cisco and/or its affiliates. All rights reserved.
Nexus 7000 and Nexus 3000"TAC Time"
Mike Pavlovich
Yogesh Ramdoss
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
• Introduction
• Nexus 7000 vPC Common Questions
• Nexus 7000 CoPP
• Nexus 7000 Ethanalyzer
• Nexus 7000 Handy Features
• Nexus 7000 Important Caveats
• Nexus 7000 References
• Nexus 3000 Basic Information
• Nexus 3000 Important Caveats / Hot Issues
• Nexus 3000 Best Practices
• Nexus 3000 References
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
• Mike Pavlovich
• CCIE R&S # 4284
• Technical Leader, Data Center Networking Team, Cisco Services
• Supports Nexus 7000, Nexus 6000, Nexus 3000, and Catalyst Switches.
• Yogesh Ramdoss
• CCIE R&S # 16183, VCP 5.0
• Technical Leader, Data Center Networking Team, Cisco Services
• Supports Nexus 7000, Nexus 6000, Nexus 3000, and Catalyst Switches.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
• Nexus 7000 vPC Common Questions • Nexus 7000 CoPP• Nexus 7000 Ethanalyzer• Nexus 7000 Handy Features • Nexus 7000 Important Caveats• Nexus 7000 References
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 5
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
Why do we need vPC Auto-Recovery?• Scenario 1: Power outage shuts down both Nexus 7000 vPC peers
simultaneously and only one switch is able to come back up (5.0(2) “reload restore” feature addressed this but not scenario 2 below)
• Scenario 2: vPC peer-link is lost first and then the primary vPC peer switch is powered down
Configuration of vPC auto-recovery (5.2(1)):
• vPC auto-recovery timeout: to see if either the vPC peer-link comes up or peer-keepalive status is up. If so then auto-recovery will not get triggered. Default 240 sec – configurable via “auto-recovery reload-delay x” (x = 240-3600 sec)
S1 (config)# vpc domain 1S1 (config-vpc-domain)# auto-recovery
S2 (config)# vpc domain 1S2 (config-vpc-domain)# auto-recovery
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Scenario 1
0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Scenario 2
0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
• Q: Which Nexus 7000 would respond to a ping or ARP request for the Virtual IP Address (HSRP/VRRP) in a vPCscenario?
• A: The HSRP Active will respond (when both vPC legs are up)
• vPC is Active/Active for HSRP/VRRP regarding L3 switching of traffic received with the destination virtual mac address. Both peer switches own the virtual MAC so which ever peer gets the packet will forward it and not send it across the peer link
• The virtual IP address on the other hand is owned by the HSRP/VRRP active Nexus 7000 in a vPC scenario
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 10
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
Did you know?...
• The CoPP rates are applied per FE (forwarding engine) on the line cards. Traffic hitting the CPU = conform rate x # of FE’s
• F1 modules do not use CoPP. They use hardware rate-limiters instead
Nexus7K(config)# hardware rate-limiter f1 ? rl-1 STP and Fabricpath-ISIS rl-2 L3-ISIS and OTV-ISIS rl-3 UDLD, LACP, CDP and LLDP rl-4 Q-in-Q and ARP request rl-5 IGMP, NTP, DHCP-Snoop, Port-Security and Mgmt traffic
M1 modules = 1 x FE (2 for M108)M2 modules = 2 x FEF2 modules = 12 x FE (SoC = Switch on Chip)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
• Four options for default policies:
• All of these options use the same class-maps and classes, but different rate & burst values
• CoPP is configured on default VDC but effects all VDC’s. CoPP is applied per FE so recommend all ports on same FE be part of same VDC if possible
• Prior to 5.2(1), the setup command was used to change the CoPP option. 5.2(1) introduced the copp profile command.
StrictModerateLenientDense (introduced in 6.0(1))
Applied if no option is selected or if set up is skipped
recommended if majority of modulesare F2 Series (12 FE’s each)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
Nexus7k# conf tNexus7k(config)# copp profile strictNexus7k(config)# exit
Nexus7k# show copp statusLast Config Operation: copp profile strictLast Config Operation Timestamp: 20:40:27 PST Apr 21 2013Last Config Operation Status: SuccessPolicy-map attached to the control-plane: copp-system-p-policy-strict
Nexus7k# show copp diff profile strict profile moderate<output left out to save space… but you get the idea>
Nexus7k# conf tNexus7k(config)# control-planeNexus7k(config-cp)# scale-factor 2.0 module 1Nexus7k# show system internal copp info<snip>Linecard Configuration:-----------------------Scale FactorsModule 1: 2.00etc…
Scale-factor-6.0 feature to increase or reduces the policer rate of the applied CoPP policy for a particular linecard-Value ranges from 0.10 to 2.0-Recommended for chassis with M & F2 modules
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
Nexus7k(config)# policy-map type control-plane copp-system-policy-strictNexus7k(config-pmap)# class copp-system-class-criticalNexus7K (config-pmap-c)# logging drop threshold 10000 level 5
Sample Syslog:
10000 = # of bytes
%COPP-5-COPP_DROPS5: CoPP drops exceed threshold in class: copp-system-class-critical, check show policy-map interface control-plane for more info.
Nexus7k# show policy-map interface control-plane | i"class|conformed|violated|module"
class-map copp-system-class-critical (match-any)module 1:conformed 123126534 bytes; action: transmitviolated 143021 bytes; action: drop
module 2:etc..
Nexus7K# show hardware rate-limiter | in "Module|f1|Class”Module: 1 R-L Class Config Allowed Dropped Total f1 rl-1 4500 0 f1 rl-2 1000 0etc…
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 15
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
• Capture traffic to/from CPU associated with module interfaces (inband) or Sup MGMT interface (mgmt)
Nexus7k# ethanalyzer local interface ?inband Inband/Outband interfacemgmt Management interface
Nexus7k# ethanalyzer local interface inband ?<CR> > Redirect it to a file>> Redirect it to a file in append modeautostop Capture autostop conditioncapture-filter Filter on ethanalyzer capturecapture-ring-buffer Capture ring buffer optiondecode-internal Include internal system header decodingdetail Display detailed protocol information display-filter Display filter on frames capturedlimit-captured-frames Maximum number of frames to be captured (default is 10)limit-frame-size Capture only a subset of a frameraw Hex/Ascii dump the packet with possibly one line summarywrite Filename to save capture to| Pipe command output to filter
Nexus7k#
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Nexus7k# ethanalyzer local interface inbandCapturing on inband2013-04-15 17:20:53.864238 c8:9c:1d:39:87:34 -> 01:80:c2:00:00:0e LLC U, func=UI; SNAP, OUI 0x00000C (Cisco), PID 0x01342013-04-15 17:20:53.922775 00:23:33:74:47:04 -> 01:80:c2:00:00:00 STP Conf. Root = 32768/1/00:23:33:74:47:00 Cost = 0 Port = 0x80052013-04-15 17:20:53.977277 00:1b:54:c1:73:53 -> 01:00:0c:cc:cc:cd STP RST. Root = 32768/95/00:24:98:6f:ba:c3 Cost = 0 Port = 0x905e2013-04-15 17:20:53.985859 00:15:fa:42:5d:98 -> 01:80:c2:00:00:00 STP MST. Root = 4096/0/00:13:5f:20:bb:80 Cost = 0 Port = 0x96862013-04-15 17:20:53.986011 00:01:00:01:00:01 -> 01:80:c2:00:00:0e LLC U, func=UI; SNAP, OUI 0x00000C (Cisco), PID 0x88402013-04-15 17:20:54.278543 70:ca:9b:95:cc:a5 -> 01:80:c2:00:00:41 0x22f4 Ethernet II2013-04-15 17:20:54.396876 f8:66:f2:e4:b9:dd -> 01:80:c2:00:00:41 0x888a Ethernet II2013-04-15 17:20:54.476706 10.10.10.2 -> 10.10.10.1 UDP Source port: 3200 Destination port: 32002013-04-15 17:20:54.515927 10.10.16.6 -> 224.0.0.10 EIGRP Hello2013-04-15 17:20:54.516058 10.10.16.6 -> 224.0.0.10 EIGRP Hello
10 packets capturedNexus7k#
Note: <cntl>C will stop the ethanalyzer captureNote: by default the output is displayed on your screen. To save the output to a file use the “write” & “read” options
Nexus7k# ethanalyzer local interface inband capture-filter "stp" limit-captured-frames 2Capturing on inband2013-04-15 17:12:42.289309 00:15:fa:42:5d:98 -> 01:80:c2:00:00:00 STP MST. Root = 4096/0/00:13:5f:20:bb:80 Cost = 0 Port = 0x96862013-04-15 17:12:42.616792 88:43:e1:c7:4d:b8 -> 01:80:c2:00:00:00 STP MST. Root = 4096/0/00:13:5f:20:bb:80 Cost = 200 Port = 0x9000
2 packets capturedNexus7k#
Nexus7k# ethanalyzer local interface inband capture-filter "host 10.10.16.6" limit-captured-frames 1 write bootflash:testCapturing on inband1 Nexus7k# ethanalyzer local read bootflash:test2013-04-15 17:29:15.679219 10.10.16.6 -> 224.0.0.10 EIGRP HelloNexus7k#
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
Capture Filter Traffic Captured
host 1.1.1.1 to or from a host
net 172.16.7.0/24 (or “net 172.16.7.0 mask 255.255.255.0”) to or from a range of IP addresses
src net 172.16.7.0/24 (or “src net 172.16.7.0 mask 255.255.255.0”) from a range of IP addresses
dst net 172.16.7.0/24 (or “dst net 172.16.7.0 mask 255.255.255.0”) to a range of IP addresses
port 53 only certain protocol e.g. DNS
port 67 or port 68 DHCP traffic
host 172.16.7.3 and not port 80 and not port 25 is not certain protocols E.g. not HTTP or SMTP
port not 53 and not arp except ARP & DNS
ip only IP traffic
not broadcast and not multicast only unicast traffictcp portrange 1501-1549 within a range of Layer 4 portsether proto 0x888e based on Ethernet type E.g. EAPOLether proto 0x86dd IPv6 captureip proto 89 IP protocol typenot ether dst 01:80:c2:00:00:0e Reject Ethernet frames based on mac address
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 19
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
Nexus7k# checkpointProcessing the Request... Please Wait................................. DoneNexus7k# show checkpoint summaryUser Checkpoint Summary--------------------------------------------------------------------------------1) user-checkpoint-1:Created by adminCreated at Fri, 06:49:06 26 Apr 2013Size is 39,156 bytesDescription: None
Nexus7k# config tNexus7k(config)# no vlan 20, 30
Nexus7k# rollback running-config ?checkpoint Rollback running configuration to checkpointfile Rollback running configuration to configuration file
Nexus7k# rollback running-configuration checkpoint user-checkpoint-1Note: Applying config parallelly may fail Rollback verificationCollecting Running-ConfigGenerating Rollback PatchExecuting Rollback PatchGenerating Running-config for verificationGenerating Patch for verification
Nexsu7k# clear checkpoint databaseProcessing the Request... Please Wait.................................. Done
Caution! Clears all saved configurationsClear Checkpoint Database:
Automatically puts VLAN 20 & 30 back into running Configuration
Rollback to configuration “user-checkpoint-1”
Rollback Configuration:
Create and Verify Checkpoint Configurations:Create checkpoint configuration (default name = “user-checkpoint-#”)You can create up to 10 configurations per VDC
Manually Removed VLAN 20 & 30
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
• GZIP
• Accounting Log
Nexus7K# show tech-support gold > bootflash:tech_goldNexus7K# dir bootflash:tech_gold5944217 Apr 26 09:14:13 2013 tech_gold
Nexus7K# gzip bootflash:tech_goldNexus7K# dir bootflash:tech_gold.gz332467 Apr 26 09:14:13 2013 tech_gold.gz
Nexus7K# config tNexus7K(config)# feature ospfNexus7K(config)# exitNexus7K# show accounting log | i “Apr 26”Fri Apr 26 09:24:18 2013:type=update:id=console0:user=admin:cmd=configure terminal ; feature ospf (SUCCESS)Nexus7K#
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
• Locator LED / Beacon
• Show {blah} | no-more
Nexus7K# locator-led ? chassis Blink chassis led fan Blink Fan led module Blink module led powersupply Blink powersupply led xbar Xbar
Nexus7K# locator-led chassisNexus7K# show locator-led status
Nexus7K# config tNexus7K(config)# interface ethernet 1/1Nexus7K(config-if)# beacon
Nexus7K# show running-config | no-more
Output of given show command will complete without the need to hit the space bar
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
• Show {blah} | diff
• Send Message Nexus7K# send ?
LINE Send message (a line) to all open sessions session Send message to specific session
Nexus7K# send We found the problem, its not the NexusBroadcast message from admin (Fri Apr 26 10:06:27 2013):We found the problem, its not the Nexus
Nexus7K#
Nexus7K# show interface ethernet 1/1 | diff2,3c2,4< admin state is up, Dedicated Interface< Hardware: 10000 Ethernet, address: d0d0.fd9d.a680 (bia d0d0.fd9d.a680)---> admin state is down, Dedicated Interface> Hardware: 10000 Ethernet, address: 0024.f714.3541 (bia d0d0.fd9d.a680)> Internet Address is 172.10.10.1/24
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 24
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
• Only effects F2 module admin down ports (Non-impacting)
• ISSU from 6.0(x) to 6.0(x) and then ISSU to 6.1(x) is trigger
• ISSU from 6.0(x) direct to 6.1(x) will not see the issue
• Workaround: Reload module
• Resolved: in 6.1(1) and later
• DDTS: CSCua03125
%DIAG_PORT_LB-2-PORTLOOPBACK_TEST_FAIL: Module:7 Test:PortLoopback failed 10 consecutive times. Faulty module: affected ports:1,3,4,10-12,14-17,20,25,27,28,37,39-41,43-46,48 Error:TestFailed, Could not identify the Faulty Device
show diagnostic result module {x} detail 6) PortLoopback:
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 -----------------------------------------------------
U U U U . . . . . . . . E E U U
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
• Some packets in an L2 vlan (No SVI configured) are still hitting CoPP although they are not sent to the CPU
• Limited to packets with unicast IP address and a multicast/broadcast mac address. This includes ARP and DHCP requests
• These packets may congest the policy and could cause other traffic hitting the same policy to be dropped
• Day 1 behavior (all software releases)
• Workaround: None
• Resolved: in progress
• DDTS: CSCub47533
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
• 6KW AC power supplies that have SN starting with “AZS” may inadvertently shutdown momentarily when power is restored to input 1 after a power failure of two or more supplies
• Only occurs on input 1, input 2 does not exhibit the problem
• 6KW AC power supplies starting with “DTM” are not effected
• This has been seen in power grid redundancy testing when the power grid feeding input 1 of two or more power supplies is shut down so that the power supplies run off of power from input 2 only
• When the power grid for input 1 is restored again a power supply might shut down momentarily and then recover
• Resolution: Replace the existing PS unit
• DDTS: CSCtt38629
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
• This can take down the entire data center for if all of the Nexus 7000’s were reloaded at the same time. This may come into effect after approximately 3 months of uptime of an active supervisor
• Irrespective of any feature turned on/off
• Workaround: Reload of the active supervisor will clear the issue in a setup with two supervisor cards. Reload of the switch will clear the issue in a setup with a single supervisor
• Resolved: 5.1(4) or 5.2(1) or later
• DDTS: CSCtq62339
%PLATFORM-2-MEMORY_ALERT: Memory Status Alert: MINOR. Usage 85% of Available Memory%PLATFORM-2-MEMORY_ALERT: Memory Status Alert: SEVERE. Usage 90% of Available Memory
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
• ISSU upgrade from earlier image to 5.2(1)-(6) then ISSU to 5.2(7) or later
• This may result in degraded performance with the following error messages and possibly an ipfib process crash
• Workaround1: Configure the following regardless if LISP is in use
• Workaround2: If the issue is already hit, reload the effected modules
• DDTS: CSCub96980
%IPFIB-SLOT2-2-FIB_TCAM_HA_ERROR: FIB recovery errors, please capture 'show tech forwarding l3 unicast' and 'show tech forwarding l3 multicast'
feature lispConfigure "ip lisp etr" for all vrfs followed by "no ip lisp etr"no feature lisp
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 30
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
• Nexus 7000 Product Page on CCO:http://www.cisco.com/en/US/products/ps9402/index.html
• Nexus 7000 White Papers:http://www.cisco.com/en/US/products/ps9402/prod_white_papers_list.html
• Nexus 7000 Data Sheets:http://www.cisco.com/en/US/products/ps9402/products_data_sheets_list.htm
• Nexus 7000 Presentations: http://www.cisco.com/en/US/products/ps9402/prod_presentation_list.html
• Nexus 7000 Recommended NX-OS:http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/recommended_releases/recommended_nx-os_releases.html
• Nexus 7000 Scalability Guide:http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
• Nexus 3000 Basic Information• Nexus 3000 Important Caveats / Hot Issues• Nexus 3000 Best Practices• Nexus 3000 References
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 33
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
3548 3064X 3064T 3016Q 3048
Algorithm Boost Yes No No No No
Switch Capacity 960 Gbps 1.28 Tbps 1.28 Tbps 1.28Tbps 176 Gbps
Interface Type 48 SFP+ 28 SFP+ and 4 QSFP+
48 RJ45 and 4 QSFP+
16 QSFP+ 48 RJ45 and 4 SFP+
Max 1 GE ports 48 48 48 48 48
Max 10 GE ports 48 Up to 64 Up to 64 Up to 64 4
Max 40 GE ports 0 Up to 4 4 Up to 16 0
Switch Latency < 250 nsec < 1 usec 3-4 usec < 1 usec 2-8 usec
Line-rate on all ports (for L2 and L3 traffic)
Yes Yes Yes Yes Yes
Rack Unit 1 1 1 1 1
Hot-swappable PSU and Fan ?
Yes Yes Yes Yes Yes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
3016/3048/3064 3064-T 3548
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Layer 2802.1w, 802.1s, RPVST+, Root Guard, Uplink Guard, Bridge Assurance, PortFast, CDP, UDLD, PVLANs, IGMP Snooping, 802.1Q trunks, Port-
Channel, LACP, SVI, SPAN, Jumbo Frames, NTP
Management/Security
DHCP snooping, DAI, Radius, Tacacs+, AAA, CallHome, SSHv1/V2, telnet, IPv4 & IPv6 mgmt, SNMP MIBs, Traps, EthAnalyzer, RBAC, syslog, core dump, RMON, first-setup script, accounting log
System/Operations POST, OHMS, OBFL
ACL/QOSPACLs, VACLs, RACLs, Session based ACLs, ACL based QOS (CoS/DSCP marking), egress Bandwidth Limiting, 802.1p priority, strict priority scheduling, Tail Drop, ECN, WRED, Storm Control (broadcast, multicast)
Layer 3L3 Physical & SVI routed interfaces, static routing, RIP-v2, OSPF-v2, OSPF
fast convergence, EIGRP-IPv4, BGP, ECMP, IGMP v1/v2/v3, MSDP, PIM-v2 for IPv4, PIM-SSM for IPv4, HSRP, VRRP, VRF-lite, SPAN for L3 interfaces
Note: 5.0.3 do NOT support IPv6, OSPFv3, BFD and PBR
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
Feature Details
L3 Interfaces L3 Physical, SVI, Port‐Channel, Sub‐Interface
IPv4 Routing Protocol RIPv2,OSPF, EIGRP,BGPv4
Multicast PIM‐SM, SSM, MSDP, IGMP v1‐3, IGMP Snooping
HSRP/VRRP Yes
ECMP Yes (32‐way)
VRF Lite Yes
L3 SPAN Yes
uRPF – Strict & Loose mode Yes
Layer 2 CDP, UDLD, PVLANs, 802.1Q trunks, NTP, LACP
Spanning Tree & Extensions 802.1w, 802.1s, RPVST+, Root Guard, Loop Guard, BPDU Guard, Bridge Assurance, PortFast
PVLAN Trunks No
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Feature Details
Traffic Storm Control Broadcast, Multicast, Unknown‐Unicast
System Management RBAC, Online Diag, SysLog, Call Home, SNMP, RMON, SPAN
Management Security AAA,RADIUS, TACACS+, SSHv1/v2, Telnet,IPv4/IPv6 Management
Security PACL, VACL, RACL, DHCP Snooping, DAI, IPSG, ACL on VTY
QOS Cos /DSCP Marking, Egress Bandwidth Limiting, Strict Priority Scheduling, WRR, WRED, ECN
DCB 802.1p, ETS
Jumbo Frames Yes (9216 Bytes)
MTU Per System
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
• Licensing scheme identical to Nexus 5000’s
• Customers must buy and install both licenses for full L3 support
Do “show license usage” to see the license in use.
Base (N3K-BAS1K9)Basic L3 features Inter‐VLAN routing, Static routes, RIPv2, ACLs, OSPFv2
(limited to 256 routes), EIGRP stub, HSRP, VRRP and uRPFIP Multicast PIM SM, SSM, MSDP
LAN Enterprise (N3K-LAN1K9)Advanced IPRouting
OSPFv2, EIGRP, BGP and VRF‐Lite
System Default (no PID)Comprehensive L2 feature set
vPC, VLAN, 802.1Q Trunking, LACP, UDLD (Std. and Aggressive), MSTP, RSTP, STP Guards, VTP Transparent
Security AAA, DHCP Snooping, Storm Control, PVLAN, CoPP
Management PTP, ERSPAN, DCNM support, Console, SSHv2 access, CDP, SNMP, Syslog
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 40
vPC peer—a vPC switch, one of a pair
vPC member port—one of a set of ports (port channels) that form a vPC
vPC—the combined port channel between the vPC peers and the downstream device
vPC peer-link—link used to synchronize state between vPC peer devices, must be 10GbEvPC
connected device
vPC member
port
vPCvPC
member port
vPC peer-link
non-vPC connected
device
vPC peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
vPC peer-keepalive link—the keepalive link between vPC peer devices
vPC VLAN—one of the VLANs carried over the peer-link and used to communicate via vPC with a peer device
non-vPC VLAN—One of the STP VLANs not carried over the peer-link
CFS—Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices
vPC Peer-keepalive link
CFS protocol
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 42
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
“show interface X” results indicate that “input discard” counter increments when PIM, EIGRP and other control-plane packets are received.
This counter increments as these packets are redirected to the CPU by an ACL entry, and stop forwarding them to other front-panel ports.
N3K# show int eth 1/1Ethernet1/1 is up<snip>
RX5 unicast packets 3714544619 multicast packets 0 broadcast packets3714544625 input packets 475461709100 bytes<snip>0 input with dribble 3714544619 input discard
No workaround available. Issue is resolved in 5.0(3)U2(1) and later releases.
Ref bug ID: CSCto53539
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
“show hardware internal indiscard-stats front-port X” results shows all the counters as zero, which is incorrect.
“show interface X” shows non-zero indiscard counter valueNexus3000-1# show interface Ethernet 1/7Ethernet1/7 is up <output omitted>0 input with dribble 33844 input discard(includes ACL drops)<output omitted>N3K# show hardware internal interface indiscard-stats front-port 7+-----------------------------------------+-----------------+-----+| Counter Description | Count | Last Increment | +----------------------------------+-------------+----------------+IPv4 Discards 0 0STP Discards 0 0Policy Discards 0 0ACL Drops 0 0Receive Drops 0 0Vlan Discards 0 0+-----------------------------------+------------+----------------+
Issue is resolved in 5.0(3)U3(1) and later releases. Reference bug ID: CSCtu29771
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 45
In a High-Performance-Trading (HPT) setup, users may experience gap in the multicast streams.
The Nexus3000 switch receives lots of IGMP leaves and joins, and none of them dropped by CoPP (control-plane policing).
NX-OS release 5.0(3)U2(2) and later has improved performance.
Reference bug ID: CSCtt18984
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
After Nexus 3016 and 3064 is upgraded to 5.0(3)U3(1) releases, the switch may report:
%NOHMS-2-NOHMS_ENV_ERR_FAN_SPEED: System minor alarm in fan tray 1: fan speed is out of range on fan 4. 2495 to 12600 rpm expected. 2457 rpm read
%NOHMS-2-NOHMS_ENV_ERR_FAN_SPEED: System minor alarm in fan tray 1: fan speed is out of range on fan 8. 2200 to 12600 rpm expected. 1944 rpm read
This is surely not an hardware issue.
Issue is fixed in 5.0(3)U3(2) and later releases.
Reference bug ID: CSCty64730
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
Nexus3000 reports following messages when a parity error is detected:
%USER-3-SYSTEM_MSG: bcm_usd_isr_switch_event_cb:431: slot_num0, event 2, memory error type 0x1, mem addr 0x5f36, cause bit <addr> -bcm_usd
Switch needs to be reloaded to recover from this situation, otherwise the device may have functional impact.
Switch upgraded to 5.0(3)U5(1a) can detect and correct single bit parity errors.
Reference bug ID: CSCtw75636
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
Nexus3048 data sheet indicates that it supports up to 16K host entries.http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps11541/data_sheet_c78-685363.html
But, the “show hardware profile status” supports only 8K host entries.N3K# show hardware profile status Total LPM Entries = 16383 Total Host Entries = 8192Reserved LPM Entries = 1024 Max Host Limit Entries = 4096 Max Host6 Limit Entries = 0Max Mcast Limit Entries = 4000 Used LPM Entries (Total) = 3
This issue is reported in 5.0(3)U5(1b) and resolved in 5.0(3)U5(1c).
Reference bug ID: CSCug25153
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49
Symptoms: Nexus3048 uplink ports are showing down.
This issue is seen when the ports are set to 1000 Mbps and Auto-negotiation enabled.
N3K# show running int e1/49interface Ethernet1/49speed 1000negotiate auto
N3K# show interface statusEth1/49 1 eth access down Link not connected
Issue is resolved in 5.0(3)U4(1) and later releases. As a workaround use “no negotiate auto” and “auto nonegotiate” commands, as applicable.
Reference big IDs: CSCty91237 and CSCtu68315
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50
BFD hap resets intermittently.
This issue is seen when BFD sessions are configured over eBGP session.
Issue is resolved in 5.0(3)U2(2d) and later releases.
Reference bug ID: CSCts95614
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 51
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 52
For Nexus 3016, 3048, 3064 and 3064-T the recommended releases are 5.0(3)U5(1) or later, and minimum recommended release is 5.0(3)U4(1).
Please review the Release-notes before upgrading the NX-OS release: http://www.cisco.com/en/US/products/ps11541/prod_release_notes_list.html
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 53
N3k‐2N3k‐1
No parallel link between vPC peers – e.g., as an uplink backup Any parallel link set up should be:
A peer keep-alive link A trunk/channel carrying non-vPC vlans
No L2 / L3 links or port-channels parallel to vPC peer-link
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 54
No router(s) behind vPC
Switch
Unicast traffic works Multicast would not work with this topology, as do not sync PIM states
between vPC-peers. Same restriction applicable to IGP protocols.
N3K-1 N3K-2N3K-1 N3K-2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 55
L2
L3
N3k‐1 N3k‐2
Router
L3L3
L2
L3N3k‐1 N3k‐2
Router
L2L2
L2
L3N3k‐1 N3k‐2
Router
VPC
✔
Not recommended / supported topology Multicast will have the issue with PIM over L3 vPC
No L3 vPC to uplink Router
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 56
How HSRP works ?
With vPC both HSRP active and standby can forward traffic.
The HSRP MAC is programmed in such a way that is L3 switched only if HSRP is in active/standby pair
Take Away is …
HSRP aggressive timers are not useful in a vPC topology
HSRP preempt-delay is not useful in a vPC topology
vPC
HSRP active HSRP standby
HSRP aggressive timers and preempt-delay are not useful
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 57
Have backup route over peer-link by having IGP peering
However, avoid unnecessary peering on all available vlans (over vPC peer-link), by using “passive” command
Have peering over only single or few VLAN
IGP peering
Have backup IGP path
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 58
vPCPrimary
vPCSecondary
L3L2
OSPF
N3K(config-vpc-domain)# delay restore ?<1-3600> Delay in bringing up vPC links (in seconds)N3K(config-vpc-domain)# delay restore 360
vPC interaction with Routing convergence on system restart
After a vPC device reloads and come back up routing protocols need time to reconverge
vPCs may black-hole routed traffic from access to core until layer 3 connectivity is reestablished
Tune vPC Delay Restore to avoid the traffic drop when device comes up
Fine-tune vPC delay-restore timer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 59
N3k‐1 N3k‐2
PIM-RP
L3L3
L3 Link
L2 Link
PIM-RP
For better use of Peer-link bandwidth Better Convergence in case of failure due to pre-built SPT
vPC peers have equal cost to RP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 60
When several hundreds of of servers joining multiple multicast groups at the same time (market opens), it generates a burst of IGMP traffic to N3K CPU/Control-Plane. Default CoPP rate-limiter for IGMP (400 pps) may drop to protect the CPU, and end-users may see delay in receiving data streams.
Also, due to the default MRT (10 sec) there is a possibility of IGMP traffic burst towards the CPU, when the network has several hundred hosts.
To handle the IGMP bursts ….
Increase the default maximum-response-time to 25 seconds
Increase the CoPP rate-limiter for IGMP to 600 pps
N3K(config)#interface vlan 101N3K(config-vlan)# ip igmmp query-max-response-time 25N3K(config-vlan)# ip igmp last-member-query-response-time 25
Increase CoPP limit and max-response-time (MRT) for IGMP
policy-map type control-plane copp-system-policyclass copp-s-igmp
police pps 600
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 61
Supported Topology
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 62
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 63
Nexus3000 Product Page on CCO: http://www.cisco.com/en/US/products/ps11541/index.html
Nexus3000 White Papers:http://www.cisco.com/en/US/products/ps11541/prod_white_papers_list.html
Nexus3000 Data Sheets: http://www.cisco.com/en/US/products/ps11541/products_data_sheets_list.html
Nexus3000 Presentations: http://www.cisco.com/en/US/products/ps11541/prod_presentation_list.html
Nexus3548 – Product Page and Algo Boost:http://www.cisco.com/en/US/products/ps12581/index.html
Understanding Nexus 3000 Switch Latency: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps11541/white_paper_c11-661939.html
Understanding “Input Discard” Interface Counter in Nexus3000:https://supportforums.cisco.com/docs/DOC-23994
Thank you.Thank you.