Troubleshooting Cisco Catalyst 4500 Series Switchesd2zmdbbm9feqrf.cloudfront.net/2015/usa/pdf/BRKCRS-3142.pdf · –problems are solved faster when knowns can be eliminated ... (i.e.GRE)

Troubleshooting Cisco Catalyst 4500 Series

SwitchesSubhash Ramanathan – Escalation Engineer, Enterprise

Campus Switching GroupBRKCRS-3142

Session Goals

At the end of this session, you should be able to:

Understand system resources and monitor their usage

Identify all areas of packet loss

Trace hardware packet path

Make use of newer tools

This content is based on questions we see in the field. Feedback is welcome!

Agenda

Products Overview

Troubleshooting

– Method

– System Resources

– Packet path / loss

– Wired/Wireless Convergence

Tools/Tips

Products Overview

4503-E 4507R+E 4510R+E 4506-E

See the appendix for supervisor, line card, and chassis product and compatibility details.

48 Gbps per slot

• +E Chassis support 12.2(53)SG4 onward

• Sup8E, Sup7E, Sup7L-E, 47xx line card

• 4507R+E, 4510R+E, 4503-E, 4506-E

Products Overview

1. Ternary Content Addressable Memory

2. Integrated on Supervisor 7E, 7L-E, Sup8E

Intelligent SupervisorsSupervisor Engine 8-E, 7-E, 7L-E, 6-E, 6L-E

Transparent Line Cards

Wire-rate, oversubscribed, PoE

10/100, 10/100/1000, GE, 10GE

Various physical media front panel ports

Dedicated per-slot bandwidth to supervisor

Switching ASICs

Packet Processor

Forwarding Engine

Specialized Hardware

TCAM1s for ACLs, QoS, L3 forwarding

NetFlow2 (NFE) for statistics gathering

Shared Packet Memory

Line Card

Stub ASICs

Front Panel Ports

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

Products Overview

1. Ternary Content Addressable Memory

2. Integrated on Supervisor 7E, 7L-E, Sup8E

Intelligent SupervisorsSupervisor Engine 8-E, 7-E, 7L-E, 6-E, 6L-E

Transparent Line Cards

Wire-rate, oversubscribed, PoE

10/100, 10/100/1000, GE, 10GE

Various physical media front panel ports

Dedicated per-slot bandwidth to supervisor

Switching ASICs

Packet Processor

Forwarding Engine

Specialized Hardware

TCAM1s for ACLs, QoS, L3 forwarding

NetFlow2 (NFE) for statistics gathering


Line Card

Stub ASICs

Front Panel Ports

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

Wireless DC

Agenda

Products Overview

Troubleshooting

– Method



– Wired/Wireless Convergence

Tools/Tips

Troubleshooting Method• General Recommendations

Design with intent

– ideally, create a deterministic network

– engineers – not traffic – should control the network

Baseline, monitor against baseline, alarm and/or adjust

– problems are solved faster when knowns can be eliminated

Characterize issues quickly with a plan

Troubleshooting MethodMethod

1. Define Problem

2. Gather Facts

3. Consider Possibilities

4. Create Action Plan

5. Execute Action Plan

6. Observe Results

Do

cu

me

nta

tio

n

Symptoms? System Messages? User

Input?

When? Frequency? Impact? Scope?

• Need to have a good understanding about

how the system looks like when it is

healthy

Want to learn more? Check out CCNP Practical Studies: Troubleshooting by Donna Harrington.

CCNP TSHOOT 642-832 Official Certification Guide by Kevin Wallace.


Category Possible Cause

Config/Design Mis-configuration

Reaching Capacity

Traffic DOS Attack

Traffic Pattern Change

Bad peer/server

Software Issue Software Limitation

Bug

Hardware Issue Hardware Limitation

Failed Hardware

Transient Hardware Issue

1. Define Problem

2. Gather Facts




6. Observe Results

Do

cu

me

nta

tio

n


1. Define Problem

2. Gather Facts




6. Observe Results

Do

cu

me

nta

tio

n

What needs to be done to isolate each

potential root cause? Make a change, measure results,

rollback change if problem persists

Problem solved? If not, continue

action plan

Troubleshooting MethodCaution

debug and show platform commands to follow

Excessive debug output to console may disable switch

show platform commands are intended for in-depth troubleshooting

Use debug and show platform commands only when advised by TAC

show platform CLIs are not officially supported IOS commands

Agenda

Products Overview

Troubleshooting

– Method



– Wired/wireless Convergence

Tools/Tips

System ResourcesCPU

• Linux based Operating System IOS-XE

• Runs IOS tasks

• Runs 4500 platform-specific jobs

• Sends/Receives control traffic

• Software-switches packets that can’t be hardware-switch

• Elevated CPU == in-use CPU, does not impact data plane


Line Card

Stub ASICs

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

IOS-XE

IOSd

Cat4500 jobs

Troubleshooting CPU: show process cpu

CPU higher than baseline

High iosd use on IOS-XE?

sh proc cpu detail

process iosd

Reference Document ID: 65591 on

http://www.cisco.com for more

detailsHigh CPU in IOS process or

Cat4k process?

Troubleshoot features related

to the process / open TAC SR

No

Yes

High CPU traffic driven?

(K5CpuMan Review)

show platform health

ios cat4k

Can the traffic be identified?

show platform cpu packet stat

No

Yes

Stop / alter traffic source,

open TAC SR if more detail

needed

monitor session 1 source cpu

OR

debug platform packet all buffer

show platform cpu packet buffer

NoYes

IOS-XE

IOS

http://www.cisco.com/

Troubleshooting CPU: Narrowing Down Process

Switch#show proc cpu sort

Core 0: CPU utilization for five seconds: 9%; one minute: 8%; five minutes: 8%




PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

6989 3788661 13695505 309 7.21 6.73 6.73 0 iosd

6984 677640 11354599 5 0.06 0.06 0.06 0 wcm

6936 211124 554564 380 0.02 0.02 0.02 0 cpumemd

6936 211124 554564 380 0.02 0.02 0.02 0 cpumemd

11879 56609 555338 101 0.02 0.01 0.01 0 wnweb_paster.py

Quad Core

IOS-XE processes

Identify which process running high

Troubleshooting CPU: Narrowing Down Process

switch# show proc cpu detail process iosd sort

Switch#show process cpu detailed process iosd sorted





PID T C TID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

(%) (%) (%)

6989 L 3884781 1372616 309 4.15 6.64 6.82 0 iosd

6989 L 1 6989 2797345 7018102 0 6.76 6.57 6.76 0 iosd

6989 L 0 10677 1065764 6669769 0 0.04 0.08 0.05 0 iosd.fastpath

6989 L 1 10678 19185 119427 0 0.00 0.01 0.01 0 CMI Thread

6989 L 0 10679 3288 261952 0 0.00 0.00 0.00 0 iosd.monitor

6989 L 3 10680 86 4203 0 0.00 0.00 0.00 34816 iosd.aux

123 I 3816054 2388033 0 7.66 7.99 8.99 0 Cat4k Mgmt LoPri

122 I 2256302 4346590 0 6.88 5.99 5.88 0 Cat4k Mgmt HiPri

Catalyst-4k Specific Management Tasks

Troubleshooting CPU: Packet-Driven CPU

switch# show platform health

…

%CPU %CPU RunTimeMax Priority Average %CPU Total

Target Actual Target Actual Fg Bg 5Sec Min Hour CPU

K5CpuMan Review 30.00 70.81 30 17 100 500 91 66 9 19:17

…

Switch# show platform cpu packet statistics

…

Packets Dropped by Packet Queue

Queue Total 5 sec avg 1 min avg 5 min avg 1 hour avg

---------------------- --------------- --------- --------- --------- ----------

Ip Option 10715071 118803 71866 15919 0

…

(config)# monitor session 1 source cpu rx

(config)# monitor session 1 destination interface Gi1/48

K5CpuMan Over Target

Recent flood of packets with IP Options (not HW routable)

If port is available, get a full capture from CPU

Troubleshooting CPU: SPAN not available?

switch# debug platform packet all buffer

platform packet debugging is on

Switch# show platform cpu packet buffered

Total Received Packets Buffered: 1024

-------------------------------------

Index 0:

3 days 23:23:18:54927 - RxVlan: 1006, RxPort: Gi1/1

Priority: Normal, Tag: No Tag, Event: 11, Flags: 0x40, Size: 64

Eth: Src 00:00:0B:00:00:00 Dst 00:22:90:E0:D6:FF Type/Len 0x0800

Ip: ver:IpVersion4 len:24 tos:0 totLen:46 id:0 fragOffset:0 ttl:64 proto:tcp

src: 10.10.10.100 dst: 172.16.100.100 hasIpOptions firstFragment lastFragment

Remaining data:

0: 0x0 0x64 0x0 0x64 0x0 0x0 0x0 0x0 0x0 0x0

10: 0x0 0x0 0x50 0x0 0x0 0x0 0x8A 0x37 0x0 0x0

20: 0x0 0x1 0xB5 0x77 0x6A 0x7E

• This debug does not require significant CPU overhead

• Be sure to use “buffer” and not “log”

Newer versions provide human-readable event

Decode on older versions with:

switch# show platform software cpu events | i Code|11

CPU Event Code PE-Q

1 2 Ip Option 11 17

Troubleshooting CPU: Common Punt ReasonsCommon Cause Recommended Solution

Same interface forwarding no ip redirect, or alter topology

ACL logging disable ACL logging, use ACL matching stats or netflow

ACL deny causing switch to send

ICMP unreachable

no ip unreachables2

Forwarding/Feature exception (out of

TCAM/adj space)

reduce TCAM usage

resize TCAM region (TCAM2/3)

SW-supported feature (i.e.GRE) disable the feature or reduce the amount of traffic

IP packets with TTL<2, IP options disable the offending traffic, regulate source with Control Plane Policing1

Unexpected control/data traffic Control Plane Policing1

1.CoPP supported on all legacy supervisors starting 12.2(31)SG, SUP6-E/6L-E /4900M/4948E on 12.2(50)SG , all Sup8E/7E/7L-E/4500X

2.Must be configured on all the L3 interfaces of the switch

System ResourcesMemory

• Leak vs Large Usage

• Large usage goes away when condition is no longer present

• Leak never decreases

• Establish baseline

• Collect multiple iterations over recorded interval

• Correlate increase with any known activity

Troubleshooting Memory: Large Usage

switch# sh authentication session | count Runn

Number of lines which match regexp = 239

switch# sh proc mem detail proc iosd sort | i Hold|Auth Manager

PID TTY Allocated Freed Holding Getbufs Retbufs Process

113 0 870624 125992 837216 0 0 Auth Manager

switch(config)# int ra gi 1/1 - 48 , gi 2/1 - 48 , gi 3/1 - 48 , gi 4/1 - 48

switch(config-if-range)# shut

switch(config-if-range)# int ra gi 7/1 - 48 , gi 8/1 - 48 , gi 9/1 - 48 , gi 10/1 - 48

switch(config-if-range)# shut

switch(config-if-range)# end

switch# sh authentication session | count Runn

Number of lines which match regexp = 0

switch# sh proc mem detail proc iosd sort | i Auth Manager

147 0 1434488 601760 514088 0 0 Auth Manager

300Kb not leaked, simply used

Troubleshooting Memoryswitch# show proc mem sort

System memory : 3870600K total, 1250447K used, 2620153K free, 323704K kernel reserved

Lowest(b) : 2031687704

PID Text Data Stack Heap RSS Total Process

6989 152256 943268 100 680 1338232 1435556 iosd

6984 20464 580524 88 14140 179240 721848 wcm

6985 692744 22048 92 164 50028 98096 mgmte_tap

6956 112 93740 88 5200 48484 134924 cli_agent

switch# show proc mem detail proc iosd sort

Processor Pool Total: 805306368 Used: 645097888 Free: 160208480

I/O Pool Total: 20971520 Used: 361576 Free: 20609944

Critical Pool Total: 4087852 Used: 40 Free: 4087812

Critical Pool Total: 106460 Used: 40 Free: 106420

PID TTY Allocated Freed Holding Getbufs Retbufs Process

153 0 1461539184 749742680 307884712 14266252 0 Auth Manager

0 0 304511544 14111208 272960272 0 0 *Init*

185 0 887586464 301222848 31368752 0 0 CDP Protocol

switch# show proc mem detail proc iosd task 153

Process ID: 153

Process Name: Auth Manager

Total Memory Held: 307882352 bytes

Processor memory Holding = 307882352 bytes

pc = 0x16FCD45C, size = 291258544, count = 4441

For Classic IOS, use:

• show process mem sort

• show process mem <pid>

Auth Manager holding too much

Collect process memory breakdown for TAC

System ResourcesTCAM

• Check TCAM usage for ACLs, security, L3 routes, PBR, DHCP Snoop, IPSG,

WCCPv2

%C4K_HWACLMAN-4-ACLHWPROGERR: Input VOIP_FROM_CE_IPv6 -

hardware TCAM limit, qos being disabled on relevant interface

%C4K_HWACLMAN-4-ACLHWPROGERR: Input Security: 101 - hardware

TCAM limit, some packet processing will be software switched

C4K_HWACLMAN-4-ACLHWPROGERRREASON: Input(75/Normal, 1/Normal)

Invalid Acl-based Feature - hardware TCAM policers exceeded


Line Card

Stub ASICs

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

Monitoring TCAMswitch# show platform hardware acl statistics utilization brief

Switch#show platform hardware acl statistics utilization brief

CAM Utilization Statistics

--------------------------

Used Free Total

--------------------------------

Input Security (160) 38 (1 %) 2010 (99 %) 2048

Input Security (320) 34 (1 %) 2014 (99 %) 2048

Input Qos (160) 15 (0 %) 2033 (100%) 2048

Input Qos (320) 8 (0 %) 2040 (100%) 2048

Input Forwarding (160) 7 (0 %) 2041 (100%) 2048

Input Forwarding (320) 24 (1 %) 2024 (99 %) 2048

Input Unallocated (160) 0 (0 %) 53248 (100%) 53248

switch# show platform hardware qos policer utilization

-------------------------------------------

Policer utilization summary:

Direction Assigned Used Free

-------------------------------------------

Input 2048 ( 12.5%) 4 ( 0.1%) 2044 ( 99.8%)

Output 2048 ( 12.5%) 1 ( 0.0%) 2047 ( 99.9%)

Free 12288( 75.0%) 0 ( 0.0%) 12288(100.0%)

Low utilization

System ResourcesTransmit Queue Memory

• Reserved queue memory for each linecard, exceeding this eats into global pool

• When global pool exhausted, the above message appears

• Options:

• decrease queue depths on a per port basis

• combine classes under the same queue

%C4K_HWPORTMAN-3-TXQUEALLOCFAILED: Failed to allocate the needed queue entries for Gi6/13

Monitoring Queue Memory

Entry Sup6-E/6L-E/7L-E Sup8E

Total queue memory 512K 1M

Free Reserve: global pool 100K 50K

CPU, recirc, drop queues 20K 40K

Queue entries per slot1 x = 400K/ nSlots2 x = 910K/nSlots

Queue entries per port on a line card y = x / nPorts3 y = x/nPorts

Queue entries per class transmit queue z = y/nTxQs4 z = y/nTxQs

1. In a redundant chassis, two supervisor slots are treated as one2. nSlots – number of Slots3. nPorts – number of Ports in a line card4. nTxQs – number of transmit queues in use

Monitoring Queue Memory

switch# show platform software qm

Drop port Tx Queue allocations (Size: 8184, Base: 0x019008)

CPU Subport Tx Queue allocations (TotalSize: 12304)

Recirc Subport Tx Queue allocations(TotalSize: 12288)

…

Global TX Queue reservations

----------------------------

Slot Size Base Addr Current Unused

Addr Entries

---- ---- --------- ------- -------

0 101488 0x021010 0x021010 101488

1 101488 0x039C80 0x039C80 101488

2 101488 0x0528F0 0x06B550 16

3 101488 0x06B560 0x06B560 101488

4 101488 0x0841D0 0x096B00 25408

5 101488 0x09CE40 0x09CE40 101488

6 101488 0x0B5AB0 0x0B5AB0 101488

7 101488 0x0CE720 0x0CE720 101488

8 101488 0x0E7390 0x0E7390 101488

• 101488 / 48 = 2114 entries/port

• >2114 entries will eat into global pool

Drop, Recirc, CPU reservations

Troubleshooting System Resources CommandsCLI Purpose

List IOS process CPU % on IOS-XE show proc cpu detail process iosd sort

Monitor Cat4k platform CPU statistics show platform health

show platform cpu packet statistics

SPAN packets to/from CPU monitor session 1 source cpu

monitor session 1 destination interface <int>

Enable/monitor Cat4k CPU buffer debug platform packet all buffer

show platform cpu packet buffered

Display process memory and buffer

holdings

show proc mem sort

show process mem <pid>

show buffers

Display process memory and buffer

holdings on IOS-XE

show proc mem detail proc iosd sort

show proc mem detail proc iosd task <pid>

show buffers detailed process iosd

Display Cat4k ACL and policer usage show platform hardware acl statistics utilization brief

show platform hardware qos policer utilization

Display Cat4k queue memory usage show platform software qm

Agenda

Products Overview

Troubleshooting

– Method




Tools/Tips

Troubleshooting Packet Loss / Path• Why is any packet sent to port(s), to CPU, or dropped?

Losing packets on the 4k without a clue why?

1. Collect “show tech” and iterations of the below

2. Step through the platform

1. Identify counters outside of baseline, find an explanation based on counter meaning

2. Identify unexpected platform programming, work upwards

• incrementing counters are most useful

• Some counters are normal

• Baseline data is useful

Areas Of Investigation

HW-based

checks

Queue/buffer

failure

PHY, stub, packet

processor, forwarding

engine

show interfaces <int> counters all

show platform hardware interf <int> statis

show platform software interf <int> statis

show platform software interf <int> stub statis

show platform software interf <int> stub cts statis all

show platform hardware ret rrq

show platform software drop-port

CPU queues CPU controller show platform cpu packet driver

show platform cpu packet statistics

STP L2 lookup show platform hardware stp vlan <vlan>

L3 entries forwarding lookup show platform hardware ip route [ipv4|ipv6] network <net> <mask>

show platform hardware ip route [ipv4|ipv6] host <ip or group>

ACL input classification,

output classification

show access-list <*acl>

show platform hardware acl input entries static

show platform hardware acl [input|output] entries interface <int> all

show platform hardware acl [input|output] entries vlan <vlan> all

show platform hardware acl [input|output] actions <action>

L2 entries,

floodsets

L2 lookup show plat hard mac add <mac>

show plat hard ret chain index <index>

show platform hardware floodset vlan <vlan>

Troubleshooting Packet Loss / PathPHY and Stub ASIC

Line Card

Stub ASICs

Front Panel Ports

Supervisor

Layer 1 issues

Malformed frames/packets

Oversubscription

Flow-control

Storm-control

Troubleshooting Packet Loss / PathLayer 1 Issues

• Match speed and duplex

• Isolate bad hardware using known good hardware

• Specific to end device? Patch/line cord? Front panel port? Linecard?

• Exclude patch panel if possible

• Peer misbehaving? Sniff wire for malformed frames

switch# show interfaces g5/5 count errors | exclude \ 0\ *0\ *0\ *0

Port CrcAlign-Err Dropped-Bad-Pkts Collisions Symbol-Err

Gi5/5 23736730 0 0 0

Port Undersize Oversize Fragments Jabbers

Port Single-Col Multi-Col Late-Col Excess-Col

Port Deferred-Col False-Car Carri-Sen Sequence-ErrSee Appendix for Error descriptions

Switch#test cable-diagnostics tdr interface gigabitEthernet 3/1

Switch#show cable-diagnostics tdr interface gigabitEthernet 3/1

Interface Speed Local pair Cable length Remote channel Status

Gi3/1 1Gbps 1-2 0 m Unknown Terminated

TDR


switch# show platform software interface gigabitEthernet 1/1 stub statistics

XgstubMan(0:N-0)Port( 1 ) Rx Stats:

…

OverrunPackets : 0

AlignmentErrorPackets : 0

FcsErrorPackets : 0

SymbolErrorPackets : 0

InvalidOversizePackets : 0

Ipv4HdrChecksumErrorPackets : 0

Ipv4HdrErrorPackets : 0

Ipv6HdrErrorPackets : 0

…

switch# show platform software interface gigabitEthernet 1/1 statistics

Superport8(Gi1/1-6) Non-Zero Software Statistics

…

RxSequenceErrors : 255

RxSymbolErrors : 255

Note: counters may increment during plug / unplug

Platform commands can narrow down stub

ASIC vs packet processor


(config)# logging event link-status global

(config-if)# logging event link-status

switch# show platform software interface all | inc downs:|PimPhyport

…

GalGlmPort(0:N/21), Active? : true, PimPhyport Name : Gi1/22, EpmPortMan Name : EpmPortMan(0:N/21)

Name( EpmPortMan(0:N/21) ), PimPhyport name( Gi1/22 )

#link downs: 41712

switch# show platform software interface gi1/1 mii

…

0x00 ControlReg 0x1140

0x01 StatusReg 0x79C9

…

0x04 AutoNegAdvReg 0x01E1

0x05 AutoNegLinkPartnerAbilityReg 0x0000

0x06 AutoNegExpansionReg 0x0064

0x07 AutoNegNextPageTransmitReg 0x2001

…

0x09 1000BaseTControlReg 0x0F00

0x0A 1000BaseTStatusReg 0x0000

Monitor for link flap via syslog

Configurable globally or per-interface

Get total number of flaps since switch boot

Compare with switch uptime

This command should be run twice

Use the second results, decode standard

802.3 registers

Troubleshooting Packet Loss / PathOversubscription: stub/supervisor port buffers

completely even traffic flow does not occur in real-world

– 2:1 1Gbps != (real world) 500 Mbps x 2 ports

– 2:1 10bps != (real world) 5Gbps x 2 ports

ingress traffic on oversubscribed ports

– control on the peer device

egress oversubscription

– consider multi-path

max

avg

min

Troubleshooting Packet Loss / PathFlow control

Switch may send pause toward end-device if rx buffer passes high watermark

Stub will pause toward supervisor if end-device signals pause

Stub ASICs

Front Panel Ports

Pause

Packet Processor

Pause

1

2

Drops31. Device sends pause to stub

2. Stub sends pause to packet processor

3. Packet processor pauses tx-queue

Troubleshooting Packet Loss / PathTx Oversubscription and Flow Control

switch# show interfaces g2/47 counters detail | begin Drops

Port Tx-Drops-Queue-5 Tx-Drops-Queue-6 Tx-Drops-Queue-7 Tx-Drops-Queue-8

Gi2/47 0 0 0 37748571

switch# show interfaces g2/47 counters detail | begin RxPause

Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames PauseFramesDrop

Gi2/47 0 130 0 0

Tx oversubscription will result in tx-queue drops

Pause frames from a peer will stop tx-queue processing

Queue 8 is the default queue with no QoS Configured

Troubleshooting Packet Loss / PathRx Oversubscription

switch # show interface gi1/13 | include overrun

0 input errors, 0 CRC, 0 frame, 86432 overrun, 0 ignored

switch# show interface gi1/13 counter all | begin Rx-No

Port Rx-No-Pkt-Buff RxPauseFrames TxPauseFrames PauseFramesDrop

Gi1/13 206658 0 0 0

switch# show platform software interface g1/13 stub stat | in Overrun

OverrunPackets : 206658 (look for Rx Stats)

RxFifo stub overrun will be seen during Rx oversubscription

Packet buffer depletion can also cause Rx-No-Pkt-Buff

Troubleshooting Packet Loss / PathPacket Processor


Line Card

Supervisor

Packet Processor

Central packet memory exhaustion

Deep transmit queues

Egress oversubscription (example: SPAN)

Jumbo frames

%C4K_SWITCHINGENGINEMAN-4-IPPLLCINTERRUPTFREELISTBELOWHIPRIORITYTHRESHOLD: IPP

LLC freelistBelowHiPriorityThreshold interrupt FreeListCount: 2058,

lowestFreeCellCnt: 0

Troubleshooting Packet Loss / PathOversubscription: packet memory exhaustion

Deep buffers and congestion

Limited gain (temporary buffering)

Switch-global expense (ingress and egress)

1. Deep egress queue fills

2. Packet memory consumed

3. Packet memory unavailable for ingress

Packet Processor


Drops

Deep Q1

2

3

Full

Troubleshooting Packet Loss / PathOversubscription: packet memory exhaustion

Reduced buffers during congestion

Limited expense (smaller threshold on given interface)

Large gain (no packet memory exhaustion)

Other solutions:

Even out packet port distribution

Egress policers

Packet Processor


Drops

Restricted

Troubleshooting Packet Loss / PathPacket memory: keeping the FreeList healthy

switch# show platform hardware interface all | include FreeListCount

FreeListCount : 125062

switch# show platform hardware interface all | include FreeListCount

FreeListCount : 124904

switch# show interfaces g2/47 counters detail | begin Drops

Port Tx-Drops-Queue-5 Tx-Drops-Queue-6 Tx-Drops-Queue-7 Tx-Drops-Queue-8

Gi2/47 0 0 0 37748571

(config)# policy-map egress_queue_limit

class class-default

queue-limit 500

(config)# hw-module system max-queue-limit <value>

128K 256 byte cells in Sup8E, Sup7E, Sup7L-E

Drop in FreeList will accompany

IPP log message

1. Locate interfaces tail dropping

2. Reduce tx-queue size OR

3. Modify default queue size

Troubleshooting Packet Loss / PathForwarding ASIC

Line Card

Supervisor

NFE

CPU

TCAMs

Forwarding Engine

Stepping through forwarding ASIC stages

Identifying packet destiny

– Punt?

– Drop?

– Forward to where?

– Replicate to where?

Working backwards from ASIC counters

Packet Loss / Path: Forwarding ASIC

Location Purpose Most Common Platform Troubleshooting Need

IM Input mapping Vlan and port mapping

L2 L2 lookup Layer 2 destination

IC Input classification ACLs (especially static ACL, which evaluate *all* traffic)

For custom ACL, IOS-level CLI typically all that is needed

NF Netflow Platform troubleshooting not commonly required

IP Input policing IOS-level policer counters typically all that is needed

FL Forwarding lookup L3 Multicast replication

OC Output classification IOS-level CLI typically all that is needed

OP Output policing IOS-level policer counters typically all that is needed

OM Output mapping,

replication

Vlan re-mapping

Replication counters useful in very high density scenarios

QM Queueing Tx-queue programming

Packet Loss / Path: Input Mapping

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Physical / aggregate port mapping

Vlan mapping

switch# show platform mapping ports

Interface Superport Subport CompactSubportId PortSet Phyport Aggport PimPhyport

Gi1/1 8 1 20 2 13 8 0

…

Gi7/48 35 4 210 8 402 Po1(417) 367

switch# show platform hardware portvlan-map-table interface gigabitEthernet 1/1

Aggport( 8 ):

----- PortVlanDirectTable -----

VlanId FwdVlanId SrcMissCtrl TxDropEn VlanTagStripEnOnTx

0 0 SrcMissCopyToCpu False False

…

----- PortVlanHashTable -----

Index PartialAggport VlanId FwdVlanId Dir SrcMissCtrl TxDropEn VlanTagStripEnOnTx

1568 8 100 200 Rx SrcMissCopyToCpu - False

3188 8 100 200 Tx - False False

All ports on an Etherchannel share an Aggport

Vlan mapping in use

Mapping information used in many platform CLI outputs

Packet Loss / Path: Input Mapping / L2 Lookup

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Confirm if routing features are enabled on a vlan

switch# show platform hardware rxvlan-map-table vlan 902

Vlan 902:

l2LookupId: 902

srcMissIgnored: 0

ipv4UnicastEn: 1

ipv4MulticastEn: 1

ipv6UnicastEn: 0

ipv6MulticastEn: 0

…

switch# show int vl 902 | i SVI

Hardware is Ethernet SVI, address is 001e.f73f.f5bf (bia 001e.f73f.f5bf)

switch# show mac address-table vlan 902 | i 001e.f73f.f5bf

902 001e.f73f.f5bf static ip,ipx,assigned,other Switch

switch# show plat hard mac add 001e.f73f.f5bf vlan 902

…

Index Mac Address Vlan Type SinglePort/RetIndex/AdjIndex

----- -------------- ----- ---------- ----------------------------

63248 001E.F73F.F5BF 902 SinglePort Cpu aggport(4) ND RouterAddr

IPv4 unicast and multicast routing enabled

SVI MAC present in MAC

table

Packet Loss / Path: L2 Lookup

STP state check

SA Learning

switch# show span int gi 7/48 state | i VLAN0002

VLAN0002 forwarding

switch# show platform hardware stp vlan 2 | i Gi7/48

Gi7/48 (375) Forwarding

switch(config)# no mac address-table learning vlan 100

switch# show platform hardware rxvlan-map-table vlan 100 | i srcMiss

srcMissIgnored: 1

switch# show mac add int gi 1/46 | i 902

902 0000.0500.0000 dynamic ip,ipx,assigned,other GigabitEthernet1/46

902 ffff.ffff.ffff system Gi1/46,Gi7/48,Switch

switch# show plat hard mac add 0000.0500.0000 | i 0500|Index


27760 0000.0500.0000 902 SinglePort Gi1/46(53) ND SrcOrDst F

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

no copies will be sent to CPU for MAC source address learning

HW matches SW

Packet Loss / Path: L2 Lookup• SA Lookup: port security

switch# show run int gi 3/19

…

interface GigabitEthernet3/19

switchport access vlan 172

switchport mode access

switchport port-security

spanning-tree portfast

switch# show platform hardware mac vl 172

Flags are:

----------

D - Drop

ND - Do not drop


----- -------------- ----- ---------- ----------------------------

2640 0017.9543.EA7F 172 SinglePort Gi3/19(74) ND SrcOrDst

49300 0017.9543.EA7F 172 SinglePort WildcardAggport D SrcOrDst

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Traffic sourced from this MAC from any port

other than Gi3/19 will be dropped on vlan 172

Packet Loss / Path: L2 Lookup

DA Lookup: multicast, broadcast

switch# show mac add multi vlan 902 | i 0100.5e01.0101

902 0100.5e01.0101 igmp Gi1/46,Switch

switch# show plat hard mac add 0100.5e01.0101 | i 0100.5E01.0101|Index


20224 0100.5E01.0101 902 Ret 104444

switch# show plat hard ret chain index 104444

RetIndex 104444

RetWordIndex: 522220 Link: 1048575(0xFFFFF) FieldsCnt: 1

SuppressRxVlanBridging: true

Vlan: 902 BridgeOnly: N Gi1/46(53)

Switch# show platform hardware floodset vlan 902

Vlan 902:

Unicast Floodset:

FloodToCpu: -

RetIndex: 902

Gi1/46(53) Po1(417)

…

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

unknown unicasts will be flooded to these ports

Multicast traffic to 0100.5e01.0101 replicated

here, unless overridden by L3/ACL

Packet Loss / Path: L2 vs L3 vs ACL

What HW programming will direct the packet?

switch# show platform hardware ip fwdsel summary

L2Value == other (port/RET) (0):

IC

L3 0 1 2 3

0 l2 ic ic ic

1 l3 ic ic ic

2 l3 l3 ic ic

3 l3 l3 l3 ic

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Fwdsel relevant to ACL (ic) only when there is a

redirect action

Example:

L3 entry present, FwdSel=2

ACL redirect entry present, FwdSel=2

Winner = ACL (ic)

L3 Entry

ACL Entry

L2 entry floodset

Depends on “fwdsel”

> >

Packet Loss / Path: Input Classification

SVI and ACL statistics require hardware resources

Not enabled by default

switch# show run

…

interface Vlan902

ip address 92.92.92.1 255.255.255.0

counter

…

ip access-list extended deny

deny ip any any

hardware statistics

…

switch# show platform hardware vlan statistic summary

Region Name First Last First LastUsed Entries Entries

Block Block Entry Entry Used Free

Size 2 Counters Region 0 510 0 0 1 2043

Size 4 Counters Region 511 1022 2044 - 0 2048

VlanStatsTable Programming Complete: Yes

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Enable hardware counters

Ensure resources are available


ACL examples: static ACL, PBR, PACL

switch# show platform hardware acl input entries static

…

CamIndex Entry Type Active Apply QoS Hit Count

-------- ---------- ------ --------- ---------

2 IgmpToCpu Y N/A 14237 (estimate)

…

switch# show platform hardware acl input entries start 2 end 2 all

…

IP Src : 0.0.0.0 / 0.0.0.0

IP Dst : 224.0.0.0 / 240.0.0.0

IP Protocol : igmp / IpProtocolMask

…

ActIdx: 252 StatsIdx: 0 FwdIdx: (Cpu, Cpu: true, CpuEvent: 1, Port: 3)

switch# show platform hardware acl input actions 252

…

FwdSel: 3

L2Action: 2

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

• Watch for increment

• Hit does not mean packet count

IGMP sent to 224/4

will go to CPU

if FwdSel wins over L3

L2Action: (0 = permit, 1 = drop, 2 = redirect)



switch# show platform hardware acl input entries vlan 901 all

…

IP Src : 1.1.1.1 / 255.255.255.255

IP Dst : 0.0.0.0 / 0.0.0.0

…

ActIdx: 244 StatsIdx: 0 FwdIdx: (Adj, Adj: 8)

switch# show platform hardware acl input actions 244

…

FwdSel: 2

…

L3Action: 2

switch# show platform hardware ip adjacency entry 8

000008: vlan: 192 port: Po1 (417) size: 1 ifaId: 20

fwdCtrl: 5 cpucode: 3 sifact4: FwdToCpu sifact6: FwdToCpu

sa: 00:1E:F7:3F:F5:BF da: 00:0C:29:6D:1A:ED rwFmt: Unicast

packets: 0 bytes: 0

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Packets sourced from 1.1.1.1/32

will be redirected to adjacency 8 (Po1)

If FwdSel wins over L3

Note: PBR ACLs are removed if

adjacency becomes unavailable



Note: packets classified as non-IP, IPv4, IPv6 (cannot MAC ACL on an IP packet)

switch# show ip access deny

Extended IP access list deny

10 deny ip any any (1056 matches)

switch# show ip int gi 1/2

Inbound access list is deny

switch# show plat hard acl inp entr int gi 1/2 all

…

IP Src : 0.0.0.0 / 0.0.0.0

IP Dst : 0.0.0.0 / 0.0.0.0

IP Protocol : IpProtocolNull / IpProtocolNull

…

ActIdx: 254 StatsIdx: 0 FwdIdx: (None, rep: 0)

switch# show plat hard acl inp act 254

…

FwdSel: 0

…

L2Action: 1

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

All IPv4 traffic will be dropped

Fwdsel doesn’t matter

L2Action: (0 = permit, 1 = drop, 2 = redirect)

Packet Loss / Path: Input Classification / Policing

Order of operations

flow record microflow

match ipv4 source address

class-map match-all microflow

match flow record microflow

policy-map ingress

class voice-signalling

set dscp cs3

police cir 32000 bc 8000

conform-action transmit

exceed-action set-dscp-transmit cs1

exceed-action set-cos-transmit 1

class microflow

police cir 100000


exceed-action drop

class class-default

set dscp default

set cos 0

IM

L2

IC

NF

IP

FL

OC

OP

OM

QMUnconditional Marking

Microflow policing

• Flexible Netflow

• Class-map matching FNF

• Policer

Normal policer

Conditional Marking

Classification

Ingress

Classification

Ingress Policing

Ingress Marking

Unconditional

Ingress Marking

Conditional

Forwarding

Packet Loss / Path: Input Classification / Policing

Monitoring ingress Qos

switch# show policy-map interface gigabitEthernet 1/46

GigabitEthernet1/46

Service-policy input: ingress

Class-map: voice-signalling (match-all)

28283457437 packets

Match: dscp ef (46)

QoS Set

dscp cs3

police:

cir 32000 bps, bc 8000 bytes

conformed 76128704 bytes; actions:

transmit

exceeded 1810581188160 bytes; actions:

set-dscp-transmit cs1

set-cos-transmit 1

conformed 32000 bps, exceed 761238000 bps

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Class-map stats are shared across interfaces with the

same policy map

• Ensure counters increment

• Classification displays using the packet counts

• Policing displays using bytes

Packet Loss / Path: Forwarding Lookup

L3 unicast destination lookups

switch# show ip route 192.168.200.200

Routing entry for 192.168.200.0/24

Known via "static", distance 1, metric 0

Routing Descriptor Blocks:

* 192.168.100.100

Route metric is 0, traffic share count is 1

switch# show ip arp | i 192.168.100.100

Internet 192.168.100.100 0 000c.296d.1aed ARPA Vlan192

switch# show mac address dynamic | i 000c.296d.1aed

192 000c.296d.1aed dynamic ip,ipx,assigned,other Port-channel1

switch# show platform hardware ip route ipv4 network 192.168.200.0 255.255.255.0

Block: 0 En: true EntryMap: LSB Width: 80-Bit Type: Dst

…

000022: v4 192.168.200.0/24 --> vrf: Global Routing Table (0)

adjStats: true fwdSel: 2 mrpf: 0 (None) fwdIdx: 0 ts: 0

adjIndex: 8 vlan: 192 port: Po1 (417)

fwdCtrl: 5 cpucode: 3 sifact4: FwdToCpu sifact6: FwdToCpu

sa: 00:1E:F7:3F:F5:BF da: 00:0C:29:6D:1A:ED

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Remember: unicast traffic won’t be destination-routed unless:

• routing is enabled on the vlan

• traffic is sent to L3 MAC

• FwdSel of route wins over ACL

Packet Loss / Path: Output Classification / Policing Order of operations

policy-map egress

class voice

set dscp ef

set cos 5

priority

police cir percent 33

class voice-control

set dscp af31

set cos 3

bandwidth remaining percent 5

class class-default

dbl

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Marking

Queuing

Policing

Output

Classification

Output Policing

Output Marking

Unconditional

Output Marking

Conditional

Queuing

Classification

Packet Loss / Path: Output Classification / Policing

• Monitoring egress Qosswitch# show policy-map int g1/36 output

GigabitEthernet1/36

Service-policy output: AutoQos-VoIP-Output-Policy

Class-map: AutoQos-VoIP-Bearer-QosGroup (match-all)

625530530 packets

Match: qos-group 46

QoS Set

ip dscp ef

cos 5

priority queue:

Transmit: 32344068480 Bytes, Queue Full Drops: 0 Packets

police:

cir 33 %

cir 330000000 bps, bc 10312500 bytes

conformed Packet count - n/a, 32335870400 bytes; actions:

transmit

exceeded Packet count - n/a, 7813435520 bytes; actions:

drop

conformed 325185000 bps, exceed 97368000 bps

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Class-map stats are shared across interfaces with the

same policy map

• Ensure counters increment

• Classification display using the packet counts

• Policing display using bytes

• Queue full drops are in packets

Packet Loss / Path: Output Queuing DBL processing (if packet is not scheduled for drop)

Descriptor enqueued in queue memory

switch# show platform hardware interface gigabitEthernet 1/1 tx-queue

…

Phyport TxQ Head Tail Pre Empty Num BaseAddr Size Shape-Ok

Empty Packets TxQ Subport

-------------------------------------------------------------------------------

Gi1/1 0 0x0000 0x0000 True 0 0x20D10 16 True True

Gi1/1 1 0x0000 0x0000 True 0 0x00000 0 True True






Gi1/1 7 0x0000 0x0000 True 0 0x20D20 3152 True True

IM

L2

IC

NF

IP

FL

OC

OP

OM

QM

Default queues configured

Currently empty

policy-map egress_queueing

class dscp32-48

police cir 990000


exceed-action drop

priority

class dscp0-15

bandwidth 250000

queue-limit 400

class dscp16-31

bandwidth 250000

queue-limit 512

class class-default

switch# show platform hardware interface g2/48 tx-queue

…

Phyport TxQ Head Tail Pre Empty Num BaseAddr Size Shape-Ok

Empty Packets TxQ Subport

-------------------------------------------------------------------------------

Gi2/48 0 0x0000 0x0000 True 0 0x5ECE8 352 True False

Gi2/48 1 0x0000 0x0000 True 0 0x00000 0 True False




Gi2/48 5 0x0000 0x0000 True 0 0x5E958 512 True False

Gi2/48 6 0x0000 0x0000 True 0 0x5EB58 400 True False

Gi2/48 7 0x008A 0x0088 False 1421 0x5EE48 1520 True False

Packet Loss / Path: Output Queuing

Tx Q Class

0 dscp32-48

5 dscp16-31

6 dscp0-15

7 dscp49-63, class-default

Low priority queues can be

starved, policer recommended

Last queue is default queue

In this example, it is non-empty

First and last appear where expected, middle reversed

Packet Loss / Path: ASIC Drop Categories

Common Drop Event Reason Typical Description

BridgeToRxPortDrop received in a vlan with no other ports, replicated to a floodset/entry where ingress port

was a member

DblDrop packets dropped by DBL (including DBL on CPU ports)

InpL2AclDrop, InpL3AclDrop,

OutL2AclDrop, OutL3AclDrop

packets denied by ACL

rplErrDrop broadcast/multicast packets dropped while being replicated, many normal reasons to

increment, including: rpf failure, floodset containing drop port, packets replicated to the

CPU but also bridged to a floodset/entry containing the CPU

SptDrop spanning-tree drop; packets dropped because a port is not in a forwarding state

SrcHitDrop dropped at source learning stage; example: static MAC drop entry

TxQueFullDrop a tx port is oversubscribed

show platform software drop-port shows global ASIC drop events (not per interface)

these counters are frequently expected; baseline and/or high packet rate very useful

Packet Loss / Path: CPU Queues

switch# show plat cpu pack driv

Forerunner Packet Engine 1.83 (0)

Receive Queues: received packets summary

Qu Capac Guara CurPo Unpro Accum Kept BperP Packets

2 2512 112 610 0 2 2 73 610

58 512 256 37 12 5 511 216 591103

Receive Queues: dropped packets summary

Qu Total Packets Drop No Cell Drop Overrun Drop Underrun

58 591103 43623295103 0 0

Transmit Queues

Qu PosAdd Pendng Packets Bytes

0 595 0 8633668179 663318795241

1 863 0 5315423 363150782

However, combine high “Kept” with:

• CurPo does not increment

• Drop No Cell does increment

… queue 58 is stuck!

• High “Kept” indicates high rate of traffic

• Incrementing “Drop No Cell” indicates queue oversubscription

Check for transient flooding / loss versus stuck queue

Decode queue meaning with show platform software cpu events

Agenda

Products Overview

Troubleshooting

– Method




Tools/Tips

Wired/wireless ConvergenceNew capability on Sup8E with IOS-XE 3.7.0E

Overview

Converged Mode

Troubleshooting

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-4500-series-switches/guide-c07-733704.html

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/XE3-7-0E/wireless/configuration-guide/b_37e_4500sup8e_cg.pdf

Quick Start /Configuring

Wireless

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-4500-series-switches/guide-c07-733704.html

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/XE3-7-0E/wireless/configuration-guide/b_37e_4500sup8e_cg.pdf

Sup8E Wired/Wireless ConvergenceOverview

In-built daughter-card which enables wireless

capabilities

Supported only with cat4500es8-universalk9*

(Crypto) images

Not supported in VSS

10th slot: No line card supported on 4510R-E

chassis, only 47xx on 4510R+E chassis

‘Install boot’ method is required

Sup8E Wired/Wireless Convergence Converged Mode

rommon 1 >boot bootflash:cat4500es8-universalk9.SPA.03.07.00.E.152-3.E.bin

Loading image !!!!!!!!!!!!!!!!!!!!!!!

Checking digital signature....

[mem:/cat4500es8-firmware]

Digitally Signed Release Software with key version A

…

Switch#software expand file bootflash:cat4500es8-universalk9.SPA.03.07.01.E.152-3.E1.bin

Preparing expand operation ...

[5]: Expanding bundle bootflash:cat4500es8-universalk9.SPA.03.07.01.E.152-3.E1.bin

[5]: Copying package files

[5]: Package files copied

[5]: Finished expanding bundle bootflash:cat4500es8-universalk9.SPA.03.07.01.E.152-3.E1.bin

Switch(config)#boot system bootflash:packages.conf

Switch#wr mem

Switch#reload

*.bin file needs to be copied into bootflash

Boot .bin file and run software

expand file command

Set switch to boot packages.conf file

Sup8E Wired/Wireless Convergence Converged Mode

Switch#show module

Chassis Type : WS-C4510R+E

…

Mod Ports Card Type Model Serial No.

…

5 12 Sup 8-E 10GE (SFP+), 1000BaseX (SFP) WS-X45-SUP8-E CAT1749L0M3

Mod Redundancy role Operating mode Redundancy status

----+-------------------+-------------------+----------------------------------

5 Active Supervisor SSO Active

Mod Submodule Model Serial No. Hw Status

----+-----------------------+-----------------+------------+----+---------

5 Daughter Card WS-UA-SUP8E CAT1749L6FL 1.0 Ok

Switch#dir bootflash: | inc dc_console

32661 ---- 11678 May 21 2015 14:28:54 -07:00 dc_console_log-20150514-094417-UTC

Switch now in converged wireless

mode (install boot mode)

DC console logs have clues if

wireless mode does not come up

Sup8E Wired/Wireless Convergence Path of Packet Wired traffic


Line Card

Stub ASICs

Front Panel Ports

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

Wireless DC Wireless Daughtercard with 2 10G links

to Packet Processor

20Gbps of throughput

Sup8E Wired/Wireless Convergence Path of Packet Wireless traffic (1)


Line Card

Stub ASICs

Front Panel Ports

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

Wireless DC

Wireless packets are redirected to DC

interface GigabitEthernet3/1

switchport access vlan 123

switchport mode access

interface Vlan123

ip address 192.168.21.1 255.255.255.0

wireless mobility controller

wireless management interface Vlan123



Line Card

Stub ASICs

Front Panel Ports

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

Wireless DC

Switch#show plat hardware acl input entries

interface gigabitEthernet 3/1 all

…

Idx: 7 Hit: false

IP Src : 0.0.0.0 / 0.0.0.0

IP Dst : 192.168.21.1 / 255.255.255.255

IP Protocol : udp / IpProtocolMask

UDP Src Port : 0 / 0

UDP Dst Port : 5246 / 65535 …

ActIdx: 244 StatsIdx: 0 FwdIdx: (Adj, Adj: 10)

Switch#show platform hardware adjacency entry 10

000010: vlan: 123 port: Po255 (671)



Line Card

Stub ASICs

Front Panel Ports

Supervisor

NFE2

CPU

TCAMs1

Packet Processor

Forwarding Engine

Wireless DC Wireless operations complete packet is

returned again to Forwarding engine

pipeline

Packet is switched out front panel port

Sup8E Wired/Wireless Convergence Wireless Daughtercard Etherchannel

Switch#show platform mapping ports | inc Te5/

Te5/1 66 0 16 17 204 392 384

Te5/2 67 0 18 17 206 393 385

Te5/3 68 0 20 18 208 394 386

Te5/4 69 0 22 18 210 395 387

Te5/9 64 0 24 16 212 Po255(671) 392

Te5/11 65 0 26 16 214 Po255(671) 394

Wireless mode only 4 uplinks available in

redundant/non-redundant configurations on 10-slot

chassis

Te5/9 and 5/11 part of internal

Portchannel configured between

Packet Processor and Wireless DC.

Sup8E Wired/Wireless Convergence Packets Rx/Tx to Wireless DC

Switch#show platform hardware interface tenGigabitEthernet5/9 statistic

Switch Phyport Te5/9 Non-Zero Hardware Statistics

TxBytesTxQ0 : 192

TxBytesTxQ7 : 288

Superport64(Te5/9) Non-Zero Software Statistics

RxPackets64 : 7

TxPackets64 : 6

RxPackets65to127 : 1

TxPackets65to127 : 1

RxPackets128to255 : 1

RxMcastPackets : 5

TxMcastPackets : 3

RxUcastPackets : 4

TxUcastPackets : 4

RxGoodBytes : 672

TxBytes : 480

Packet processor superport interface

to Wireless DC

Sup8E Wired/Wireless Convergence Wireless DC CPU, Memory usage

Switch#show processes cpu location active-dc





PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 1560 1093 1427 0.00 0.00 0.00 0 init

2 10 283 35 0.00 0.00 0.00 0 kthreadd

Switch#show processes memory location active-dc

System memory : 1934480K total, 847036K used, 1087444K free, 123824K kernel reserved

Lowest(b) : 640243296

PID Text Data Stack Heap RSS Total Process

1 324 384 88 276 1736 4604 init

2 0 0 0 0 0 0 kthreadd

Location keyword introduced to

distinguish baseboard vs wireless

daughercard

Agenda

Products Overview

Troubleshooting

Method

System Resources

Packet path / loss

Wired/wireless Convergence

Tools/Tips

Tools: Wireshark

Wireshark Best Practices

Do not display directly to console without a buffer, file or a duration limit

Write to PCAP file on storage, display on switch or using laptop Wireshark GUI

Only the core filter is implemented in hardware as ACLs. Use a restricted filter to avoid high CPU

Available on Sup8E, Sup7E, Sup7L-E, 4500X

Onboard full packet capture, filter, decode / display

Up to 8 instances supported

Tools: Wireshark

Forwarding Engine

IOS-XE

Ring Buffer

Console

FileCore Filter

Display

Filter

Display

Filter

Capture

Filter

switch# monitor capture mycap int gi 1/46 in match ipv4 protocol tcp 10.1.1.1/32 any file location

bootflash:mycap.pcap limit duration 3

switch# monitor capture mycap start

*Apr 15 17:56:24.291: %BUFCAP-6-ENABLE: Capture Point mycap enabled.

*Apr 15 17:56:27.720: %BUFCAP-6-DISABLE_ASYNC: Capture Point mycap disabled. Reason : Wireshark session

ended

switch# show monitor capture file bootflash:mycap.pcap display-filter "ip.ttl == 100“

1 0.000000 10.1.1.1 -> 91.91.91.100 TCP [TCP ZeroWindow] 0 > 0 [<None>] Seq=1 Win=0 Len=2

Tools: Wireshark

Troubleshooting Steps Commands

Create a monitor monitor capture mycap <interface | vlan | control-plane>

Add core filter monitor capture mycap [access-list <acl> | match <in-line match CLI>]

Display monitor details show monitor capture

Start/stop a monitor session monitor capture mycap start | stop

Display a pcap file show monitor capture file <filename>

Display a pcap file in detail show monitor capture file <filename> detailed

Display a pcap file with filter show monitor capture file <filename> display-filter “filter-detail”

Check if wireshark is running show proc cpu | inc dumpcap

Tools: Embedded Event Manager

Extremely versatile tool for monitoring, automating, working around issues

(a) What do I want to detect? (b) What do I want to do after that?

event manager applet high-cpu

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1 get-type exact entry-op ge entry-val “80" poll-interval 10

action 1.0 syslog msg "HIGH_CPU! CPU is at: $_snmp_oid_val“

action 2.0 cli command "enable"

action 2.1 cli command "show process cpu | redirect bootflash:cpu.txt"

action 2.2 cli command "configure terminal"

action 2.3 cli command "event manager scheduler suspend“

%HA_EM-6-LOG: TEST: HIGH_CPU! CPU is at: 99

event manager applet interface-flapping

event syslog pattern ".*UPDOWN.*GigabitEthernet1/1.*" occurs 4

action 1.0 syslog msg “GigabitEthernet Interface 1/1 changed state 4 times“

action 2.0 cli command "enable"

action 2.2 cli command "configure terminal"

action 2.3 cli command “interface GigabitEthernet1/1 “

action 2.4 cli command “shutdown”

Collect process CPU usage when CPU is high

Bring an interface down when it flaps too frequently

Tools: EEM Netflow Integration1. Packets with TTL=1 sent to the switch (TTL=1 streams can cause high CPU)

2. NetFlow Engine collects the flow capturing the TTL value:

%HA_EM-6-LOG: ttl: Flow Monitor ttl reported Low TTL for 10.10.10.3 10.10.10.4

3. EEM triggers a syslog

when flow is detected:

switch# sh runn flow record ttl

match ipv4 ttl

match ipv4 protocol

match ipv4 source address

match ipv4 destination address

collect counter bytes

collect counter packets

collect timestamp sys-uptime first

collect timestamp sys-uptime last

switch# sh runn flow monitor ttl

Current configuration:

flow monitor ttl

record ttl

cache timeout active 40

switch# sh runn int gi 6/1

no switchport

ip flow monitor ttl input

ip address 10.10.10.2 255.255.255.254

switch(config)# event manager applet ttl

event nf monitor-name "ttl"

event-type create event1 entry-value "2"

field ipv4 ttl entry-op lt

action 1.0 syslog msg

"Flow Monitor $_nf_monitor_name reported Low TTL

for $_nf_source_address $_nf_dest_address"

check – show flow monitor ttl cache format record for IP TTL: 1

Tips: Crashes

Switch#show exception information

Exception configuration information

Coredump file - enabled,compressed

Maximum number of files

Switch#dir crashinfo:

Directory of crashinfo:/

24194-rw- 0 Dec 18 2013 04:13:06 -08:00 koops.dat

24198 -rwx 1679107 Oct 22 2014 14:38:41 -07:00 crashinfo_plogd_20141022-213819-UTC

24199-rwx 923370 Oct 22 2014 14:38:41 -07:00 fullcore_plogd_20141022-213819-UTC

Switch#dir kinfo:

Directory of kinfo:/

No files in directory

65624064 bytes total (65361920 bytes free)

koops.dat

Coredump not produced by default; configure with exception coredump

Gather latest files from both these

directories

Tips: Crashes (Wireless DC)

Switch#show exception files all

Exception crashinfo files all

NODE: LOCAL

============

Recent Crashinfo file:

crashinfo:crashinfo_plogd_20141022-213819-UTC

crashinfo:

crashinfo_iosd_20141022-213712-UTC

crashinfo_plogd_20141022-213819-UTC

fullcore_plogd_20141022-213819-UTC

crashinfo-dc:

Switch#dir crashinfo-dc:

Directory of crashinfo-dc:/

12 -rw- 0 May 14 2015 14:52:48 -07:00 cilogs

13 -rw- 0 May 14 2015 14:52:49 -07:00 koops.dat

Lists wireless DC crash files

Tips: Miscellaneous

Enable NTP to troubleshoot across switches

Include date and time for debug and log messages

service timestamps [debug, log] msec localtime show-timezone

Automatically output time and CPU utilization with each command (exec mode)

terminal exec prompt timestamp

When logging the console, add comments and prefix with “!” to avoid error messages

switch#!!! show module after peer reload

switch# show module

Tips: Make Life Easier

Search Bug Toolkit for known issues

Output Interpreter to decode command output

System Message Guide for mitigation recommendations

Smart Call Home in 12.2(52)SG

Catalyst 4000 Troubleshooting TechNotes

Catalyst 4500 Configuration Guide and Release Notes

NetPro discussion groups on http://www.cisco.com

http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs

https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl

http://www.cisco.com/en/US/products/hw/switches/ps4324/products_system_message_guides_list.html

http://www.cisco.com/

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Troubleshooting Cisco Catalyst 4500 Series Switchesd2zmdbbm9feqrf.cloudfront.net/2015/usa/pdf/BRKCRS-3142.pdf · –problems are solved faster when knowns can be eliminated ... (i.e.GRE)

Documents