Troubleshooting Routers

8/18/2019 Troubleshooting Routers

1/71

The Cisco Router as a HiddenTroubleshooting Tool

Ryan Determan

CCIE #5276


2/71

Agenda

• Introduction

• Presentation: The Cisco Router as a

Hidden Troubeshootin! Too

• "uestion $ns%er


3/71

Outline

The Cisco Router as a

Hidden Troubeshootin! Too:

I& Indi'idua toos and their secrets

II& Interna router (rocessesIII& )sin! the a((ro(riate command

I*& +hat am I oo,in! at-

*& +hen the router is the (robem


4/71

(I) Individual Tools and Their Secrets

.& IC/P Pin! and its o(tions

2& Cisco Tenet and its o(tions

0& Debu!!in! Pro(ery

1& 3$R5& Test command

6& Csim start command

7& 4$$ RTR res(onders


5/71

ICMP Ping and its options

4tandard Cisco (in!:

corerouter#(in! .&.20&.20&7

Ty(e esca(e se8uence to abort&

4endin! 59 .byte IC/P Echos to .&.20&.20&79 timeout is 2 seconds:

;;;;;4uccess rate is . (ercent


6/71


E>tended Cisco (in!:corerouter#(in!

Protoco @i(A:

Tar!et IP address: .&.20&.20&7

Re(eat count @5A:

Data!ram siBe @.A:

Timeout in seconds @2A:

E>tended commands @nA: y

4ource address or interace:

4et D bit in IP header- @noA:

oose9 4trict9 Record9 Timestam(9 *erbose@noneA:


;;;;;

4uccess rate is . (ercent


7/71


Record o(tion:corerouter#(in!

Protoco @i(A:

Tar!et IP address: .0.&.F&.&..5

Output Omitted…


Output Omitted…

oose9 4trict9 Record9 Timestam(9 *erbose@noneA: r

umber o ho(s @ G A:


8/71


Record (tion


9/71


Record (tion


10/71


4ource Interace (tion:corerouter#(in!

Protoco @i(A:


Output Omitted…


4ource address or interace: loopback0 (or an actual local IP address)

ut(ut mittedL



;;;;;

4uccess rate is . (ercent


11/71


/T) testin!:corerouter#(in!


Output Omitted…


4et D bit in IP header- @noA: y

Output Omitted…

4%ee( ran!e o siBes @nA: y

4%ee( min siBe @06A: !

4%ee( ma> siBe @.F21A: "##0

4%ee( inter'a @.A: "0


4endin! 7159 @!$$"##0Abyte IC/P Echos to .&.20&.20&79 timeout is 2 seconds:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;;;;;;;;;;;;;;;;;;;;;;;;;L&

4uccess rate is G6 (ercent


12/71

Telnet and its Options

4tandard Cisco Tenet:

corerouter#tenet .&.20&.20&251

R:

corerouter#.&.20&.20&251


13/71


Cisco Tenet (tions:corerouter#tenet .&.20&.20&251 -

debu! Enabe tenet debu!!in! mode

i('1 orce use o IP 'ersion 1

i('6 orce use o IP 'ersion 6

ine Enabe tenet ine mode

noecho Disabe oca echo

8uiet 4u((ress o!ino!out messa!es

route: Enabe tenet source route mode

sourceinterace 4(eciy source interace

stream Enabe stream (rocessin!

terminaty(e 4et termina ty(e

65505K Port number


14/71


4ource Interace o(tion:

corerouter#tenet .&.20&.20&251 sourceinterace ethernet

Debu! o(tion:

corerouter#tenet .&.20&.20&251 debu!

/uti(e o(tions:

corerouter#tenet .&.20&.20&251 sourceinterace ethernet debu!


15/71

Using ebug Appropriatel!

• 4ystem messa!es !enerated by the router


16/71

Using ebug Appropriatel!

Debu! )sa!e Ouideines:• Ensure you understand %hat you are oo,in! or

• Enabe debu! timestam(s to sim(y timerame

corerouter9 a%ays tenet

corerouter#term mon

• )se a iter %hene'er (ossibe

$ccessist debu! iterin!

Interace debu! iterin!


17/71

Access"list ebug #iltering

corerouter#con t

Enter coni!uration commands9 one (er ine& End %ith CT&

corerouter


18/71

Inter$ace ebug #iltering

corerouter#debu! ist -

26GGK access ist

Diaer Diaer interace

Ethernet IEEE F2&0

/utiin, /utiin,!rou( interace

Output Omitted…

crK

corerouter#debu! ist ethernet

corerouter#debu! i( (ac,et detai

IP (ac,et debu!!in! is on

or interace: Ethernet


19/71

Co%bination ebug #iltering

corerouter#debu! ist ethernet -

26GGK access ist

crK

corerouter#debu! ist ethernet .

corerouter#debu! i( (ac,et detai

IP (ac,et debu!!in! is on

or interace: Ethernet and access ist: .


20/71

&'AR

• 3$R et%or, 3ased $((icationReco!nition

• u unctionaity in I4 .2&.&5


21/71

nabling &'AR

• Cisco E>(ress or%ardin! must be enabed

corerouter#con t


corerouter


22/71

&'AR $or valuation

corerouter#sh i( nbar (rotocodisco'ery

Ethernet In(ut ut(ut

Protoco Pac,et Count Pac,et Count

3yte Count 3yte Count

5 minute bit rate


23/71

&'AR $or etection

corerouter#con t


corerouter


24/71

&'AR $or etection

corerouter#sh (oicyma( int e 2

4er'ice(oicy in(ut: na(sterdetect

Cassma(: nbardetect


25/71


26/71

&'AR $or Code Red

corerouter#con t


corerouter


27/71

Test Co%%and

• *arious dierent MtestN commandscorerouter#test -

aaa $$$ $uthentication9 $uthoriBation and $ccountin!

interaces et%or, interaces

memory on'oatie andor mutibus memory

(as Port $da(tor Tests

s!b( cac test the 2 cac unctionaity

ca Ca test commands

cry(to Test cry(to unctions

Output Omitted…


28/71

ocu%ented Test Co%%and

• test cry(to isa,m( 60&227&.5&22G

60&F.&251&.2. es(des

• test memory

• test 'oice (ort . reay rin! on


29/71

UNocu%ented Test Co%%and

• test dhc( @aocate >>>&>>>&>>>&>>>A @reeaseA

@rene%A

• test crash @'aueA or crK to enter crash

menu

• test ds( memory


30/71


31/71

SAA RTR responders

• eature im(emented in eary I4 'ersions9.&0

• ri!inay desi!ned or 4$ net%or,s

• e% ca(abiities ao% or intricate

TCP)DPIP testin!• Recent additions or *oice tests


32/71

SAA RTR Options

corerouter


33/71

SAA RTR T!pes

corerouter


34/71


35/71

SAA RTR States

corerouter#sh rtr o(erationastate

Entry umber: .

/odiication Time: .0:1G:27& mdt +ed $(r 1 20

Dia!nostics Te>t:

ast Time this Entry %as Reset: e'er

Connection oss ccurred: $4E

Timeout ccurred: $4E

'er Threshods ccurred: $4E

umber o (erations $ttem(ted: 07F

Current 4econds et in ie: ininite runs ore'er

(erationa 4tate o Entry: acti'e

atest Com(etion Time


36/71

(II) Router Processes

• 4%itchin! Ty(es

• IP CE

• E>aminin! the CP)

• Deci(herin! sho% 'ersion


37/71

S*itching T!pes

• Process 4%itchin!


38/71

Process S*itching

• )ses CP) to e>amine e'ery (ac,et o e'erydata o%

• /atches destination IP address a!ainst

routin! tabe or each indi'idua (ac,et

• CP) intensi'e9 but accurate• $chie'es MtrueN oad baancin!


39/71

#ast S*itching

• )ses CP) to e>amine irst (ac,et o e'erydata o%

• Ta,es oo,u( inormation and (o(uates an

inbound cache or the interace

• Cache is made u( o 0 ieds: Destination IP address

oca interace to use %hen or%ardin!

/$C header to (ace on the ne% rame


40/71

#ast S*itching cont+

• ast s%itchin! is ess CP) intensi'e thanProcess s%itchin!

• Route oo,u( time is decreased due to

(ro>imity o cache

• Due to cache imitations9 ast s%itchin! %ine'er use 2 (aths to the same destination

• Resut is Mdestination oadN baancin!


41/71


42/71

Cisco ,press #or*arding

corerouter


43/71

,a%ining the CPU

• 4ho% (rocesses memory

• 4ho% (rocesses CP)

• 4ho% (rocesses CP) o(tions


44/71

,a%ining Me%or! Usage

corerouter#sh (rocesses memory

Tota: .0F0.2G69 )sed: ...6F.F9 ree: 2660..6

PID TTZ $ocated reed Hodin! Oetbus Retbus Process

..FF1 .F1F 7015FG6 InitJ

F21 26FF52 F21 J4chedJ

.62622 51112F F.772 .7F11 JDeadJ

. 2F1 2F1 0F2F oad /eter

2 G6 6G21 CRZPT IV/P IPC


45/71

,a%ining CPU Usage

corerouter#sh (rocesses c(u

CP) utiiBation or i'e seconds: [[\ one minute: [\ i'e minutes: [

PID Runtime


46/71

,a%ining the CPU Histor! (-.s)

corerouter#sh (rocesses c(u history

.....

.

G

F

7

6

5 1

0

2

.

&&&&5&&&&.&&&&.&&&&2&&&&2&&&&0&&&&0&&&&1&&&&1&&&&5&&&&5&&&&

5 5 5 5 5

CP)[ (er second


47/71

,a%ining the CPU Histor! (-.%)

66

2 .. .. .. 21 . . .....22. . . 2 2......

.

G

F

7

6 JJ

5 JJ 1 JJ

0 JJ

2 JJ

. ##

&&&&5&&&&.&&&&.&&&&2&&&&2&&&&0&&&&0&&&&1&&&&1&&&&5&&&&5&&&&

5 5 5 5 5

CP)[ (er minute imum CP)[ # ? a'era!e CP)[


48/71

eciphering Sho* /ersion

I4


49/71


50/71

Sho* IP Protocols

Routin! Protoco is ]ei!r( 22F1]

ut!oin! u(date iter ist or a interaces is not set

Incomin! u(date iter ist or a interaces is not set

Deaut net%or,s a!!ed in out!oin! u(dates

Deaut net%or,s acce(ted rom incomin! u(dates

EIORP metric %ei!ht V.?.9 V2?9 V0?.9 V1?9 V5?

EIORP ma>imum ho(count .

EIORP ma>imum metric 'ariance . Redistributin!: ei!r( 22F1

$utomatic net%or, summariBation is not in eect

/a>imum (ath: 1

Routin! or et%or,s:

60&&&

25&22G&.GF& /ore


51/71

Sho* IP Protocols cont+

Continued rom (re'ious out(ut:Passi'e Interaceterna .7


52/71

Using Sho* Run

• I %e sti ha'e to use sho% run9 etNs use it(ro(ery

• 4ho% run oo%ed by the o(tion

• 4ho% run usin! the o(tion

• oo%in! the o(tion is a re!uar e>(ression


53/71

Sho* Run 0 options

E>am(es o sho% run corerouter#sh run - be!in 3e!in %ith the ine that matches

e>cude E>cude ines that match

incude Incude ines that match

corerouter#sh run inc accessist . *shows any access-list that begins with

the nu(bers " and 0

corerouter#sh run inc accessist . *shows any access-list that begins with

the nu(bers " and 0 followed by a deli(iter4 in this case :space;


54/71

Other 0 Co%%ands

ther commands that can utiiBe • 4ho% accessist .G

• 4ho% i( ar(

• 4ho% i( route

• 4ho% i( interace

• 3asicay any sho% command that dis(ays iterabe ino


55/71


56/71

Sho* Controllers

4ho% Controers @interaceA

0611#sh controers seria 0 *or c


57/71

Sho* Inter$ace ..

Ethernet is u(9 ine (rotoco is u(

Hard%are is $mdP29 address is aa00$0!"/$"/!


58/71

Sho* Inter$ace .. cont+

5 minute out(ut rate bitssec9 (ac,etssec

2606. (ac,ets in(ut9 06F1.6G bytes9 0 no buffer

Recei'ed 21.6 broadcasts9 runts9 !iants9 throttes

in(ut errors9 CRC9 rame9 o'errun9 ignored

in(ut (ac,ets %ith dribbe condition detected

26251 (ac,ets out(ut9 7176. bytes9 underruns out(ut errors9 2 coisions9 interace resets

babbes9 ate coision9 .7 deerred

ost carrier9 no carrier

out(ut buer aiures9 out(ut buers s%a((ed out


59/71

Sho* 4IP5 inter$ace

Ethernet is u(9 ine (rotoco is u(

Internet address is .&.20&.20&25121

3roadcast address is 255&255&255&255

/T) is .5 bytes

Directed broadcast or%ardin! is disabed

ut!oin! access ist is not set

Inbound access list is "02

Pro>y $RP is enabed

4(it horiBon is enabed

IC/P redirects are a%ays sent

IC/P unreachabes are a%ays sent

IC/P mas, re(ies are ne'er sent

IP fast switching is enabled

IP ast s%itchin! on the same interace is disabed

IP o% s%itchin! is disabed

more


60/71

Sho* 4IP5 inter$ace cont+

IP 5EB switching is enabled

IP CE eature ast s%itchin! turbo 'ector

IP out(ut (ac,et accountin! is disabed

IP access 'ioation accountin! is disabed

TCPIP header com(ression is disabed

RTPIP header com(ression is disabed

Probe (ro>y name re(ies are disabed

Policy routing is enabled4 using route (ap ficude is disabed

3OP Poicy /a((in! is disabed

IP muticast mutiayer s%itchin! is disabed

Inbound inspection rule is deter(an$org

Inbound audit rule is outside


61/71

Sho* TCP /T6 .

corerouter#sh tc( 'ty

tty.09 'irtua tty rom host .&.20&.20&..

Connection state is E4T$39 I status: .9 unread in(ut bytes:

Cocal host1 "0$"/$"/$/#!4 Cocal port1 /

Boreign host1 "0$"/$"/$""4 Boreign port1 /0#!

En8ueued (ac,ets or retransmit: 9 in(ut: misordered:

SRTT1 00 (s9 RTT: 00 ms9 RT*: 0 ms9 VRTT: msminRTT: ms9 ma>RTT: 0 ms9 $CV hod: 2 ms

(/) 1hen the Router is the


62/71

(/) 1hen the Router is theProble%

• Enabin! P4T /essa!es• Perormin! a 4tac, Trace

• Decodin! a 4tac, Trace

• Core Dum(s

• $ccessists


63/71

nabling POST Messages

• 3y deaut9 P4T messa!es are su((resseddurin! boot

• P4T messa!es can dia!nose %hy a router

isnNt bootin! correcty9 or not res(ondin!

• Enabin! P4T messa!es re8uiresmodiication o the coni!re!ister

• 3it 2.2

• Too add P4T messa!es use >$.2


64/71

Tracebac2s and Stac2 Traces

• +hen a router ais it (roduces a tracebac,• 4ome tracebac,s cause a reboot

• +e can e>amine the tracebac, %ith a stac,

trace

• $ter decodin!9 the stac, trace can (oint to

the (robem $(r . .7:27:: [4Z40CP)HO: Tas, ran or 17F1 msec ec9 PC ? 61032F&

Tracebac,? 61032. 60F00 60F0E.1 60F063 61F1611 60$1$7F

60372D 612. 721 612.7.>


65/71


66/71

Output Interpreter

• htt(s:%%%&cisco&comc!ibin4u((ortut(utInter(reterhome&(


67/71

Core u%ps

• +hen a router crashes it can (erorm a coredum(9 i coni!ured

corerouterce(tion dum( .&.20&.20&7 *a file called hostname-core will

be placed on the tftp ser,er "0$"/$"/$2 when a du(p is perfor(ed

• Zou can aso manuay (erorm a core dum(

or troubeshootin!

corerouter#%rite core *you will be pro(pted for a tftp-ser,er IP address and a

filena(e to use


68/71

Access"lists

• /ost common coni!uration mista,e isincorrect accessists

• 4ecurity based $C usuay are buit

(ermitK(ermitKdeny

• /issin! ines in $C (re'ent desired net%or,usa!e

• Im(icit deny iters anythin! not (ermitted

hi!her in the ist


69/71

en! IP An! An! 7og

• Ti(: turn im(icit deny into e>(icit deny %itho!!in!

corerouter


70/71

Sources o$ Other In$or%ation

• htt(:%%%&cisco&comicia Cisco Inormation %ebsite

• htt(:boerand&comdotu

Document the undocumented

• htt(:%%%&sunsetearnin!&comorumsur ree consutin! inormationa community

• htt(:cidrre(ort&or!

3OP %ebsite


71/71

Than2 6ou

Ryan W& Determan

rdeterman^sunsetearnin!&com

htt(:%%%&sunsetearnin!&com

.&F&56G&.FG1

Troubleshooting Routers

Documents