Troubleshooting Wireless LANs with Centralized Controllers

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKEWN-3011 1

Troubleshooting Wireless

LANs with Centralized

Controllers

BRKEWN-3011

Wesley Terry


Troubleshooting Wireless LANs

Supportability

Software and Support Model

Troubleshooting Basics

The Client Debug

WLC Config Analyzer (WLCCA)

Additional Troubleshooting


Supportability


Supportability

WLC Supportability

Methods of Management

Using the GUI

Important Show Commands (CLI)

Important Debugs (CLI)

Best Practices

AP Supportability

Methods of Accessing the AP

Important Show Commands


WLC Supportability

Methods of Management

GUI

HTTPS (E) / HTTP (D)

CLI

Console

SSH (E) / Telnet (D)

SNMP

V1 (D) / V2 (E) – Change me!

V3 (E) – Change me

Note: Management Via Wireless Clients (D)

Default Mode

(E)=Enabled (D)=Disabled


WLC Supportability

Using the GUI

Monitor

AP/Radio Statistics

WLC Statistics

Client Details

Trap Log


WLC Supportability

Using the GUI

Wireless > All APs

AP list shows AP Physical UP Time

APs are sorted by Controller Associated Time

Check bottom of AP list for any recent AP disruptions

Select AP to see Controller Associated Time (duration)


WLC Supportability

Using the GUI

Management

SNMP Config

Logs

Tech Support


Important Show Commands (CLI)

Show run-config

Must have! No exceptions!

“show run-config commands” (like IOS show running-config)

“show run-config no-ap” (no AP information added)

Show tech-support

CLI Tip

Log all output

Config Paging Disable

WLC Supportability


Important Debugs (CLI)

Debug client <client mac address>

Client Involved? Must Have! No Exceptions

Debug capwap <event/error/detail/info> enable

CLI Tips

Log all output

Debugs are session based, they end when session ends

“Config session timeout 60”, sets 60 minute idle timeout

Debug mac addr <mac address>

Used to filter debugs on specific Mac Address

Debug disable-all (Disables all debugs)

WLC Supportability


WLC Supportability

Best Practices

Change default SNMP Parameters

Configure Syslog for WLC and AP

Enable Coredump for WLC and AP

Configure NTP Server for Date/Time


Methods of Accessing the AP

Console

Telnet (D) / SSH (D)

No GUI support

AP Remote Commands

Enabling Telnet/SSH

WLC CLI: config ap [telnet/ssh] enable <ap name>

WLC GUI: Wireless > All APs > Select AP > Advanced

Select [telnet/ssh] > Apply

AP Supportability

Default Mode

(E)=Enabled (D)=Disabled


AP Remote Commands (WLC CLI)

Debug AP enable <AP name>

Enables AP Remote Debug

AP Must be associated to WLC

Redirects AP Console output to WLC session

Debug AP command “<command>” <AP name>

Output is redirected to WLC session

AP runs IOS, numerous generic IOS commands available

AP Supportability


Show Commands (AP CLI or WLC Remote Cmd)

Show controller Do[0/1] (or Show Tech)

Must have! Before/During/After event

Show log

WLC: show ap eventlog <ap name>

Show capwap client <?>

CLI Tips

Debug capwap console client

Debug capwap client no-reload

AP Supportability


Software and

Support Model


Software and Support Model

Opening a TAC Service Request

Cisco Support Model

TAC vs Business Unit

What to expect from TAC

How does escalation work?

WLC Software Trains

CCO (ED/MD/AW)

“Engineering Special” vs “Escalation”


What should I have ready?

Clear problem description

Always: Show run-config

If client involved, always: “debug client <mac address>”

Your analysis of any data provided

Expectations for customer involvement

TAC SR severity level descriptions state that You and Cisco will commit necessary resources according to severity

You must set correct expectation of timeline and severity




Potential reasons to slow a TAC SR‟s resolution

Information about the problem is missing

The severity level was not set appropriately

Data, such as traces or logs, has not been forwarded to the engineer

The scope or time requirements are not well understood by the engineer

The problem cannot be reproduced in the Cisco Technical Assistance Center lab

Access to the affected equipment for debugging purposes is not available


Cisco Support Model – TAC vs. BU

TAC

Customer advocate

Technology focused with cross technology collaboration

Escalation path within TAC exists

Business Unit - Escalation

Work in conjunction with TAC during specific engagements

Product specific focus

Engages development resources when necessary


Cisco Support Model – Expectations

What not to expect from TAC

Design and deployment

Complete configuration

Sales related information

What to expect from TAC

Configuration assistance

Problem analysis / bug isolation

Workarounds or fixes

Action plan to resolve SR

Hardware replacement

Engage BU when appropriate


Cisco Support Model - Escalation

TAC Escalation Process

Multi-Tier support resources within a technology

TAC to engage resources (TAC/BU) when appropriate

SR ownership might not change hands

Customer Escalation Process

Raise SR priority (S1/S2)

Engage account team

Your satisfaction is important to the Cisco TAC. If you have concerns about the progress of your case, please contact your regional TAC.


WLC Software Trains

CCO - Cisco.com release

6.0.202.0, 7.0.116.0, etc…

Full test cycle

Classified as ED when posted

AssureWave

AW is no longer tagged on CCO, but AW validation results are available at: http://www.cisco.com/go/assurewave

Results available 4 weeks after CCO

MD

MD tag represents stable releases for mass adoption

MD tag will be considered on CCO after AW release validation, 10 weeks in field and TAC/Escalation signoff

http://www.cisco.com/go/assurewave


WLC Software Trains - ES vs. Escalation

Engineering Special

Development “special” image for fix validation or limited use

Sanity tested

“As-is”

Escalation Code

Escalation is a post-CCO maintenance release with specific/minimal customer impacting SW fixes

Fix must be fully committed to the next CCO MR

Sanity + focus tested

Fully TAC+BU supported

“Running-Master” so each release builds upon the previous




The 10-Point Capture

IP

WLC

WLC

IP

IP

ACS

CAPWAP EO

IP802.11 Data

802.11 Management

CAPWAP802.11 Management

RADIUS

Su

pp

.

Driv

er

Ra

dio

EAP

Supplicant

Logs

Driver

Debugs/

Adapter

Capture

chan. 1

Wireless

Sniff

AP

DebugsWired

Sniff

WLC

Debugs

Wired

Sniff

ACS

Logs

DHCP

DHCP

Logs

NTP

Spectrum

Analysis



Troubleshooting 101

Clearly define the problem

Understand any possible triggers

Know the expected behavior

Reproducibility

Recommended Tools

Spectrum Analyzer

Wireless Sniffer and Wired Captures

Problem

Definition

Questions

Tests

Solution(s)

Analysis


Troubleshooting 101

Troubleshooting is an art with no right or wrong procedure, but best with a logical methodology.

Step 1: Define the problem

It is crucial to understand all possible details of a problem

Knowing what is and is not working will go a long way

With a proper understanding of the problem description you can skip many steps

Bad description: “Client slow to connect”

Good description: “Client associations are rejected with Status17 several times before they associate successfully.”


Troubleshooting 101

Step 2: Understand any possible triggers

If something previously worked but no longer works, there should be an identifiable trigger

Understanding any and all configuration or environmental changes could help pinpoint a trigger

Step 3: Know the expected behavior

If you know the order of expected behavior that is failing, defining where the behavior breaks down (Problem Description) is better than defining the end result.

Example: “One way audio between Phone A and B, because Phone A does not get an ARP Response for Phone B”


Troubleshooting 101

Step 4: Reproducibility

Any problem that has a known procedure to reproduce (or frequently randomly occurs) should be easy to diagnose

Being able to easily validate or disprove a potential solution saves time by being able to quickly move on to the next theory

If a problem is reproducible in other environments with a known procedure, TAC/BU can facilitate internal testing and proposed fix/workaround verification

Debugs and Captures of working scenarios can help pin point where exactly the difference is


Recommended Tools

Wireless Sniffer

Example: Linksys USB600N with Omnipeek

TAC can publish Omnipeek-RA if you have compatible HW

Wired Packet Capture

Example: Wireshark

Use for spanned switchports of AP/WLC or client side data

Spectrum Analyzer

Spectrum Expert with Card or Clean-Air AP


The Client Debug


Steps to Building an 802.11 Connection

1. Listen for Beacons

2. Probe Request

3. Probe Response

4. Authentication Request

5. Authentication Response

6. Association Request

7. Association Response

8. (Optional: EAPOL Authentication)

9. (Optional: Encrypt Data)

10. Move User Data

State 1:

Unauthenticated,

Unassociated

State 2:

Authenticated,

Unassociated

State 3:

Authenticated,

Associated

802.11

AP

WLC


A multi-debug macro(Cisco Controller) >debug client 00:16:EA:B2:04:36

(Cisco Controller) >show debug

MAC address ................................ 00:16:ea:b2:04:36

Debug Flags Enabled:

dhcp packet enabled

dot11 mobile enabled

dot11 state enabled

dot1x events enabled

dot1x states enabled

pem events enabled

pem state enabled

CCKM client debug enabled

The Client Debug

debug client <mac address>


Understanding the Client State

Name Description

8021X_REQD 802.1x (L2) Authentication Pending

DHCP_REQD IP Learning State

WEBAUTH_REQD Web (L3) Authentication Pending

RUN Client Traffic Forwarding

(Cisco Controller) >show client detail 00:16:ea:b2:04:36

Client MAC Address............................... 00:16:ea:b2:04:36

…..

Policy Manager State............................. WEBAUTH_REQD

00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)


The Client Debug - Walkthrough

Association (Start)

L2 Authentication (8021X_REQD)

Client Address Learning (DHCP_REQD)

L3 Authentication (WEBAUTH_REQD)

Client Fully Connected (RUN)

Deauth/Disassoc

Tips and Tricks


Client Debug - Association


(Cisco Controller) >debug client 00:16:EA:B2:04:36

(Cisco Controller) >

(Cisco Controller) >

Association received from mobile on AP 00:26:cb:94:44:c0

0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1621)

Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface '3'

Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3„

STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0

Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36

0.0.0.0 START (0) Initializing policy

0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)

0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)

0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client

0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1

apfMsAssoStateInc

apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Associated

Scheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds

Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

Association


Association

Association received from mobile on AP 00:26:cb:94:44:c0

0.0.0.0 START (0) Changing ACL 'none' (ACL ID 0) ===> 'none' (ACL ID 255) --- (caller apf_policy.c:1621)

Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site 'default-group', interface '3'

Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface '3'

Association received

Association Request, client did not “Roam” (Reassociate)

AP Base Radio = 00:26:cb:94:44:c0

vapId 1, site 'default-group', interface '3„

vapId = WLAN # (Wlan 1)

site = AP Group (default-group)

Interface = Dynamic Interface name (3)

vlan 3

Vlan = Vlan # of Dynamic Interface


Association

STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0


STA - rates

Madatory Rates (>128) = (#-128)/2

Supported Rates (<128) = #/2

1m,2m,5.5m,11m,6s,9s,12s,18s,24s,36s,48s,54s

Processing RSN IE type 48

WPA2-AES

Processing WPA IE type 221 = WPA-TKIP


Association0.0.0.0 START (0) Initializing policy

0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)

0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)

0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client

0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1

apfMsAssoStateInc

apfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to Associated


0.0.0.0 START

0.0.0.0 = IP we know for client (In this case nothing)

Change state to 8021X_REQD

Passed association, moving client to next state: 8021X_REQD

Scheduling deletion

Session Time on WLAN (1800 seconds in this case)


Association

Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

Slot 0 = B/G(2.4) Radio

Slot 1 = A(5) Radio

Sending Assoc Response Status 0 = Success

Anything other than Status 0 is Failure

Common Assoc Response Failures:

1 – Unknown Reason – Anything not matching defined reason codes

12 – Unknown or Disabled SSID

17 – AP cannot handle any more associations

18 – Client is using a datarate that is not allowed

35 – WLAN requires the use of WMM and client does not support it

201 – Voice client attempting to connect to a non-platinum WLAN

202 – Not enough available bandwidth to handle a new voice call (CAC Rejection)


Association - FSR

FSR aIOS CUWN

CCKM - WPA yes yes

CCKM - WPA2 yes yes

WPA2 PKC no yes

WPA2 "Sticky" yes no*

Processing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36

CCKM: Mobile is using CCKM

CCKM: Processing REASSOC REQ IE

Including CCKM Response IE (length 62) in Assoc Resp to mobile

Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) Vap Id 6 Slot 1

OR


Received RSN IE with 1 PMKIDs from mobile 00:16:ea:b2:04:36

Received PMKID: (16)

[0000] cb bc 27 82 88 14 92 fd 3b 88 de 6a eb 49 be c8

Found an entry in the global PMK cache for station

Computed a valid PMKID from global PMK cache for mobile


Association - Takeaway

Association vs. Reassociation

Debug shows

AP, Slot, AP-Group, WLAN ID, Interface, Data Rates, Encryption type

Association Response

Confirms if Client is associated

Defines reason if denied

Further troubleshooting

May require Wireless Sniffer or capture at AP Switchport

If not sending Assoc Request, must know why from Client

Trying disabling WLAN features to “dumb it down”


Client Debug –

L2 Authentication


802.1X Authentication

Server

EAP-ID-Request

Rest of the EAP Conversation

Radius-Access-Accept

(Key)EAP-Success

EAPOL-START

EAP-ID-ResponseRADIUS (EAP-ID_Response)

SupplicantAuthenticator

The Supplicant Derives the

Session Key from User Password or

Certificate and Authentication

Exchange

Session

Key


WPA2-AES-802.1XSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

Station 00:16:ea:b2:04:36 setting dot1x reauth timeout = 1800

dot1x - moving mobile 00:16:ea:b2:04:36 into Connecting state

Sending EAP-Request/Identity to mobile 00:16:ea:b2:04:36 (EAP Id 1)Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36

Username entry (cisco) created for mobile

Received Identity Response (count=1) from mobile 00:16:ea:b2:04:36EAP State update from Connecting to Authenticating for mobile 00:16:ea:b2:04:36

dot1x - moving mobile 00:16:ea:b2:04:36 into Authenticating state

…………………..

Entering Backend Auth Req state (id=3) for mobile 00:16:ea:b2:04:36

Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3)

Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36

Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25)

...........................


Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36

Processing Access-Challenge for mobile 00:16:ea:b2:04:36

Entering Backend Auth Req state (id=11) for mobile 00:16:ea:b2:04:36




Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36

Processing Access-Accept for mobile 00:16:ea:b2:04:36

***OR***

Processing Access-Reject for mobile 00:16:ea:b2:04:36


Common EAP Types 1 – Identity

2 – Notification

3 – NAK

4 – MD5

5 – OTP

6 – Generic Token

13 – EAP TLS

17 – LEAP

18 – EAP SIM

21 – EAP TTLS

25 – PEAP

43 – EAP-FAST





802.1X (Cont.) (WPA2-AES-PSK)Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0

Creating a PKC PMKID Cache entry for station 00:16:ea:b2:04:36 (RSN 2)

Adding BSSID 00:26:cb:94:44:c0 to PMKID cache for station 00:16:ea:b2:04:36

New PMKID: (16)

[0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd

Initiating RSN PSK to mobile 00:16:ea:b2:04:36

dot1x - moving mobile 00:16:ea:b2:04:36 into Force Auth state

Skipping EAP-Success to mobile 00:16:ea:b2:04:36

Including PMKID in M1 (16)

[0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cd

Starting key exchange to mobile 00:16:ea:b2:04:36, data packets will be dropped

Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36

state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00

Received EAPOL-Key from mobile 00:16:ea:b2:04:36

Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36

Received EAPOL-key in PTK_START state (message 2) from mobile 00:16:ea:b2:04:36

Stopping retransmission timer for mobile 00:16:ea:b2:04:36

Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36

state PTKINITNEGOTIATING (message 3), replay counter 00.00.00.00.00.00.00.01

Received EAPOL-Key from mobile 00:16:ea:b2:04:36

Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36

Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile 00:16:ea:b2:04:36

apfMs1xStateInc

0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state L2AUTHCOMPLETE (4)


WPA2-AES-PSK - FailedStarting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be dropped

Sending EAPOL-Key Message to mobile 00:1e:8c:0f:a4:57

state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00

Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57

Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57

Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57

Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57

802.1x 'timeoutEvt' Timer expired for station 00:1e:8c:0f:a4:57

Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57

Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57

Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57

Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57

Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57


Retransmit 2 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57

…………………


Retransmit failure for EAPOL-Key M1 to mobile 00:1e:8c:0f:a4:57,

retransmit count 3, mscb deauth count 3

Blacklisting (if enabled) mobile 00:1e:8c:0f:a4:57

apfBlacklistMobileStationEntry2 (apf_ms.c:4192) Changing state for mobile 00:1e:8c:0f:a4:57 on

AP 00:16:9c:4b:c4:c0 from Associated to Exclusion-list (1)


8021X_REQD means L2 Authentication pending

Authentication/Encryption has not be established

PSK is 802.1X, key is derived from PSK not AAA

If “Processing Access-Reject”

AAA/RADIUS Rejected the user (not the WLC)

If “Processing Access-Accept”

AAA/Radius Accepted the user

M1-M4 should follow

Further Troubleshooting

Debug aaa [all/event/detail/packet] enable

Debug dot1x [aaa/packet] enable

L2 Authentication - Takeaway


Client Debug –

IP Learning State


Client DHCP

00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state

00:16:ea:b2:04:36 apfMs1xStateInc

00:16:ea:b2:04:36 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4)00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 3 apVapId 3for this client

00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 3 apVapId 3

00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7)00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4755, Adding TMP rule

00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

00:16:ea:b2:04:36 Stopping retransmission timer for mobile 00:16:ea:b2:04:36

*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0

...................

00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 29, encap 0xec03)

...................

00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)

...................

00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0


Client DHCP

Client is in DHCP_REQD state

Proxy Enabled:

DHCP Relay/Proxy

Between WLC and Server

Required for Internal DHCP

Proxy Disabled:

Between Client and Server

DHCP is broadcast out VLAN

IP helper or other means required

Client State =

“DHCP_REQD“

DHCP Proxy Enabled

Client DHCP Discover

Unicast to DHCP

Servers

DHCP Offer from Server

DHCP ACK from Server

IP Address Learned

Client DHCP Request

DHCP Proxy Disabled

Client DHCP Discover Is

Bridged to DS


DHCP Proxy Enabled – DHCP Discover


32.151: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)

32.151: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings:

dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0,

dhcpGateway: 0.0.0.0, dhcpRelay: 0.0.0.0 VLAN: 0

32.151: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.1

(local address 10.10.1.4, gateway 10.10.1.1, VLAN 0, port 29)

32.151: 00:16:ea:b2:04:36 DHCP transmitting DHCP DISCOVER (1)

32.151: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 1

32.151: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 0

32.152: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36

32.152: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0

32.152: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.4

32.152: 00:16:ea:b2:04:36 DHCP requested ip: 10.99.76.147

32.152: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.1 (len 346, port 29, vlan 0)




32.152: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE


DHCP Proxy Enabled – DHCP Offer

34.166: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)

34.166: 00:16:ea:b2:04:36 DHCP setting server from OFFER (server 10.10.1.3, yiaddr 10.10.1.103)

34.167: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)

34.167: 00:16:ea:b2:04:36 DHCP transmitting DHCP OFFER (2)

34.167: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0





34.168: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.3


DHCP Proxy Enabled – DHCP Request

38.169: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)




38.169: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.3

(local address 10.10.1.4, gateway 10.10.1.3, VLAN 0, port 29)

38.169: 00:16:ea:b2:04:36 DHCP transmitting DHCP REQUEST (3)

38.169: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 1





38.170: 00:16:ea:b2:04:36 DHCP requested ip: 10.10.1.103


38.170: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.3 (len 354, port 29, vlan 0)




38.171: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE


DHCP Proxy Enabled – DHCP Ack

38.172: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)

38.173: 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Reached PLUMBFASTPATH: from line 5273

38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Replacing Fast Path rule

38.173: 00:16:ea:b2:04:36 Assigning Address 10.10.1.103 to mobile

38.173: 00:16:ea:b2:04:36 DHCP success event for client. Clearing dhcp failure count for interface management.

38.174: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)

38.174: 00:16:ea:b2:04:36 DHCP transmitting DHCP ACK (5)

38.174: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0






38.179: 00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0


DHCP Proxy Disabled – Discover/Offer


*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)

*00:16:ea:b2:04:36 DHCP processing DHCP DISCOVER (1)

*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0

*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0

*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36

*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0

*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0

*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86

*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS

*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)

*00:16:ea:b2:04:36 DHCP processing DHCP OFFER (2)

*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0





*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3

*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA


DHCP Proxy Disabled – Request/Ack

*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)

*00:16:ea:b2:04:36 DHCP processing DHCP REQUEST (3)

*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0





*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86


*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS

*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)

*00:16:ea:b2:04:36 DHCP processing DHCP ACK (5)

*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0






*00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

*00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile

*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA

*00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0


Learning IP without DHCP

Client IP can be learned by ways other than DHCP

Client sends gratuitous ARP or ARP Request (Static Client)

Client sends IP packet (Orphan Packet), we learn IP

DS sends packet to client, we learn IP from DS

Seen with mobile devices that talk before validating DHCP

Up to client to realize their address is not valid for the subnet

DHCP Required on WLAN for prevent this

*Orphan Packet from 10.99.76.147 on mobile

*0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

*Installing Orphan Pkt IP address 10.99.76.147 for station

*10.99.76.147 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)


DHCP_REQD means Learning IP State

Only “Required” if enabled on WLC

If Proxy is enabled

Confirm DHCP Server on Interface (or Wlan) is correct

DHCP Server may not respond to WLC Proxy (Firewalls?)

If Proxy is disabled, DHCP is similar to wired client


Check DHCP Server for what it believes is happening

If WLC does not show a BOOTREQUEST, confirm the client request arrives to the WLC and leaves in the configured way

If still believed to be on WLC: debug dhcp message enable

Client DHCP - Takeway


Client Debug –

L3 Authentication


Webauth*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)

*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)

……………………………...

*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to

WEBAUTH_REQD (8) last state WEBAUTH_REQD (8)*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) pemAdvanceState2 5170, Adding TMP rule*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Successfully plumbed mobile rule (ACL ID 255)

*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile

*pemReceiveTask: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 2, dtlFlags 0x0*pemReceiveTask: 00:16:ea:b2:04:36 Sent an XID frame

*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile

*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile

*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile ………………………………

*emWeb: 00:16:ea:b2:04:36 Username entry (cisco) created for mobile *emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14) last state

WEBAUTH_NOL3SEC (14)

*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last

state RUN (20)*emWeb: 00:16:ea:b2:04:36 Session Timeout is 1800 - starting session timer for the mobile

*emWeb: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID =

5006 IPv6 Vlan = 3, IPv6 intf id = 8

*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Successfully plumbed mobile rule (ACL ID 255)

*pemReceiveTask: May 17 22:25:16.578: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1,

dtlFlags 0x0


Webauth Redirect

Client in WEBAUTH_REQD state

ARP and DNS must be functional

Client attempts to browse internet

WLC “Hijacks” the handshake

Client redirects to Virtual Interface

Certificate negotiation if applicable

Webauth page is displayed

Client authenticates

Webauth

Client State =

“WEBAUTH_REQD“

ARP and DNS Function

3-Way Handshake HTTP

HTTP GET

200 Response

3-Way Handshake

HTTP(S) GET

Successful Authentication

Client State = “RUN“

Webauth Page Displayed


Confirm ARP and DNS Function

ARP and DNS Function


Capture from Wireless AdapterWebauth Redirect

WLC Responding

with SYN, ACK

WLC Responding

with SYN, ACK

Address for Client

to Redirect to

(Virtual IP/Name)

Redirect to Virtual

Interface Comes

from Here

Client Is Talking to

Webauth….

3-Way Handshake

HTTP GET

200 Response

3-Way Handshake

HTTP(S) GET

Webauth Page Displayed


If WEBAUTH_REQD, then not authenticated

Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv6*

If not redirected, can client browse to virtual IP?

Cert issue? Consider disabling HTTPS for HTTP webauth

Most common scenario involves ARP/DNS failure

Must confirm that client actually sends TCP SYN (http) to IP

If proven that TCP SYN is sent and WLC does not SYNACK, then there may be a WLC side problem

Debug webauth enable <client ip address>

debug client <MAC Address>

debug pm ssh-appgw enable

debug pm ssh-tcp enable

Webauth - Takeaway


Client Debug - Run


Run State

RUN State is the Client Traffic Forwarding State

Client is Connected and should be functional

10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)

10.10.3.82 RUN (20) Reached PLUMBFASTPATH: from line 5273

10.10.3.82 Added NPU entry of type 1, dtlFlags 0x0

OR

10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14)

10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20)

Session Timeout is 1800 - starting session timer for the mobile

10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063

10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0


Client Debug –

Deauth/Disassoc


Deauthenticated Client

Received Idle-Timeout from AP 00:26:cb:94:44:c0, slot 0 for STA 00:1e:8c:0f:a4:57

apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4


apfMsExpireCallback (apf_ms.c:608) Expiring Mobile!

Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)

Idle Timeout

Occurs after no traffic received from Client

Default Duration is 300 seconds

Session Timeout

Occurs at scheduled duration (default 1800 seconds)

Will force WEBAUTH user to WEBAUTH again


apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on

AP 00:26:cb:94:44:c0 from Associated to Disassociated





Manual Deauth

From GUI: Remove Client

From CLI: config client deauthenticate <mac address>


apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile

00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated

Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)


apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1



apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on

AP 00:26:cb:94:44:c0 from Associated to Disassociated


WLAN Change

Modifying a WLAN in anyway Disables and Renables WLAN



Retransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3,

mscb deauth count 0

Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)

Authentication Timeout

Auth or Key Exchange max-retransmissions reached

Cleaning up state for STA 00:1e:8c:0f:a4:57 due to event for AP 00:26:cb:94:44:c0(0)

apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile

00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated

Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)

AP Radio Reset (Power/Channel)

AP disasassociates clients but WLC does not delete entry


Deauthentication - Takeaway

Client can be removed for numerous reasons

WLAN change, AP change, configured interval

Start with Client Debug to see if there is a reason for a client‟s deauthentication


Client debug should give some indication of what kind of deauth is happening

Packet capture or client logs may be require to see exact reason


Client Debug – Tips and Tricks


Tips and Tricks

Collect a client debug for an extended duration

Several roams, deauths, failures, etc…

Use an enhanced text editor with filter or “find all”

I use Notepad++

Find All

“Association Received” (will also pull reassociations)

“Assoc Resp”

“Access-Reject”

“timeoutEvt”


Tips and Tricks


Tips and Tricks


Client Debug – Summary


Client Connectivity

Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585

Configuration Issues

SSID Mismatch

Security Mismatch

Disabled WLAN

Unsupported Data-Rates

Disabled Clients

Radio Preambles

Cisco Features - Issues with Third Party Clients

Aironet IE

MFP

http://www.cisco.com/en/US/products/ps6366/products_tech_note09186a00809d45a2.shtml



802.11n Speeds

Troubleshoot 802.11n Speeds Document ID: 112055

Configuration Issues

11n Support Enabled

WMM is Allowed or Required

Open or WPA2-AES

5Ghz Channel Width

2.4Ghz does not support 40-Mhz Channels

http://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a0080b42507.shtml


802.11n A-MPDU/A-MSDU

Aggregation methods used could impact interop or performance

WLC Default 11n Config:802.11n Status:

A-MPDU Tx:

Priority 0............................... Enabled

Priority 1............................... Disabled







A-MSDU Tx:










WLC Config Analyzer (WLCCA)


What Is the WLCCA?

It is a Post Sales tool

Main objective: Save time while analyzing configuration files from WLCs

Secondary objective: Carry out RF analysis

It is NOT a management or monitoring tool

Focused to work off-line to the WLC

Not TAC supported

Development: [email protected]

General internal alias:[email protected]

“Pet project”: no official Cisco product.

mailto:[email protected]













Where?

Support Forums DOC-1373

https://supportforums.cisco.com/docs/DOC-1373




Complete config output from WLC

Show run-config

It does not work with old “show running-config” or with TFTPbackup, or with show tech

The show run-config acts as “snapshot” of current config + RF state

Likely best to obtain config from SSH with

config paging disable

Input Needed


Functionality Overview - Checks

Audit Checks

More than 100 config detail verifications

Based on TAC/Escalation cases experience

Some obvious, some hard to catch

No “change this” messages, some need “contextualization”


Functionality Overview

Audit Checks


Functionality Overview

Config View


WLCCA – High RF Index APs


Reducing CCI

Turn off excess 2.4 radios. May want to do this gradually, e.g. turn off 20% of radios per attempt

After turning off excess radios, could set DCAsensitivity to high

Let DCA/power settings settle down overnight.

See how things look in the morning

Repeat till you see the desired coverage in 2.4GHz


2.4GHz – Target Coverage

Most all 2.4GHz radios are at power 2 - 5 (don't want 7 or 8)

In all locations, you have coverage that looks like this (take these as guidelines, not gospel):

Hottest channel's AP is at least -67dBm

Next hottest AP on that channel is at least 19 dB below the hottest

Next hottest channel's AP is at least -67dBm

OK if next hottest AP on that channel is less than 19 dB below the hottest


5 GHz – Target Coverage

Most all 5GHz radios are at power 1 – 3 (at least 14dBm)

Consider the RRM min power setting in 6.0

Consider a radically high tx-power-threshold, like -55 dBm

8 – 12 channels in use (20 seem to be too many for the 792x to scan)

In all locations, seek this:Hottest channel's AP is at least -67dBm

Next hottest AP on that channel is at least 19 dB below the hottest

Next hottest channel's AP is at least -67dBm OK if next hottest AP on that channel is less than 19 dB below the hottest





Wireshark Tutorial

Clean Air SE-Connect / AP Sniffer Mode

AP Join

RRM

Multicast/Broadcast

Mobility

VoWiFi


Wireshark Tutorial


Wireshark Tutorial

Default Wireshark view might look like this:


Wireshark Tutorial

Newer versions of Wireshark have a feature for “Apply as Column”

This will take any decodable parameter and make a column


Wireshark Tutorial

Within seconds your wireshark can also have:


Wireshark Tutorial

Filtering data is just as easy


Wireshark Tutorial - CAPWAP

User data is encapsulated in CAPWAP


Wireshark Tutorial

Wireshark can also de-encapsulate CAPWAP DATA

Edit > Preference > Protocols > CAPWAP


Wireshark Tutorial

With CAPWAP de-encapsulated you can see all the packets to/from client (between AP and WLC)


SE-Connect – Clean Air

AP Sniffer Mode


SE-Connect and Sniffer Mode

Clean Air APs can be used in lieu of Spectrum Card for Spectrum Analysis

AP can be placed in SE-Connect mode for full functionality

AP in local mode can be used now for Spectrum Analysis of current channel

AP Sniffer Mode can be used in lieu of Wireless Sniffer

Packets can be sent from either radio upstream to a packet capture software (Wireshark or Omnipeek for example)


Spectrum Expert with Clean Air

Obtain Spectrum Key

Connect to Remote Sensor


Spectrum Expert with Clean Air


Sniffer Mode AP

Select channel to Sniff

Select destination for traffic


Sniffer Mode AP

Omnipeek has a Remote Adapter to capture this data

Wireshark, just capture network adapter

NOTE: Wireshark does not open the port UDP 5000

PC will send ICMP Unreachables


Sniffer Mode AP

With wireshark, filter !icmp.type == 3

Data (UDP 5000) still not intelligible yet

Decode as Airopeek


Sniffer Mode AP


AP Discover/Join


AP Discover/Join

AP Runs Hunting Algorithm to Find

Candidate Controllers to Join


AP - Discover Process

AP Discovery Req to known and learned WLCs

Broadcast

Reaches WLCs with MGMT Interface in local subnet of AP

Use “ip helper-address <ip>” with “ip forward-protocol udp”

Dynamic

DNS: cisco-capwap-controller

DHCP: Option 43

Configured (nvram)

High Availability WLCs – Pri/Sec/Ter/Backup

Last WLC

All WLCs in same mobility group as last WLC

Manual from AP - “capwap ap controller ip address <ip>”


AP - Discover Process

X

bro

adcast

Discover Request sent to all methods the AP knows

Discover Response sent from all WLCs that received the Discovery Request


AP – WLC Selection/Join

WLCs send Discovery Response back to AP

Name, Capacity, AP Count, Master?, AP-MGR, Load per AP-MGR

AP selects the single best WLC candidate from

High Availability Config: Primary/Secondary/Tertiary/Backup

Master Controller

Greatest available capacity

Ratio of total capacity to available capacity

AP sends single Join Request to best candidate

WLC responds with Join Response

AP joins and receives config (or downloads image if not correct)


“Lightweight AP (LAP) Registration to a Wireless LAN Controller (WLC)”, Document ID 70333

Make sure time on WLC is accurate!

From AP:

Debug ip udp

Debug capwap client events

From WLC

Debug mac addr <AP ethernet mac>

Debug capwap [event/error/packet] enable

Debug pm pki enable

Troubleshooting AP Discovery/Join

http://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtml

http://www.cisco.com/en/US/tech/tk722/tk809/technologies_tech_note09186a00806c9e51.shtml


RRM


RRM

There are usually only two common scenarios or issues involving RRM

APs not changing channel

Check if other APs are in each others neighbor list

APs not changing power

Nearby APs list meets the general rule of RSSI from 3rd

closest AP is better than TPC Threshold


RRM Debugs

AP

debug capwap rm mesurements

debug capwap rm rogue

WLC – debug airewave-director <?>


RRM Show AP Auto-RF (In Run-Config)

show ap auto-rf [802.11a/b] <AP Name>

Load Information

Receive Utilization.. 0 % Rx load to Radio

Transmit Utilization.. 2 % Tx load from Radio

Channel Utilization.. 12 % % Busy

Nearby APs

AP 00:16:9c:4b:c4:c0 slot 0.. -28 dBm on 11 (10.10.1.5)

AP 00:26:cb:94:44:c0 slot 0.. -32 dBm on 11 (10.10.1.4)


Broadcast/Multicast


Broadcast/Multicast


Broadcast/Multicast

AP Multicast Mode – Multicast

Address must be unique among WLCs

Broadcast Traffic is delivered via the Multicast Mode

AP/WLC/Client Subnets must be Multicast enabled

For Multicast Mode - Multicast

Quick check for Multicast is to confirm that Multicast-Unicast mode works


Broadcast/Multicast

AP Show Commands

Show capwap mcast

Show capwap mcast mgid all


Client Mobility


Mobility—Intra-Controller

Client roams between two APs on the same controller


Mobility—Inter-Controller (Layer 2)


Mobility—Layer 3

Layer 3 roaming (a.k.a. anchor/foreign)

New WLC does not have an interface on the subnet the client is on

New WLC will tell the old WLC to forward all client traffic to the new WLC

Asymmetrictraffic path established (deprecated)

Symmetrictraffic path


Mobility—Messaging Flow

When a client connects to a WLC for the first time, the following happens:

New WLC sends MOBILE_ANNOUNCE to all controllers in the mobility group when client connects

Old WLC sends HANDOFF_REQUEST

New WLC sends HANDOFF_REPLY


Mobility— L2 Inter WLCDebug Client <Mac Address>

Debug Mobility Handoff Enable








*mmListen: Mobility packet received from:

*mmListen: 10.4.22.55, port 16666

*mmListen: type: 3(MobileAnnounce) subtype: 0 version: 1 xid: 783 seq: 1453 len 116 flags 0

*mmListen: group id: e42cb3a9 87f62b45 57c0f8a3 92747b23

*mmListen: mobile MAC: 00:23:33:41:71:10, IP: 0.0.0.0, instance: 0

*mmListen: VLAN IP: 10.4.23.97, netmask: 255.255.255.0

*mmListen: Switch IP: 10.4.22.55

*mmListen: Handoff Virtual IP Mismatch, Local = 1010101, Request = 1020304

**** Handoff Request Ignored

*apfReceiveTask: 10.4.122.127 RUN (20) State Update from Mobility-Complete to Mobility-Incomplete

*apfReceiveTask: Mobile 00:23:33:41:71:10 associated with another AP elsewhere, delete mobile

*apfReceiveTask: 10.4.122.127 RUN (20) mobility role update request from Local to Handoff

Peer = 0.0.0.0, Old Anchor = 10.4.130.70, New Anchor = 0.0.0.0

*apfReceiveTask: Clearing Address 10.4.122.127 on mobile

*apfReceiveTask: apfMsRunStateDec

*apfReceiveTask: 10.4.122.127 RUN (20) Change state to DHCP_REQD (7) last state RUN (20)

*apfReceiveTask: apfMmProcessDeleteMobile (apf_mm.c:548) Expiring Mobile!

*apfReceiveTask: Mobility Response: IP 0.0.0.0 code Handoff Indication (2), reason Client handoff successful -

anchor retained (0), PEM State DHCP_REQD, Role Handoff(6)

*apfReceiveTask: apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:23:33:41:71:10 on

AP 10:8c:cf:eb:69:80 from Associated to Disassociated

*apfReceiveTask: Deleting mobile on AP 10:8c:cf:eb:69:80(1)

*pemReceiveTask: 0.0.0.0 Removed NPU entry.

Mobility— L3 Handoff Ignored


Mobility Group vs. Mobility Domain

Mobility Group - WLCs with the same group name

L2/L3 Handoff

Auto Anchoring

Fast Secure Roaming

APs get all of these as a Discover candidate

Mobility Domain - WLCs in the mobility list

L2/L3 Handoff

Auto Anchoring


Mobility Data/Control Path

Sent between all WLCs, by member with lowest MAC

Control Path = UDP 16666 (30 Seconds)

Data Path = EoIP Protocol 97 (10 Seconds)

debug mobility keep-alive enable <IP Address>


Voice over WiFi


VoWiFi

Wireless IP Phone Deployment Guide

http://www.cisco.com/en/US/docs/voice_ip_comm/cuipph/7925g/7_0/english/deployment/guide/7925dply.pdf

Best Practices

-67 dBm signal with 20-30% cell overlap

802.11A

CCKM for Fastest Roaming

Avoid designs where AP is seen at superb signal, but drops off instantly




VoWiFi - Troubleshooting

Must know if problem occurs during roaming events or when no association change takes place

If no change in connection

Interference

Coverage loss with no other candidate

End to End QOS missing/problem

If during roaming event

How long did the roam take?

Does the client associate to another AP again within seconds?

Does the client associate to the same AP again?

Is the phone roaming to the designed next candidate?


VoWiFi - Troubleshooting

Define a reproducible area where you believe you have perfect voice coverage but have problems

Place phone in Neighbor List Mode (On a call)

Real Time current AP RSSI and candidate list

Confirm AP as next best candidate is realistically a good candidate

Confirm devices roams to correct candidate where the intended design specifies

Watch out for sudden drops in coverage


VoWiFi - Debugs

Phone can Trace (debug) to file or syslog

Recommend USB Connection and SYSLOG

Configured via GUI

Enable Debug level for Kernel, WLAN MGR, WLAN Driver

WLC Debugs

Debug client <mac>

Debug cac all enable

Wireless Packet Captures


Summary


SummaryClient

WLC - show run-config, debug client <mac>, debug dhcp message enable,

debug dot1x <?> enable, debug aaa <?> enable,

AP - Show tech, show controller D<0/1>

Data - Driver/Supplicant Logs, Wireless Capture, AAA Logs, DHCP Logs

WebauthWLC - (Client debugs), debug webauth enable <IP>, debug pm ssh-appgw enable,

debug pm ssh-tcp enable

Client - local capture

MobilityWLC - debug mobility handoff enable, debug mobility keepalive enable <IP>

Data - Wired capture

AP JoinWLC - debug capwap [events/error/packet] enable

AP - debug capwap client events, debug ip udp

Data - Wired capture

RRMWLC - show run-config, debug airewave-director <?>

AP - debug capwap rm measurements, debug capwap rm rogue

Multicast/BroadcastAP - show capwap mcast, show capwap mcast mgid all

Data - Infrastructure Configuration

VoiceWLC - (Client debugs), debug cac all enable

Data – Wireless capture, Phone traces


Summary

Links:

Understanding Debug Client on Wireless LAN Controllers (WLCs) Document ID: 100260

Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585

Troubleshoot 802.11n Speeds Document ID: 112055

Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller Document ID: 99948

http://www.cisco.com/en/US/products/hw/wireless/ps430/products_tech_note09186a008091b08b.shtml










http://www.cisco.com/en/US/products/ps6366/products_tech_note09186a00808f8599.shtml

http://www.cisco.com/en/US/products/ps6366/products_tech_note09186a00808f8599.shtml


Complete Your Online Session Evaluation

Receive 25 Cisco Preferred Access points for each session evaluation you complete.

Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don‟t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.

http://www.ciscolivevirtual.com/


Visit the Cisco Store for Related Titles

http://theciscostores.com

http://theciscostore.com/



Thank you.

Troubleshooting Wireless LANs with Centralized Controllers

Documents

cisco public

cisco andor

wlc session ap

ap remote debug ap

wlc gui

ap important

ap eventlog

ap information