BRKAPP-3003 Advanced Troubleshooting the Cisco Application Control Engine
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 2
Session Agenda
ACE Architecture
Discuss the Architecture
Functions of control plane and data plane
Common debugging commands
Packet Capturing and logging
Traffic Forwarding on ACE
Admin Context and ACL Merge
Flow Management
Connection Handling on ACE
Layer 4/7 Troubleshooting and Performance
Health Monitoring on ACE
High Availability on ACE
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 4
ACE20 Module Hardware Architecture
Switch
Fabric
Interface
16G
SSL
Crypto
10G
2G
Console
port
Sup
Connect
100M
Control
Plane
Network
Processor 1
Network
Processor 2
10G10G
Classification
Distribution
Engine
(CDE)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 5
ACE30 Module Hardware Architecture
Switch
Fabric
Interface
16G
2G
Console
port
Sup
Connect
100M
Control
Plane
8G
Daughter Card 1
NP1 NP2
8G
Daughter Card 2
NP3 NP4
Classification
Distribution
Engine
(CDE)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 6
2x 700MHz MIPS
1 GB Memory
Control Plane Software
Supervisor
Connection
DBUS
16 Gbps
Bus
RBUS
EOBC
Cisco
ASIC
100 Mbps 8 Gbps
8 Gbps
1 Gbps
ACSW OS
60Gbps switching Capacity
IPv4, IPv6 Classifications
TCP Checksum
Generation/Verification
Variable Load Distribution
Daughter Card 1
16 Gbps
CEF720 Linecard
20 Gbps
20 Gbps
Switch Fabric
ACE30 Detailed Hardware Architecture
CPU
Classification Distribution
Engine (CDE)
NetworkProcessor
1
Verni FPGA
DRAM 4 GB
DRAM 4 GB
NetworkProcessor
2
shared memory
Daughter Card 2
NetworkProcessor
3
Verni FPGA
DRAM 4 GB
DRAM 4 GB Network
Processor4shared memory
Cavium Octeon CN5860 (OcteonPlus)
16 core, 600 MHz CPUs with 4G DRAM 32k iCache, 16k dCache, 2MB L2 cache
On chip support for Encryption/Decryption Coprocessors for Compression/Decompression
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 7
Data Traffic vs Management Traffic
ACE30 Control plane architecture is very similar to ACE20
Device control
Configuration manager (CLI, XML API, SSH, …)
Server health monitoring (native probes, TCL scripts)
Syslog's, SNMP, …
ARP, DHCP relay
High-Availability
ACL Compilation
ACE30 data plane architecture is very similar to ACE 4710
Connection management
TCP termination
Access lists
NAT
SSL Offload
Regular expression matching
Load Balancing & forwarding
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 9
Common Debugging
Show commands on the Catalyst 6500 Supervisor
show version
show clock
show module
show power
show asic slot <n>
show interface TenGigabitEthernet <n>/1
show interface TenGigabitEthernet <n>/1 trunk
show svclc vlan-group
[no] power enable <module>
show svclc module <n> traffic
Show fabric utilization detail
Make sure the module status is OK
VLAN‘s used by ACE must be
configured in the MSFC
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 10
Common Debugging
Show commands available on ACE
show version
show cde health
show ft group status
show ip int br
show int vlan <n>
show arp
show service-policy
show serverfarm
show rserver
show probe
show conn
show stat
show ip traffic
show resource usage
show np 1 me-stats “-s norm”
show np 1 me-stats “-s norm –M1”
System Information
L2, L3
Performance,
Resources
Debugging
Flows
L4, L7
This provides the DELTA
If incorrect version, check ‗boot‘ parameter
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 11
Show Module from the Catalyst 6500 Supervisor
cat6k#show mod
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 1 Application Control Engine 10G Module ACE20-MOD-K9 SAD12345678
2 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45 SAD12345L44
5 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD12345D5L
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0001.0002.0003 to 0001.0002.000a 2.4 8.7(0.22)ACE A2(2.3a) Ok
2 00d0.d32e.1b42 to 00d0.d32e.1b71 1.5 5.4(2) 8.5(0.46)RFW Ok
5 000f.f7be.b17c to 000f.f7be.b17f 4.0 8.1(3) 12.2(PP_R31_ Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD123456N2 1.3 Ok
5 MSFC3 Daughterboard WS-SUP720 SAD123455VE 2.1 Ok
Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
5 Pass
Module status shows OK
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 12
Verifying Version and Licenses
ACE/Admin# show version
Cisco Application Control Software (ACSW)
<snip>
Software
loader: Version 12.2[121]
system: Version A2(2.3a) [build 3.0(0)A2(2.3a)
system image file: [LCP] disk0:c6ace-t1k9-mz.A2_2_3a.bin
installed license: ACE-08G-LIC ACE-VIRT-250 ACE-SSL-15K-K9
Hardware
Cisco ACE (slot: 1)
cpu info:
number of cpu(s): 2
cpu type: SiByte
cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz
cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz
Installed Licenses
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 13
Available System Memory and Uptime
ACE/Admin# show version – Continuation of output
[...]
memory info:
total: 827128 kB, free: 335372 kB
shared: 0 kB, buffers: 3540 kB, cached 0 kB
cf info:
filesystem: /dev/cf
total: 1014624 kB, used: 529472 kB, available: 485152 kB
last boot reason: NP 2 Failed: NP ME Hung
configuration register: 0x1
ACE kernel uptime is 7 days 23 hours 42 minute(s) 25
second(s)
Displays ACE module uptimeUseful information in
case of system reload
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 14
ACE File System
Use the dir command to view directory listing for files
ACE/Admin# dir ?
core: Directory or filename
disk0: Directory or filename
image: Directory or filename
probe: Directory or filename
volatile: Directory or filename
The internal File system is mapped as below
/mnt/cf - Image:
Also the following compressed file systems are used
/TN-HOME = disk0:
/TN-CONFIG = Startup config
/TN-LOGFILE = Internal Storage for audit logs
/TN-CERTKEY-STORAGE : internal storage for Cert and Keys
/TN-COREFILE = core:
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 15
ACE File System
Load debug plug-in to access ACE file system
Startup configuration located at /mnt/cf/TN-CONFIG
ACE will generate / fix any missing or corrupted file systems during boot
When to use the format command?
If you receive the following error
Warning!! This will erase everything in the compact flash including startup configs for all the contexts and reboot the system!!
ACE/Admin# write memory
ERROR!config filesystem is not mounted on compact flash
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 16
Working with Core Files
If ACE creates a core file you can locate the files in the core directory
All cores files are stored in dir core: (core names are self explanatory)
ACE/Admin# dir core:
99756 Apr 1 17:57:05 2011 ixp2_crash.txt
13047 Apr 1 17:56:59 2011 loadBalance_core_log.tar.g
ixpx_crash.txt will have some details on the core dump
If it is a kernel crash , then a file named crash info will be available in core
Show version will show last reload reason
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 18
Logging Features
Each virtual context generates logs independently and sends to specified destinations
Syslog server, console, buffer, SNMP station, etc..
Rate limiting of syslog messages is recommended. Never log to the console using level 7
ACE can log connection setup/teardown at the connection speed
Access-List deny entries are logged
Use the terminal monitor command to display log message when not using console
Useful commands to troubleshoot syslog issues:
show logging statistics show logging history
show logging queue
Make sure logging queue size is set properly
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 19
Basic Configuration to Enable Logging
Enable logging on the ACElogging enable
logging timestamp
logging monitor 4
logging trap 4
logging buffer 4
logging history 4
logging queue 1024
no logging message 111008
It is recommended to disable or change the severity level of some syslog messages. Use logging message syslog_id [level severity_level] command
To enable the logging of connection setup and teardown messages, use the logging fastpath command. Use the logging rate-limit to limit the rate at which the ACE generates messages to the syslog server
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 20
Real-Time ―TCP Dump‖
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 21
Real-Time ―TCP Dump‖
Supportability and analysis of load balanced traffic is a major requirement in today's load balanced environment
ACE can capture real-time packet information for the network traffic that passes through it
The attributes of the packet capture are defined byan ACL
The ACE buffers the captured packets, and you can copy the buffered contents to a file in flash memory on the ACE or to a remote server
User can also display the captured packet information on your console or terminal; capture can also be exported and viewed using Ethereal or Wireshark
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 22
Real-Time ―TCP Dump‖
To enable the packet capture on ACE use the capturecommand
capture c1 interface vlan 211 access-list FILTER bufsize 64
Buffer in Kbytes
(can be circular)
Pre-defined ACL to
identify relevant traffic
Interface to apply
capture
One capture session per context
Capture triggered at flow setup
Capture configured on client interface where flow is received
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 23
Real-Time ―TCP Dump‖
ACE can capture traffic based on a configured access-list and interface
Follow the following procedure to capture traffic on ACE:
1. Specify an ACL
2. Capture on an interface or globally
access-list FILTER line 10 extended permit tcp any any eq www
capture c1 interface vlan 211 access-list FILTER
Show capture status show status and buffer size
ACE/Admin# show capture c1 status
Capture session : c1
Buffer size : 64 K
Circular : no
Buffer usage : 1.00%
Status : stopped
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 24
Real-Time ―TCP Dump‖
Start the capture on the ACE
ACE/Admin# capture c1 start
23:40:37.236868 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 58:
172.16.11.190.443 > 209.165.201.11.1180: S
1389739009:1389739009(0) ack 617249474 win 17408 <mss 1460>
(ttl 255, id 2401, len 44, bad cksum 0!)
23:40:37.239102 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 54:
172.16.11.190.443 > 209.165.201.11.1180: . ack 71 win 17408
(ttl 255, id 2402, len 40, bad cksum 0!)
ACE/Admin# capture c1 stop
To copy the packet capture to disk0: use the copy capture
ACE/Admin# copy capture c1 disk0: c1
Maximum buffer size is 5MB of data
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 25
Traffic Forwarding on ACE
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 26
ACE Load Balancer Policy Lookup Order
There can be many features applied on a given interface, so feature lookup ordering is important
The feature lookup order followed by data path in ACE is as follows:
1. Access-control (permit or deny a packet)
2. Management traffic
3. TCP normalization/connection parameters
4. Server load balancing
5. Fix-ups/application inspection
6. Source NAT
7. Destination NAT
The policy lookup order is implicit, irrespective of the order in which the user configures policies on the interface
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 27
Checking VLAN Configuration
Show interface provides you with valuable information ACE/Admin# show interface vlan 248
vlan248 is up
Hardware type is VLAN
MAC address is 00:16:36:fc:b3:36
Virtual MAC address is 00:0b:fc:fe:1b:02
Mode : routed
IP address is 172.16.10.21 netmask is 255.255.255.0
FT status is active
Description:WAN Side
MTU: 1500 bytes
Last cleared: never
Alias IP address is 172.16.10.23 netmask is 255.255.255.0
Peer IP address is 172.16.10.22 Peer IP netmask is 255.255.255.0
Assigned on the physical port, up on the physical port
499707 unicast packets input, 155702918 bytes
1485258 multicast, 5407 broadcast
0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops
497610 unicast packets output, 46804782 bytes
6 multicast, 8201 broadcast
0 output errors, 0 ignored
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 28
MAC Addresses
Virtual MAC (VMAC) is used for the alias IP, VIP address
Alias IP and Virtual IP (VIP) are associated with a VMAC only if high availability is configured
Active context responds to ARPs for alias IP with VMAC
One unique VMAC per FT Group 00:0b:fc:fe:1b:XX(XX=FT group number in hex)
Packets destined to the VMAC are blocked on standby context
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 29
MAC Addresses
The VMAC is a function of ft-group-id. Therefore different cards must have different ft-group-ids
Use the show interface internal iftable to locate the VMAC
Each ACE supports 1,024 shared VLAN‟s, and uses only one bank of MAC addresses randomly selected at boot time
ACE‟s may select the same address bank so avoid this conflict use the shared-vlan-hostid command
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 30
Admin Context Resource Reservation
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 31
Admin Context Resource Reservation
If Admin context is not configured correctly, Admin could be starved of all resources
Default resource class in ACE has no minimum allocation
When configuring resource allocations in ACE, it is possible to allocate 100% of resources to non-Admin contexts, so that the Admin context is no longer reachable via ICMP, telnet, SNMP, etc
Highly recommended to put some safeguard in place to ensure that the Admin context always receives at least a small percentage of resources
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 32
Admin Context Resource Reservation
Shows starved resources and drops for throughput
ACE/Admin# show resource usage context Admin
Allocation
Resource Current Peak Min Max Denied
-------------------------------------------------------------------------------
Context: Admin
conc-connections 9 9 0 0 0
mgmt-connections 2 12 0 0 0
proxy-connections 0 0 0 0 0
xlates 0 0 0 0 0
bandwidth 0 4715 0 0 3704068
throughput 0 4247 0 0 3704068
mgmt-traffic rate 0 468 0 125000000 0
connection rate 0 7 0 0 8
ssl-connections rate 0 0 0 0 0
mac-miss rate 0 1 0 0 0
inspect-conn rate 0 0 0 0 0
acl-memory 26816 26880 0 0 0
sticky 0 0 0 0 0
regexp 0 0 0 0 0
syslog buffer 1024 4096 0 1024 0
syslog rate 0 7 0 0 118
No resources reserved
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 33
Admin Context Resource Reservation
Suggesting the following reserved resources for Admin
resource-class Admin
limit-resource conc-connections min 5.00 max equal-to-min
limit-resource mgmt-connections min 5.00 max equal-to-min
limit-resource rate bandwidth min 5.00 max equal-to-min
limit-resource rate ssl-connections min 5.00 max equal-to-min
limit-resource rate mgmt-traffic min 5.00 max equal-to-min
limit-resource rate conc-connections min 5.00 max equal-to-min
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 34
Access-Control Lists and ACL Merge
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 35
ACL Merge Process and Enhancements
ACL merge is responsible for merging all the features and generating a single merged list for any given interface. ACL compiler is responsible for programming the merged list into MTrie data structure – “Fast retrieval of data”
ACL memory usage has been optimized to better support incremental changes
The new implementation provides a consistent ACL memory usage during system boot up time and during incremental changes after the system comes up
This feature also provides an early detection of failure if the configuration needs more ACL resources than available
Also, note ACL masks are in 255.255.x.x format (not 0.0.y.y)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 36
View Total Action Nodes
Use the show np 1 access-list resource to view action nodes
ACE/Context3# show np 1 access-list resource
ACL Tree Statistics for Context ID: 3
=======================================
ACL memory max-limit: None
ACL memory guarantee: 0.00 %
MTrie nodes(used/guaranteed/max-limit):
6 / 0 / 262143 (compressed)
2 / 0 / 21999 (uncompressed)
Leaf Head nodes (used/guaranteed/max-limit):
3 / 0 / 262143
Leaf Parameter nodes (used/guaranteed/max-limit):
7 / 0 / 524288
Policy action nodes used: 4
memory consumed: 4696 bytes resource-limited 128 bytes other
4824 bytes total.
min-guarantee: 0 bytes total.
max-limit: 78610432 bytes total, 0 % consumed
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 37
Connection Handling in ACE
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 38
Flow Management
Level of Flow Processing Type of Processing Function
Layer 3 and Layer 4 Balance on first packet Basic Load Balancing
Applies to TCP/UDP for layer 4 rules Source IP Sticky
Applies to all other IP protocols TCP/IP Normalization
Layer 7 TCP Splicing Terminate TCP Connection HTTP Layer 7 rules based on first
request (URL LB)
Buffer request, inspect, LB Cookie Sticky (Persistence)
Create Hardware Shortcut Generic TCP Payload Parsing
Layer 7 Re-proxy TCP Splicing + ability to parse
subsequent HTTP requests within
the same TCP
HTTP Layer 7 rules with HTTP
1.1 connections keepalive
(“persistence rebalance”)
Layer 7 Full-Proxy Fully terminate clients connection SSL Offload
TCP re-use
HTTP 1.1 Pipelining
Protocol Inspection (FTP,SIP)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 39
Internal Mapping of TCP/UDP Flows
TCP and UDP Flows = 2 X Internal Half Flows
ACE/Admin# show conn
conn-id np dir proto vlan source destination stat
-------------+--+----+--------+-----+--------------------------+-------------------------------+---------+
9 1 In TCP 211 209.165.201.11:1867 172.16.11.190:80 ESTAB
6 1 Out TCP 411 192.168.1.11:80 209.165.201.11:1867 ESTAB
Client IP:port VIP Address
Server IP Returning half flow
automatically created for
both TCP and UDP flows
INIT, SYNACK,
ESTAB, CLOSED
SYN_SEEN, SYN_SEEN,
ESTAB, CLOSED
Non TCP shows as ―--‖
Use conn-id
to track flow
through ACE
Check the
Network
Processor
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 40
Troubleshooting Connections
Use the show stats connection command to show connections statistics
Use the clear stats connection command to clear these counters
ACE/Admin# show stats connection
+------------------------------------------+
+------- Connection statistics ------------+
+------------------------------------------+
Total Connections Created : 288232
Total Connections Current : 2
Total Connections Destroyed: 283404
Total Connections Timed-out: 892
Total Connections Failed : 3934
Note: ACE does not destroy connection. These connections are closed correctly!!!
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 41
Troubleshooting Connections
Use the show stats loadbalance command to view the load balance statistics
To clear the load balance statistical information stored in the ACE buffer, use the clear stats loadbalance command
ACE/Admin# show stats loadbalance
+------------------------------------------------------------+
+------- Loadbalance statistics ----------------------+
+------------------------------------------------------------+
Total version mismatch : 0
Total Layer4 decisions : 0
Total Layer4 rejections : 0
Total Layer7 decisions : 24
Total Layer7 rejections : 0
Total Layer4 LB policy misses : 0
Total Layer7 LB policy misses : 0
Total times rserver was unavailable : 0
Total ACL denied : 0
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 42
Troubleshooting Individual Connections
Use the NP and connection ID from ‟show conn‟ command to view the front-end and back-end connection statistics using show np <#> me-stats “-c <connection ID> -v”
ACE/Admin# show np 1 me-stats “-c 4096 –v”
+------------------------------------------------------------+
+------- Individual connection statistics -------------------+
+------------------------------------------------------------+
Connection ID:seq: 4096[0x1000].2
Other ConnID : 8194[0x2002].14
Proxy ConnID : 0[0x0].0
Next Q : 0[0x0]
10.1.1.22:23 -> 12.2.2.14:8739 [RX-NextHop: TX] [TX-NextHop: CP]
Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No
L3 Protocol : IPv4 L4 Protocol : 6
Inbound Flag : 0
Interface Match : Yes
Interface MatchID:24
……… <snip>
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 43
Troubleshooting Individual Connections
To further debug and check if the traffic pattern matches the correct rule, the following command can be used: show np 1 access-list trace vlan <inbound vlan> in protocol <IP protocol #> source <source IP> <source port or „0‟> destination <destination IP> <destination port>
ACE/Admin# show np 1 access-list trace vlan 10 in protocol 6
source 10.10.10.1 0 destination 10.20.30.40 80
<snip> <look for NAT pool ID, vserver ID, etc.>
src nat 0x0 dst nat 0x0 vserver 0x66 fixup 0x0
<snip> <vserver ID here is 0x66 or 102 decimal>
Now, the internal vserver ID 102 can be looked up in the config: ACE/Admin# show cfgmgr internal table l3-rule | inc 102
102 224 249 0 0 DATA_VALID
Internal Policy Map # is 224 and Class Map # is 249: ACE/Admin# show cfgmgr internal table policy-map | inc 224
224 MyPolicy9 0 DATA_VALID
ACE/Admin# show cfgmgr internal table class-map | inc 249
249 MyClass4 0 DATA_VALID
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 44
Troubleshooting VIPACE/Admin# show service-policy client-vips detail
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 211
service-policy: client-vips
class: VIP-HTTPS
VIP Address: Protocol: Port:
172.16.11.190 tcp eq 443
loadbalance:
L7 loadbalance policy: HTTPS-POLICY
VIP Route Metric : 77
VIP Route Advertise : DISABLED
VIP ICMP Reply : ENABLED-WHEN-ACTIVE
VIP State: INSERVICE
curr conns : 22 , hit count : 22
dropped conns : 0
client pkt count : 0 , client byte count: 0
server pkt count : 0 , server byte count: 0
max-conn-limit : 0 , drop-count : 0
conn-rate-limit : 0 , drop-count : 0
bandwidth-rate-limit : 0 , drop-count : 0
L7 Loadbalance policy : HTTPS-POLICY
class/match : class-default
LB action :
primary serverfarm: backend-ssl
backup serverfarm : -
hit count : 22
dropped conns : 0
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 45
Troubleshooting Serverfarm
Use this command for checking server status and load
ACE/Admin# show serverfarm HTTPS-FARM detail
serverfarm : HTTPS-FARM, type: HOST
total rservers : 4
active rservers: 4
description : -
state : ACTIVE
predictor : ROUNDROBIN
failaction : -
back-inservice : 0
partial-threshold : 0
num times failover : 0
num times back inservice : 0
total conn-dropcount : 0
----------connections-----------
real weight state current total failures
---+---------------------+--------+---------------------+-----------+-----
-
rserver: linux-1
192.168.1.11:0 8 OPERATIONAL 10 1000 1
max-conns : - , out-of-rotation count : -
min-conns : -
conn-rate-limit : - , out-of-rotation count : -
bandwidth-rate-limit : - , out-of-rotation count : -
retcode out-of-rotation count : -
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 46
Layer 7 Troubleshooting
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 47
Layer 7 Policy Hits
Expanding the show service-policy using the detail option to provide hit count for layer 7 matches
ACE/Admin# show service-policy client-vips detail
Status : ACTIVE
Description: -
-----------------------------------------
Interface: vlan 211
service-policy: client-vips
<snip>
L7 Loadbalance policy : pslb
class-map : curl1
LB action :
serverfarm: s1
hit count : 3
dropped conns : 0
class-map : curl2
LB action :
serverfarm: s2
hit count : 0
dropped conns : 0
Shows hit count for layer 7
load balanced policy
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 48
Match URL Hit Count
show service-policy url-summary option to provide visibility on which match http url‟s are getting hits
ACE/Admin# show service-policy url-summary
Service-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: VH-01
match http url /ACCOUNTING/.* hit: 42
Service-Policy: VIRTUAL-HOSTING-02 L3-Class: WEB-SSL L7-Class: VH-02
match http url /BUSINESS/.* hit: 93
match http url /SALES/.* hit: 102
match http url /SPECIAL/.* hit: 67
match http url /BUSINESSOBJECTS/.* hit: 78
match http url /CUSTOMERS/.* hit: 84
Use the show service-policy <service-policy-name> class-map <L3-class map-name> url-summary for more granularity
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 49
Troubleshooting HTTP
To effectively troubleshoot HTTP use the show stat http commandACE/Admin# show stats http
+------------------------------------------+
+-------------- HTTP statistics -----------+
+------------------------------------------+
LB parse result msgs sent : 6288 , TCP data msgs sent : 9143
Inspect parse result msgs : 0 , SSL data msgs sent : 6041
TCP fin/rst msgs sent : 135 , Bounced fin/rst msgs sent: 19
SSL fin/rst msgs sent : 13 , Unproxy msgs sent : 0
Drain msgs sent : 3107 , Particles read : 37917
Reuse msgs sent : 1539 , HTTP requests : 3145
Reproxied requests : 0 , Headers removed : 1549
Headers inserted : 1598 , HTTP redirects : 2
HTTP chunks : 0 , Pipelined requests : 0
HTTP unproxy conns : 0 , Pipeline flushes : 0
Whitespace appends : 0 , Second pass parsing : 0
Response entries recycled : 3032 , Analysis errors : 0
Header insert errors : 15 , Max parselen errors : 0
Static parse errors : 9 , Resource errors : 0
Invalid path errors : 0 , Bad HTTP version errors : 0
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 50
Troubleshooting HTTP Cookies
ACE parses HTTP requests for cookies with the name given in the configuration and can skip a certain number of bytes and look for another specific number of bytes.
If the cookie is not found, then the ACE looks for a string in the URL, starting with one of the characters /?&#+ and followed by a "=", then parses that value.
If no cookie or HTTP URL cookie exists ACE defaults to the predictor for that farm
ACE can parse HTTP headers (includes cookies) up to 64KB (default header max parse length is 4KB)
Make sure that sticky timeout (note this is more like an idle timeout) matches the session timeout on the application
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 51
Troubleshooting TCP Reuse
When using TCP connection reuse,"Connection: keep-alive" is inserted and "Connection: close" is removed from the clients HTTP request, to avoid closing the server connection early
User needs to configure Source NAT in the policy map when using TCP connection re-use
Use the show stats http | include Reuse counters to check if see if TCP Reuse is in effect
ACE/Admin# show stats http | include Reuse
Reuse msgs sent : 1 , HTTP requests : 4
„sh conn detail‟ will also show information about server side connection reuse
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 52
Troubleshooting HTTP Compression
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 53
HTTP Compression Overview
ACE uses Cavium Octeon zip engine
Implement deflate block as defined in RFC 1951
History buffer is supported to achieve better compression ratio
Support two output file formats. GZIP (RFC1952) or X-
GZIP (RFC2616) and ZLIB (aka DEFLATE) RFC1950
Compression is used with HTTP connection only
Compression only supports HTTP 1.1 protocol
No decompression support
Mainly for text based content (html,xml…)
Feature available on ACE 4710 and ACE30
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 54
ACE Compression Traffic Flow Example
2. ACE rewritesClient‟s request
GET / HTTP/1.1
Accept-Encoding: gzip,
deflate
1. Request before ACE
GET / HTTP/1.1
Accept-Encoding: identity
Request after ACE
4. ACE Inspects response
HTTP/1.1 200 OK
Content-type: text/html
Content-Encoding: deflate
Transfer-Encoding:
chunked
6. Response after ACEServer sends uncompressed HTTP payload of 5963 bytes
7. Client receives compressed HTTP payload 2577 bytes
Cisco ACEClientLAN
HTTP/1.1 200 OK
Content-type: text/html
Content-Length: 5963
3. Response before ACE
5. ACECompresses
Response
Server
WAN
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 55
Debugging HTTP Compression
Check the following for errorsFrom client side:
1. Accept-Encoding is not present or has invalid type
2. User-Agent is being excluded from the configuration
3. HTTP version is not 1.1 or higher
From server side
4. Invalid HTTP response header
5. HTTP response code not 200
6. Content type is not allowed
7. Content length is too small
8. Chunk encoding has invalid format
Additional details using show np x me-stat “-s http”
Get request from client:
GET HTTP/1.1
Host: www.yahoo.com
User-Agent: Mozilla/5.0 Windows; U; Windows NT 5.1;
Accept: text/html,application/xhtml+xml,
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 56
Troubleshooting Secure Socket Layer (SSL)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 57
Troubleshooting SSL
Configuration of SSL on ACE is relatively simple. However if you experience an issue, how to troubleshoot?
Make sure the certificate and key used in ssl-proxy are valid. Use the crypto verify command
ACE/Admin# crypto verify RSA2048.key RSA2048.cert
Keypair in RSA2048.key matches certificate in RSA2048.cert
Check the size and location of the key. Use the show crypto key command
ACE/Admin# show crypt key all
Filename Bit Size Type
-------- -------- ----
RSA2048.key 2048 RSA
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 58
Troubleshooting SSL
Review the certificate details. Use the show crypto certificate command
ACE/Admin# show crypto certificate cisco-sample-cert
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
ad:e4:e2:f1:50:b7:ce:bd
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=IN, ST=KA, L=BLR, O=CISCO, OU=AAAA, CN=SSL-TEST
Validity
Not Before: Apr 3 09:50:55 2009 GMT
Not After : Apr 1 09:50:55 2019 GMT
Subject: C=IN, ST=KA, L=BLR, O=CISCO, OU=AAAA, CN=SSL-TEST
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:cf:a2:60:66:5b:ce:b6:38:6f:94:df:0d:1c:61:
26:af:7a:05:49:ed:8d:93:3b
Exponent: 65537 (0x10001)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 59
Troubleshooting SSL – CRL Download
Check to make sure you can download the CRL
ACE/Admin# show crypto crl test2 detail
test2:
URL: http://119.60.60.23/test.crl
Last Downloaded (Cached): Sat Aug 8 16:14:24 2009 UTC
Total Number Of Download Attempts: 1
Failed Download Attempts: 0
Successful Loads: 1 Failed Loads: 0
Hours since Last Load: 0 No IP Addr Resolutions: 0
Host Timeouts: 0 Next Update Invalid: 0
Next Update Expired: 0 Bad Signature: 0
CRL Found-Failed to load: 0 File Not Found: 0
Memory Outage failures: 0 Cache Limit failures: 0
Conn failures: 0 Internal failures: 0
Not Eligible for download: 3 HTTP Read failures: 0
HTTP Write failures: 0
To look for all best-effort CRLs in the system and their
download status, use show crypto crl best-effort
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 60
Advanced SSL Debugging
This command provides the current crypto statistics ACE/Admin# sh np 1 me-stats "-s crypto”
Crypto Statistics: (Current)
------------------
Internal Error: 172 0
ARC4 operations: 376572 0
TCP msgs received: 285260 0
APP msgs received: 235151 0
Nitrox messages forwarded to XScale: 381041 0
SSL ctx allocated: 47758 0
SSL ctx freed: 47758 0
SSL received bytes: 61070430 0
SSL transmitted bytes: 283256220 0
SSL received application bytes: 7679113 0
SSL transmitted application bytes: 275120867 0
SSL received non-application bytes: 53391317 0
SSL transmitted non-application bytes: 3292887 0
Bulk flush operations: 95037 0
ME records sent to XScale: 285808 0
ME records received from XScale: 47723 0
ME hw responses: 471516 0
First segments received: 47400 0
Handshake failure alert: 94 0
CM close: 446 0
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 61
Advanced SSL Debugging
The show stats crypto server command provides statistics of the SSL handshake
ACE/Admin# show stats crypto server
+---- Crypto server termination statistics -----+
+------- Crypto server alert statistics --------+
+--- Crypto server authentication statistics ---+
+------- Crypto server cipher statistics -------+
+------ Crypto server redirect statistics ------+
+---- Crypto server header insert statistics ---+
Other useful commands:
show crypto hardware (applies only to ACE20)
show crypto cdp-errors
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 62
Health Monitoring on ACE
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 63
Fundamentals for ACE probing
ACE probes are fundamental to the system. It is key to not oversubscribe the ACE health monitoring system
Note both the primary and standby ACE send out probes; also, the interface IP is used and not the alias IP
Some key probe parameters:
faildetect <number of consecutive failed probes>
interval <time interval between probes>
open <timeout for completing 3-way TCP handshake>
receive <timeout for receiving a probe response from the server>
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 64
Fundamentals for ACE probing
Use the show resource internal socket to determine how many sockets ACE has open. This is an Admin command
ACE/Admin# show resource internal socket
Application MaxLimit Current Creates Frees
--------------------------------------------------------------
SYSTEM 4000 0 0 0
CRITICAL 50 0 0 0
AAA 256 0 0 0
MGMT 256 0 0 0
XINETD 512 1 12 11
HEALTH_MON 2500 532 193494 192962
USER_TCL 200 0 0 0
SYSLOG 256 10 14 4
VSH 256 0 0 0
OverAll - 650 194812 194162
Non Reg App Usage: 107
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 65
Health Monitoring Process
If you see probe related issues, check the health monitoring process. The show proc cpu command provides useful information
ACE/Admin# show proc cpu
CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%
PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process
972 1072965 613352 1749 35.9 18.5% 21.67% 20.90% arp_mgr
HM process is only consuming 1.40%. Why is the control plane CPU running at 30%? Check what process is consuming CPU
ACE/Admin# show proc cpu | inc hm
CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%
PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process
987 90257 57805 1561 0.0 1.40% 1.46% 1.43% hm
988 90198 58952 1530 0.0 1.49% 1.49% 1.44% hm
989 851 2947 288 0.0 0.0 % 0.1 % 0.0 % hm
990 0 2 56 0.0 0.0 % 0.0 % 0.0 % hm
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 66
Health Monitoring on ACE
Use the show probe detail command to determine the status of the probe or possible last failure
ACE/Admin# show probe detail
--------------------- probe results --------------------
probe association probed-address probes failed passed health
------------------- ---------------+----------+----------+----------+-------
rserver : CAS1
10.7.53.55 24 24 0 FAILED
Socket state : CLOSED
No. Passed states : 0 No. Failed states : 1
No. Probes skipped : 0 Last status code : 403
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : Received invalid status code
Last probe time : Wed Nov 25 18:48:16 2009
Last fail time : Wed Nov 25 18:25:16 2009
Last active time : Never
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 67
High Availability on ACE
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 68
High Availability Basic building blocks
FT PEER
Only one FT peer per ACE device
1:1 peer relationship
FT GROUPOne FT group per ACE virtual context
FT VLAN
Designated VLAN between the redundant peers
All HA related traffic sent over this VLAN
FT VLAN can be trunked between two Catalyst 6500 Chassis
Should not be used for normal traffic
Admin Context
Context A
Context B
Context A
Context B
ACE2 (FT PEER)
FT VLAN
FT Group 2
FT Group 3
ACE1 (FT PEER)
FT Group 1
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 69
High Availability Control Traffic
TCP Connection between FT Peers
State Machine (Election, Preempt, Relinquish)
Configuration sync
State Sync for ARP
Heartbeats between FT peers
Heartbeats are sent over UDP
Monitors the health of the peer
Heartbeat interval and count are configurable
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 70
FT Heartbeats
If heartbeats missed is increasing, heartbeats are not reaching the peer. Possibility for both ACE‟s to go Active/Active
ACE/Admin# sh ft stats
HA Heartbeat Statistics
------------------------
Number of Heartbeats Sent : 1095573
Number of Heartbeats Received : 1092586
Number of Heartbeats Missed : 2987
Number of Unidirectional HB's Received : 2640
Number of HB Timeout Mismatches : 0
Num of Peer Up Events Sent : 1
Num of Peer Down Events Sent : 1
Successive HB's miss Intervals counter : 0
Successive Uni HB's recv counter : 0
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 71
ACE High Availability State Machine
STANDBY_BULK State
ARP Sync (knob to turn on/off)
Connection Table Sync
Sticky Database Sync (knob to turn on/off)
STANDBY_HOT State
Standby FT group member is ready to take over
Incremental Configuration Sync from Active to Standy
Incremental State Sync from Active to Standby
STANDBY_COLD State
Due to error during Config Sync or SSL certs mismatch
No Config or State Sync happens from Active to Standby‟
STANDBY_WARM State
Major version mismatch between peers (example 2.x and 4.x)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 72
ACE High Availability State Machine
Mismatch in software version
FT Peer may become INCOMPATIBLE
Could result in ACTIVE ACTIVE state on both FT group members
Mismatch in Virtual Context Licenses
Configuration Sync (all types) for Admin context is disabled
State Sync for Admin context will continue to happen
For matching user contexts – Configuration State Sync will work
Mismatch in Other Licenses
Configuration and State Sync will work
After switchover, new Active will handle traffic as per its licenses
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 73
Preventing Active-Active Scenarios
When no heartbeat is received, ACE can use the Query Vlan to check the HA status
ACE tries to do a ping to the destination via the Query VLAN
If ping fails, the Standby will transition to the ACTIVE state
If ping succeeds, the Standby will transition to a STANDBY_COLD state
To configure a query interface, enter the following:
ACE/Admin(config-ft-peer)# query-interface vlan 110
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 74
More debugging commands
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 75
Additional Debugging
Some more ACE debugging commands
show np <#> me-stats -cpu
show np <#> me-stats –Q
show np <#> me-stats “-s fp”
show np <#> me-stats “-s tcp”
show np <#> me-stats “-s icm”
show np <#> me-stats “-s ocm”
show proc cpu
show netio stats
Show service-policy summary
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 767
6
Receive 25 Cisco Preferred Access points for each session evaluation you complete.
Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don‟t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.
Complete Your Online Session Evaluation
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 777
7
Visit the Cisco Store for Related Titles
http://theciscostores.com
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 78
Recommended Reading
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 80
Appendix and Additional Troubleshooting Information
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 81
Additional Information
Layer 4 flow setup
Layer 7 flow setup
TCP Connection States
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 82
Layer 4 Flow Setup
SYN
SYN_ACK
Shortcut
ACK
Shortcut
Data
Shortcut
Data
Shortcut
Matches Existing
Flow
Rewrites L2/L3/L4
Matches VIP
Selects Server
Rewrites
L2/L3/L4
Basic Load Balancing
Source IP sticky
TCP/IP Normalization
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 83
Layer 7 Flow SetupClient Connects to “L7” VIP
SYN
Starts
BufferingACK
Data
ACK‘s Client Packets
Keeps Buffering
Matches VIP w/L7
logic
Chooses SEQ #
Replies w/SYN_ACK
HTTP L7 rules on first request
(cookie sticky, URL parsing, …)
Generic TCP payload parsing
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 84
Layer 7 Flow Setup—ContinueACE Establishes Connection to Server
Data
SYN_ACK
Empties Buffer
Sends Data to Server
Acts as Client
Does Not Forward
SYN_ACK
Parses the Data
Selects Server
Initiates TCP
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 85
Layer 7 Flow Setup—Continue ACE Splices the Flows (UNPROXY)
ACK
Data
Shortcut
ACK
Shortcut
Data
Shortcut
Matches Existing Flow
Rewrites L2/L3/L4
and SEQ/ACK
Does Not Forward ACK
Ready to
Splice the Flows
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 86
Layer 7 Flow SetupACE Reproxies the Connection
ACK
Data
ACK
Data
Shortcut
…ACK
…Shortcut
Shortcut
ShortcutData
REPROXY
ACK‘s GET & Buffer…
HTTP L7 rules with HTTP 1.1
connection keepalive
(―persistence rebalance‖)
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 87
Layer 7 Flow SetupACE Acts as a Full Proxy
Fu
ll Pro
xy
Ind
ep
en
den
t clie
nt &
se
rve
r co
nn
ec
tion
s
SYNSYN_ACK
ACK
DataGET/HTTP 1.1
ACK SYN
SYN_ACKACK
Data—GET
ACK
ACKData
DataHTTP/1.1 200 OKHTTP/1.1 200 OK
Client connection Server connection
… …
SSL offload
TCP re-use
Protocol inspections
HTTP 1.1 pipelining
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 88
TCP Connection States
L4 TCP Connections
SYNSEEN (Client „SYN‟ received)
INIT (Server side half flow initialized)
SYNACK („SYN ACK‟ sent by server)
ESTAB (Client and Server; TCP Handshake completed)
L7 TCP Connections
SYNSEEN (Client „SYN‟ received)
ESTAB (Client side TCP Handshake completed; „SYN ACK‟ sent by ACE, Client ACK received)
ESTAB (Server side TCP Handshake completed from ACE after L7 data received from the client and parsed)
CLOSED (Client or Server „FIN ACK‟ followed by ACK)