Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to collaborate on Cisco product documentation. You do not need to log in to read the text. However, you must log in to edit the text. Select the "edit" tab to edit an article or select the "discussion" tab to submit questions or comments about documentation content. See Terms of Use and About DocWiki for more information about Cisco DocWiki. Click here to go to the Cisco ACE Module documentation on www.cisco.com. Click here to go to the Cisco ACE Appliance documentation on www.cisco.com. Contents 1 Audience • 2 Organization • 3 Creating a PDF of the ACE Troubleshooting Wiki • 4 Related Documentation 4.1 ACE Module Documentation ♦ 4.2 ACE Appliance Documentation ♦ • This article provides a systematic approach to identifying and remedying problems that may arise as you use your ACE over a period of time. This guide is not intended to replace configuration best practices or to be an all-inclusive guide for every application. Rather, it is an attempt to provide you with the knowledge and skills necessary to correct the most common issues that you may encounter. Audience This article is intended for all trained network administrators who have experience with the configuration and maintenance of the ACE. Organization This article consists of the following major sections: Overview of ACE Troubleshooting Understanding the ACE Module Architecture and Traffic Flow Preliminary ACE Troubleshooting Troubleshooting ACE Boot Issues Troubleshooting with ACE Logging Cisco Application Control Engine (ACE) Troubleshooting Guide 07/26/11 1
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Welcome to Cisco DocWiki. We encourage registered Cisco.com users to contribute to this wiki to collaborateon Cisco product documentation. You do not need to log in to read the text. However, you must log in to edit thetext. Select the "edit" tab to edit an article or select the "discussion" tab to submit questions or comments aboutdocumentation content.
See Terms of Use and About DocWiki for more information about Cisco DocWiki.
Click here to go to the Cisco ACE Module documentation on www.cisco.com.
Click here to go to the Cisco ACE Appliance documentation on www.cisco.com.
Contents
1 Audience• 2 Organization• 3 Creating a PDF of the ACETroubleshooting Wiki
•
4 Related Documentation4.1 ACE Module Documentation♦ 4.2 ACE Appliance Documentation♦
•
This article provides a systematic approach to identifying and remedying problems that may arise as you use yourACE over a period of time. This guide is not intended to replace configuration best practices or to be anall-inclusive guide for every application. Rather, it is an attempt to provide you with the knowledge and skillsnecessary to correct the most common issues that you may encounter.
Audience
This article is intended for all trained network administrators who have experience with the configuration andmaintenance of the ACE.
Organization
This article consists of the following major sections:
Overview of ACE Troubleshooting
Understanding the ACE Module Architecture and Traffic Flow
Preliminary ACE Troubleshooting
Troubleshooting ACE Boot Issues
Troubleshooting with ACE Logging
Cisco Application Control Engine (ACE) Troubleshooting Guide
You can create a PDF of one or more articles in this wiki, including the entire ACE Troubleshooting Wiki. Fordetails, see the Creating a PDF article.
Related Documentation
ACE Module Documentation
Customer Documentation for the Cisco Application Control Engine (ACE) Module• Cisco Application Control Engine (ACE) Configuration Examples on DocWiki•
Cisco Application Control Engine (ACE) Troubleshooting Guide
This article introduces the basic concepts, methodology, and general troubleshooting guidelines for problems thatmay occur when you configure and use your ACE.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of the ACE Troubleshooting Process• 2 Verifying the ACE Image• 3 Enabling ACE Logging• 4 Gathering ACE Troubleshooting Information
4.1 Rebooting the ACE♦ 4.2 Using show Commands♦ 4.3 Capturing Packets in Real Time♦ 4.4 Copying Core Dumps♦ 4.5 After Gathering Troubleshooting Information♦
•
5 Verifying the Physical Connectivity Between the ACE and the EndHosts
•
6 Verifying the ACE Layer 2 Connectivity• 7 Verifying the ACE Layer 3 Connectivity• 8 Contacting Cisco Technical Support•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 4
Overview of the ACE Troubleshooting Process
To troubleshoot your ACE, follow these general guidelines:
Maintain a consistent and recommended software version across all your ACEs. See the "Verifying theACE Image" section.
1.
See the ACE module release notes for your software version for the latest features, operatingconsiderations, caveats, and CLI command changes.
2.
Before you introduce configuration changes, use the ACE checkpoint feature to bookmark a known goodconfiguration and save your configuration. If you run into problems with the new configuration, you canroll back the new configuration to the known good configuration. See the Cisco Application ControlEngine Module Administration Guide. Troubleshoot new configuration changes immediately after addingthem.
3.
Verify that your configuration is correct for your network application. Make any required changes to therunning-config file, and then test the configuration. If it is satisfactory, save it to the startup-config fileusing the copy running-config startup-config command for a particular virtual context or the writememory command from the Admin context to copy all running-config files in every virtual context totheir respective startup-config files.
4.
Enable system message logging. See the "Enabling ACE Logging" section.5. Gather information that defines the specific symptoms. See the "Gathering ACE TroubleshootingInformation" section.
6.
Verify the physical connectivity between your device and end devices. See the "Verifying the PhysicalConnectivity Between the ACE and the End Hosts" section.
7.
Verify the ACE Layer 2 connectivity. See the "Verifying the ACE Layer 2 Connectivity" section.8. Verify the ACE end-to-end connectivity and the routing configuration. See the "Verifying the ACE Layer3 Connectivity" section.
9.
After you have determined that your troubleshooting attempts have not resolved the problem, contact theCisco Technical Assistance Center (TAC) or your technical support representative. See the "ContactingCisco Technical Support" section.
10.
Verifying the ACE Image
To display the version of the software image and the image filename that is currently running in your ACE, enterthe following command:
ACE_module5/Admin# show versionCisco Application Control Software (ACSW)TAC support: http://www.cisco.com/tacCopyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned byother third parties and are used and distributed under license.Some parts of this software are covered under the GNU PublicLicense. A copy of the license is available athttp://www.gnu.org/licenses/gpl.html.
Software
Cisco Application Control Engine (ACE) Troubleshooting Guide
configuration register: 0x1ACE_module5 kernel uptime is 4 days 22 hours 42 minute(s) 41 second(s)
This command provides other useful information, for example:
Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5)• Available control plane memory• Last boot reason• Configuration register (confreg) value (0x0 boot to rommon, 0x1 boot using boot string)• ACE uptime•
Enabling ACE Logging
To enable logging on the ACE module and to send system logging (syslog) messages to the monitor, enter thefollowing commands:
Note: Use the terminal no monitor command to stop viewing log messages in your remote session.
For more information about logging, see the "Troubleshooting with ACE Logging" section.
Gathering ACE Troubleshooting Information
The following sections recommend ways to gather information that is relevant to the problem that is occurring.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 6
Rebooting the ACE
Do not reboot the ACE unless it is absolutely necessary. Some information that is important to troubleshootingyour problem may not survive a reboot. Try to gather as much information as possible before rebooting.
Using show Commands
You can use a number of show commands in Exec mode to gather information specific to the symptoms you areobserving in your ACE. In most cases, you can gather the information you need to troubleshoot the ACE byentering the show tech-support command. This command runs many show commands that are useful fortroubleshooting the ACE. You can redirect the output of the show tech-support command to one the followingdestinations:
ACE_module5/Admin# show tech-support > ?<File> Name of file to redirect stdout.disk0: Enter the URI to redirect the output.ftp: Enter the URI to redirect the output.sftp: Enter the URI to redirect the output.tftp: Enter the URI to redirect the output.volatile: Enter the URI to redirect the output.
Capturing Packets in Real Time
Capturing packets (sometimes referred to as a "TCP dump") is a useful aid in troubleshooting connectivityproblems with the ACE or for monitoring suspicious activity. The ACE can track packet information for networktraffic that passes through the ACE. The attributes of the packet are defined by an ACL. The ACE buffers thecaptured packets, and you can copy the buffered contents to a file in Flash memory on the ACE or to a remoteserver. You can also display the captured packet information on your console or terminal.
The ACE captures packets subject to the following guidelines:
One capture session is used per context• Capture is triggered at flow setup• Capture is configured on the client interface where the flow is received•
Note: Probe traffic will not hit a security ACL, so ACLs cannot control the capture of those packets.Therefore, probe traffic cannot be captured by the packet capture utility.
If possible, you should capture packets using the ACE packet capturing utility before and after symptoms appear.Save the packet captures to a file.
To capture packets in real time, follow these steps:
1. Create an ACL for packet capturing or use an existing ACL if it meets the packet capture requirements byentering the following command:
ACE_module5/Admin(config)# access-list FILTER line 10 extended permit tcp any any eq wwwACE_module5/Admin# exit
2. Enter the capture command, for example:
Cisco Application Control Engine (ACE) Troubleshooting Guide
Note: Ensure that the ACL you specify in the capture command is for an input interface. If you configure thepacket capture on the output interface, the ACE will fail to match any packets.
3. Display the capture status to determine the capture status and the buffer size by entering the followingcommand:
Notice that the capture has not started yet. The default buffer size is 64 KB. You can specify a maximum of buffersize of 5000 KB and you can specify a circular buffer.
4. Start the packet capture on the ACE by entering the following command:
ACE_module5/Admin# capture CAPTURE1 start11:56:15.354930 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 62: 209.165.201.10.4144 > 172.16.1.100.80: S [bad tcp cksum 2aee!] 805889668:805889668(0) win 64240 <mss 1460,nop,nop,nop,nop> (DF) (ttl 127, id 2355, len 48)11:56:15.355257 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 62: 209.165.201.10.4144 > 192.168.1.11.80: S [bad tcp cksum c6d3!] 1247081510:1247081510(0) win 64240 <mss 1460,nop,nop,nop,nop> (DF) (ttl 127, id 2355, len 48, bad cksum aa70!)11:56:15.355669 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 58: 192.168.1.11.80 > 209.165.201.10.4144: S [tcp sum ok] 1187651879:1187651879(0) ack 1247081511 win 5840 <mss 1460> (DF) (ttl 64, id 0, len 44)11:56:15.355979 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 58: 172.16.1.100.80 > 209.165.201.10.4144: S [bad tcp cksum 641b!] 746460037:746460037(0) ack 805889669 win 5840 <mss 1460> (DF) (ttl 64, id 0, len 44, bad cksum de68!)11:56:15.356442 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 1 win 64240 (DF) (ttl 127, id 2356, len 40)11:56:15.356839 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9be4!] ack 1 win 64240 (DF) (ttl 127, id 2356, len 40, bad cksum aa77!)11:56:15.357203 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 494: 209.165.201.10.4144 > 172.16.1.100.80: P [tcp sum ok] 1:441(440) ack 1 win 64240 (DF) (ttl 127, id 2357, len 480)11:56:15.357918 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 494: 209.165.201.10.4144 > 192.168.1.11.80: P [bad tcp cksum 9be4!] 1:441(440) ack 1 win 64240 (DF) (ttl 127, id 2357, len 480, bad cksum a8be!)11:56:15.358436 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack 441 win 6432 (DF) (ttl 64, id 59820, len 40)11:56:15.358582 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641b!] ack 441 win 6432 (DF) (ttl 64, id 59820, len 40, bad cksum f4bf!)11:56:15.358822 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 272: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 1:219(218) ack 441 win 6432 (DF) (ttl 64, id 59822, len 258)11:56:15.359106 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 272: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 641b!] 1:219(218) ack 441 win 6432 (DF) (ttl 64, id 59822, len 258, bad cksum f3e3!)11:56:15.359391 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 407: 192.168.1.11.80 > 209.165.201.10.4144: P [tcp sum ok] 219:572(353) ack 441 win 6432 (DF) (ttl 64, id 59824, len 393)11:56:15.359751 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 407: 172.16.1.100.80 > 209.165.201.10.4144: P [bad tcp cksum 641b!] 219:572(353) ack 441 win 6432 (DF) (ttl 64, id 59824, len 393, bad cksum f35a!)11:56:15.360101 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: F [tcp sum ok] 572:572(0) ack 441 win 6432 (DF) (ttl 64, id 59826, len 40)11:56:15.360238 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: F [bad tcp cksum 641b!] 572:572(0) ack 441 win 6432 (DF) (ttl 64, id 59826, len 40, bad cksum f4b9!)11:56:15.360378 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 572 win 63669 (DF) (ttl 127, id 2358, len 40)11:56:15.360523 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9be4!] ack 572 win 63669 (DF) (ttl 127, id 2358, len 40, bad cksum aa75!)11:56:15.360686 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: . [tcp sum ok] ack 573 win 63669 (DF) (ttl 127, id 2359, len 40)11:56:15.360831 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: . [bad tcp cksum 9be4!] ack 573 win 63669 (DF) (ttl 127, id 2359, len 40, bad cksum aa74!)11:56:15.360973 0:1c:f9:9:18:0 0:b:fc:fe:1b:1 0800 56: 209.165.201.10.4144 > 172.16.1.100.80: F [tcp sum ok] 441:441(0) ack 573 win 63669 (DF) (ttl 127, id 2362, len 40)11:56:15.361130 0:b:fc:fe:1b:1 0:c:29:f3:cd:e6 0800 56: 209.165.201.10.4144 > 192.168.1.11.80: F [bad tcp cksum 9be4!] 441:441(0) ack 573 win 63669 (DF) (ttl 127, id 2362, len 40, bad cksum aa71!)11:56:15.361290 0:c:29:f3:cd:e6 0:18:b9:a6:89:d 0800 56: 192.168.1.11.80 > 209.165.201.10.4144: . [tcp sum ok] ack 442 win 6432 (DF) (ttl 64, id 59828, len 40)11:56:15.361436 0:b:fc:fe:1b:1 0:1c:f9:9:18:0 0800 56: 172.16.1.100.80 > 209.165.201.10.4144: . [bad tcp cksum 641b!] ack 442 win 6432 (DF) (ttl 64, id 59828, len 40, bad cksum f4b7!)
ACE_module5/Admin# capture CAPTURE1 stop
5. Copy the packet capture to disk0: by entering the following command:
Note: If you view the ACE capture file in a third-party sniffer (for example, Wireshark), you will notice onlythe messages or type PKT_RCV and PKT_XMT are displayed. This situation is expected because thesniffer is not aware of the ACE's internal messaging.
Copying Core Dumps
If the ACE fails with a core dump, the core dump files may contain useful information. The core dump files residein the core: directory. To view the contents of the core: directory, enter the following command:
ACE_module5/Admin# dir core:
123589 Feb 22 00:34:20 2009 qnx_1_mecore_log.999.tar.gz 30361 Feb 22 00:34:22 2009 ixp1_crash.txt
Usage for core: filesystem 153950 bytes total used 202943138 bytes free 203097088 total bytes
You can copy the contents of the core: directory to several locations by using the copy core: command. Thesyntax of this command is as follows:
The ACE provides core dumps for both the control plane and the data plane. Each core dump file contains thefollowing information:
Version• Time of failure• Number of CPUs• Current CPU• BKL status• IRQ lock status• Buffers•
After Gathering Troubleshooting Information
After you have gathered all the above information, be prepared to send the information to your customer servicerepresentative or TAC. You can send the information in the following ways:
FTP• SFTP• TFTP•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 11
Verifying the Physical Connectivity Between the ACE and theEnd Hosts
To verify the physical connectivity of the ACE, follow these steps:
Check all cable connections on the Catalyst 6500 series switch or Cisco 7600 series router that mayimpact the ACE.
1.
Use the extended ping command to send an ICMP Echo request to the end devices.2.
ACE_module5/Admin# pingTarget IP address: 10.1.1.2Repeat count [5]: 4Datagram size [100]: 200Timeout in seconds [2]: 10Extended commands [n]: 4Pinging 10.1.1.2 with timeout = 10, count = 4, size = 200 ....
Response from 10.1.1.2 : seq 1 time 0.494 msResponse from 10.1.1.2 : seq 2 time 0.367 msResponse from 10.1.1.2 : seq 3 time 0.264 msResponse from 10.1.1.2 : seq 4 time 0.237 ms4 packet sent, 4 responses received, 0% packet loss
If a host is one hop away and you are unable to reach the host, then ping the intermediary gateway. If the gatewayis not reachable, enter the show ip route command and check to make sure that the correct route is displayed. Forexample, enter:
ACE_module5/Admin# show ip route
Routing Table for Context Admin (RouteId 0)
Codes: H - host, I - interface S - static, N - nat A - need arp resolve, E - ecmp
Destination Gateway Interface Flags------------------------------------------------------------------------0.0.0.0 10.2.2.1 vlan130 S [0xc]10.2.2.0/24 0.0.0.0 vlan130 IA [0x30]172.27.15.0/24 0.0.0.0 vlan100 IA [0x30]172.27.16.0/24 0.0.0.0 vlan200 IA [0x30]172.19.110.0/26 0.0.0.0 vlan55 IA [0x30]172.27.16.16/29 0.0.0.0 vlan200 N [0x280]172.27.16.33/32 0.0.0.0 vlan100 N [0x280]
Total route entries = 7
If necessary, enter a static route for the gateway.
Verifying the ACE Layer 2 Connectivity
To verify the Layer 2 connectivity of the ACE, follow these steps:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 12
1. Verify that the ARP table is populated with the IP addresses and corresponding MAC addresses of the ACE,the gateway, the local interface, and other IPs that the ACE has learned.
2. Verify that the ACE is connected to the switch fabric of the Catalyst 6500 series switch or the Cisco 7600series router. The ACE uses a 10-Gigabit Ethernet switch fabric interface (SFI) to connect to the chassisbackplane as opposed to the CSM, which uses a port channel. The ACE uses the following format for thisinterface:
Te<slot>/1
For example, if the ACE is in slot 5, you can see the status of the backplane connection by entering the followingcommand on the Catalyst 6500 series switch or the Cisco 7600 series router:
cat6k# show interface te5/1 status
Port Name Status Vlan Duplex Speed TypeTe5/1 connected trunk full 10G MultiService Module
If there is no output from this command, then either the ACE is not installed properly or the ACE is powereddown.
3. Verify the association of the ACE MAC entries with the allocated VLAN interfaces. Enter the followingcommand at the Supervisor CLI:
cat6k# show mac-address-table dynamicLegend: * - primary entry age - seconds since last seen n/a - not available
vlan mac address type learn age ports------+----------------+--------+-----+----------+--------------------------...* 130 0018.b9a6.9115 dynamic Yes 40 Te5/1 <------- MAC address should be in the range displayed by the show module command...
cat6k# show module 5Mod Ports Card Type Model Serial No.--- ----- -------------------------------------- ------------------ -----------
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 13
5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S
Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ ------- 5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- MAC address range assigned to the ACE
Mod Online Diag Status---- ------------------- 5 Pass
4. Check the status of the Te5/1 port to ensure that it is in the forwarding state by entering the followingcommand:
cat6k# show spanning-tree vlan 130
MST0 Spanning tree enabled protocol mstp Root ID Priority 32768 Address 0001.632f.2c17 Cost 200019 Port 642 (GigabitEthernet6/2) Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32768 (priority 32768 sys-id-ext 0) Address 0011.bc06.f800 Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Use the traceroute command to check the route between the ACE and the end devices.
ACE_module5/Admin# traceroute 10.20.12.153traceroute to 10.20.12.153 (10.20.12.153), 30 hops max, 40 byte packets 1 10.20.215.2 (10.20.215.2) 0.532 ms 0.436 ms 0.362 ms 2 10.20.239.161 (10.20.239.161) 0.421 ms 0.488 ms 0.404 ms 3 10.20.238.93 (10.20.238.93) 0.471 ms 0.422 ms 0.413 ms 4 172.27.16.177 (172.27.16.177) 0.488 ms 0.435 ms 0.430 ms 5 172.27.16.226 (172.27.16.226) 0.474 ms 0.363 ms 0.368 ms 6 192.168.0.134 (192.168.0.134) 0.624 ms 0.510 ms 0.494 ms 7 10.20.12.153 (10.20.12.153) 23.982 ms 24.702 ms 25.976 ms
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 14
Contacting Cisco Technical Support
If you are unable to resolve a problem after using the troubleshooting suggestions in the articles in this wiki,contact the Cisco Technical Assistance Center (TAC) for assistance and further instructions. Before you call, havethe following information ready to help your TAC engineer assist you as quickly as possible:
Date that you received the ACE• Chassis serial number (located on a label on the right side of the rear panel of the chassis)• Type of software and release number (if possible, enter the show version command)• Maintenance agreement or warranty information• Brief description of the problem• Brief explanation of the steps that you have already taken to isolate and resolve the problem•
For information on steps to take before calling Technical Support, see the "Gathering ACE TroubleshootingInformation" section.
You can reach TAC in several ways as follows:
Create a service request online•
Call the TAC at the telephone numbers on this page.•
Contact the Cisco Small Business Support Center•
Cisco Application Control Engine (ACE) Troubleshooting Guide
This article describes the ACE architecture and how data flows into, gets processed, and flows out of the ACE. Itprovides a basic understanding of these concepts to assist you in troubleshooting the ACE.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Understanding the ACE Architecture1.1 Overview of the ACEHardware Architecture
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 16
Understanding the ACE Architecture
Having a basic understanding of the ACE architecture and data flow can help to make troubleshooting the ACEeasier. This section describes the major functional areas of the ACE and how they work together.
Overview of the ACE Hardware Architecture
The ACE hardware architecture is divided into a series of functional areas or subsystems that are defined byprocessors or groups of processors and interfaces as shown in Figure 1.
Figure 1. ACE Module Architecture
A console connection allows direct access to the ACE control plane (CP) for initial configuration, management,and troubleshooting. The supervisor engine connection allows you to determine the status of the ACE, to loadimages into the ACE, to reboot the ACE, and to provide remote access to the ACE from the Catalyst 6500 series
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 17
switch or Cisco 7600 series router when you use the session command. Because the ACE has no external ports,packets enter the ACE through the Switch Fabric Interface (SFI) connected to the Catalyst 6500 series switch orCisco 7600 series router back plane. The two major functional areas of the ACE are as follows:
Control plane• Data plane•
Control Plane
The control plane (CP) is used to configure the ACE and for management traffic, syslogs, ARP, DHCP, and soon. You can access the CP directly by using the console port. For remote management, you must configure amanagement interface and enable remote access using a management policy to permit Telnet or SSH access, forexample. The CP is responsible for the following ACE functions:
Device management and control• Configuration management (CLI or XML interface)• Server health monitoring• syslogs• SNMP• Address Resolution Protocol (ARP)• DHCP relay• Redundancy (also known as high availability or fault tolerance)• Access control list (ACL) compilation•
Data Plane
The data plane (DP) is responsible for distributing and processing packets and connections that do not match amanagement policy.
In the ACE, the CP and the DP are separated and run on different processors for maximum performance. SeeFigure 2.
Figure 2. ACE Data Plane
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 18
The DP is responsible for the following ACE functions:
Access control lists (ACLs)• Connection management• TCP termination• Network address translation (NAT)• SSL processing (termination, initiation, encryption, and decryption)• Regular expression matching• Load balancing and forwarding• Application protocol inspection•
The DP consists of the following functional areas:
The Classification and Distribution Engine (CDE) is the traffic controller for the ACE. Its main purpose is toforward packets that it receives from the SFI to the two network processors (NPs). It also acts as the central pointof contact among all the major subsystems within the ACE. The CDE computes, and if necessary, adjusts the IP,TCP, and UDP checksums of every packet that it receives.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 19
The CDE appends a special header known as the IMPH header to each packet before sending it to the fast path.The IMPH header is 18 bytes long and contains information from the DBUS header (the header sent to the ACEModule by the Catalyst 6500 series switch or Cisco 7600 series router) as well as special messaging directlyunderstood by the fast path. Fields in the IMPH header can include notification of a checksum error, Layer 3 orLayer 4 offsets, source and destination ports of the CDE, the VLAN for determining the interface that the fastpath will use, and so on.
Network Processors
The ACE has two network processors (NP1 and NP2) that perform most of the packet processing in the ACE. Alltraffic entering the ACE must traverse one or both NPs after being forwarded by the CDE. Each NP contains aCPU (XScale) and several components called microengines (MEs). See Figure 3.
Figure 3. Network Processor Micorengines
Each microengine can handle eight simultaneous threads or processes and performs a specific function for the NPas follows:
Receive - One ME for receiving incoming packets• Fast Path - Four MEs for the hardware accelerated data path that is used for MAC rewrite, NAT, TCPnormalization, and so on (essentially all operations performed on a per-packet basis)
•
ICM - One ME for the inbound connection manager• OCM - One ME for the outbound connection manager• CCM - One ME for the connection close manager• TCP - Two MEs for TCP termination with a full TCP stack• HTTP - Two MEs for HTTP parsing• Unused (future expansion) - One ME• SSL Record Layer - One ME for the SSL record layer•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 20
IP fragmentation timers - One ME for IP fragmentation reassembly and timer management• DNS and ICMP Inspection - One ME for DNS and ICMP packet inspection•
The XScale microprocessor is programmed to handle the following features:
Load-balancing algorithms• SSL handshake• FTP and Real-Time Streaming Protocol (RTSP) inspection• HTTP inspection (although a considerable part is performed by the microengines)• High-availability heartbeat generation• Returned statistics for most connection-related commands•
Each network processor has RDRAM memory to store ACL entries, routing table entries, ARP entries, andinspection policies. Additional SRAM memory provides faster access times and is used to store regularexpressions and statistics on a per?virtual system basis, among other things.
SSL Crypto Module
The SSL Crypto Module is responsible for SSL record layer processing. This processing includes encrypting anddecrypting data for SSL flows.
Understanding the ACE Traffic Flow
Because the ACE has no native ports, it relies on the switch fabric in the Catalyst 6500 series switch or the Cisco7600 series router back plane to send and receive packets to and from the network. Packets that are marked with adestination VLAN and Layer 2 information enter the ACE through the SFI on the 10 Gbps Ethernet link. Packetsentering or leaving the ACE traverse this link using VLAN tagging. The switch fabric interface (SFI) forwards tothe CDE all packets that are destined to the ACE. See Figure 1.
The CDE classifies the packets based on the configured traffic policies, fills out the IMPH header information,and forwards the traffic to one of the NPs. To determine which NP to forward the traffic to, the CDE hashesincoming packets based on the traffic type as follows:
TCP/UDP - Hash of source/destination port• Non-TCP/UDP IP - Hash of source/destination IP address• Non-IP - Hash of source/destination Layer 2 MAC address•
The NPs process the traffic and forward it back through the CDE to either the control plane or the SSL CryptoModule for further processing.
To-the-ACE Traffic
To-the-ACE traffic is traffic that is destined to an interface VLAN IP address on the ACE. This traffic must matcha class map of type management, which is associated with a policy map and applied as a service policy on aninterface VLAN. The management class map supports the following protocols:
Hypertext Transfer Protocol (HTTP)•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 21
Hypertext Transfer Protocol Secure (HTTPS)• Internet Control Message Protocol (ICMP)• Keepalive Application Protocol (KAL-AP)• User Datagram Protocol (UDP)• Simple Network Management Protocol (SNMP)• Secure Shell (SSH) Protocol• Telnet•
This management traffic is called control plane traffic because it is destined to the CP. Because of the separationof the CP traffic from the data plane traffic on different processors, the control plane traffic will never interferewith data plane traffic, even if the control plane is oversubscribed.
Through-the-ACE Traffic
The CDE sends traffic that requires load balancing, forwarding, routing, or other processing by the ACE to one ofthe NPs.
The NPs comprise two parallel forwarding paths that maintain their own connection state information and forwardtraffic independently.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 22
This article describes some basic troubleshooting steps that you can perform to rule out some of the simpler issuesbefore delving deeper into the troubleshooting process.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Preliminary ACE Troubleshooting Steps• 2 Checking the ACE Status from the Supervisor Engine• 3 Verifying the MSFC VLAN Configuration• 4 Establishing a Session with the ACE from the SupervisorEngine
•
5 Verifying the ACE is Receiving VLAN Allocations from theMSFC
•
6 Verifying the ACE Image• 7 Verifying Your ACE Licenses• 8 Configuring an ACL to Permit Input Traffic to the ACE• 9 Verifying that the ACE is Sending and Receiving Traffic• 10 Verifying to-the-ACE Traffic•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 23
Preliminary ACE Troubleshooting Steps
Check the status of the ACE module from the Catalyst 6500 series switch or Cisco 7600 series router. Seethe "Checking the ACE Status from the Supervisor Engine" section.
1.
Verify that you have allocated the correct VLANs to the ACE in the Multilayer Switch Feature Card(MSFC) VLAN configuration. See the "Verifying the MSFC VLAN Configuration" section.
2.
Verify that you can establish a session to the ACE from the supervisor engine. See the "Establishing aSession with the ACE from the Supervisor Engine" section.
3.
Verify that the ACE is receiving VLAN allocations from the MSFC. See the Verifying the ACE isReceiving VLAN Allocations from the MSFC" section.
4.
Verify your ACE bandwidth, SSL, and virtualization licenses. See the "Verifying Your ACE Licenses"section.
5.
Verify that you have configured an access control list (ACL) to permit traffic on the interfaces on whichyou wish the ACE to receive traffic. If you do not configure an ACL to permit traffic on an interface, alltraffic destined to that interface will be blocked by the ACE. See the "Configuring an ACL to PermitInput Traffic to the ACE" section.
6.
Verify that the ACE is sending and receiving traffic. See the "Verifying that the ACE is Sending andReceiving Traffic" section.
7.
Verify the management traffic to the control plane. See the "Verifying to-the-ACE Traffic"section.8.
Checking the ACE Status from the Supervisor Engine
Before you begin to troubleshoot your ACE, Telnet to the Catalyst 6500 series switch or Cisco 7600 series routersupervisor engine, log in, and check the status of the ACE.
telnet 10.1.1.2
User Access Verification
Password:cat6k> enablePassword:cat6k# show moduleMod Ports Card Type Model Serial No.--- ----- -------------------------------------- ------------------ ----------- . . 5 1 Application Control Engine Module ACE20-6500-K9 SAD1031044S . .
Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ -------
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 24
. . 5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 8.7(0.22)ACE A2(2.0) Ok <------- ACE status
. .
Mod Sub-Module Model Serial Hw Status--- --------------------------- ------------------ ----------- ------- ------- 6 Policy Feature Card 3 WS-F6K-PFC3A SAL09094NUB 2.5 Ok 6 MSFC3 Daughterboard WS-SUP720 SAL09094N33 2.5 Ok
Mod Online Diag Status--- ------------------- . . 5 Pass
. .
Verifying the MSFC VLAN Configuration
To verify that the VLANs that you intend to use in your ACE have been configured and allocated to the ACE inthe MSFC, follow these steps:
1. Check the VLANs configured and allocated to the ACE by entering the following command from thesupervisor engine:
cat6k# show run | include svclc
svclc module 5 vlan-group 123,130,133
2. Ensure that the VLAN groups that you intend to use for your ACE are allocated properly in the MSFCconfiguration by entering the following commands:
cat6k# show svclc module 5 vlan-groupModule Vlan-groups------ ----------- 05 123,130,133
cat6k# show svclc vlan-groupDisplay vlan-groups created by both ACE module and FWSM commands
Group Created by vlans----- ---------- -----123 ACE 103,105,107,111-112,119,134,160,171,200,203,205,207,211-212,226,253,260130 ACE 130133 ACE 100,194,221,256-257
3. Verify that the VLANs you intend to use in your ACE are configured in the MSFC by entering thefollowing command:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 25
cat6k# show interface te5/1 trunk
Port Mode Encapsulation Status Native vlanTe5/1 on 802.1q trunking 1
Port Vlans allowed on trunkTe5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257,260
Port Vlans allowed and active in management domainTe5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257
Port Vlans in spanning tree forwarding state and not prunedTe5/1 100,103,105,107,111-112,119,130,134,160,171,194,200,203,205,207,211-212,221,226,253,256-257
4. Ensure that traffic is routed to two ACEs in the same chassis when both client- and server-side VLANsare configured as switched virtual interfaces (SVIs) on the MSFC in routed mode by entering thefollowing command:
cat6k# show svclc multiple-vlan-interfacesMultiple ACE vlan interfaces feature is enabled
Establishing a Session with the ACE from the SupervisorEngine
To verify that you can establish a session with the ACE from the supervisor engine in the Catalyst 65000 seriesswitch or the Cisco 7600 series router, enter the following command:
cat6k# session slot 5 processor 0The default escape character is Ctrl-^, then x.You can also type 'exit' at the remote prompt to end the sessionTrying 127.0.0.50 ... Open
ACE_module5 login:
Verifying the ACE is Receiving VLAN Allocations from theMSFC
Ensure that the VLAN that you intend to use on ACE are allocated properly in the MSFC configuration byentering the following command:
ACE-1/Admin# show vlansVlans configured on SUP for this module vlan123 vlan130 vlan133
If interface VLANs are already assigned on the ACE you can use the show interface vlan <num> command toverify the interface is properly assigned on the MSFC and up on the MSFC:
ACE-1/Admin# show interface vlan 123
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 26
vlan10 is up Hardware type is VLAN MAC address is 00:18:b9:a6:89:0d Mode : routed IP address is 10.10.10.1 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set
To display the version of the software image and the image filename that is currently running in your ACE, enterthe following command:
ACE_module5/Admin# show versionCisco Application Control Software (ACSW)TAC support: http://www.cisco.com/tacCopyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned byother third parties and are used and distributed under license.Some parts of this software are covered under the GNU PublicLicense. A copy of the license is available athttp://www.gnu.org/licenses/gpl.html.
This command provides other useful information, for example:
Slot in which the ACE resides in the Catalyst 6500 series switch (in this case, slot 5)• Available memory• Last boot reason• Configuration register (confreg) value• ACE uptime•
Verifying Your ACE Licenses
Log in to your ACE and enter the following command to display the SSL, virtualization, and bandwidth licensesthat are currently installed and in use in your ACE:
ACE_module5/Admin# show license usageLicense Ins Lic Status Expiry Date Comments Count--------------------------------------------------------------------------------ACE-08G-LIC No - Unused -ACE-16G-LIC No - Unused -ACE-UPG1-LIC No - Unused -ACE-UPG2-LIC No - Unused -ACE-VIRT-020 No - Unused -ACE-VIRT-050 No - Unused -ACE-VIRT-100 No - Unused -ACE-VIRT-250 Yes 1 In use never -ACE-VIRT-UP1 No - Unused -ACE-VIRT-UP2 No - Unused -ACE-VIRT-UP3 No - Unused -ACE10-16G-LIC No - Unused -ACE-SEC-LIC-K9 No - Unused -ACE-SSL-05K-K9 No - Unused -ACE-SSL-10K-K9 No - Unused -ACE-SSL-15K-K9 No - Unused -ACE-SSL-20K-K9 No - Unused -ACE-SSL-UP1-K9 No - Unused -ACE-SSL-UP2-K9 No - Unused -ACE-SSL-UP3-K9 No - Unused -
ACE_module5/Admin# show license statusLicensed Feature Count------------------------------ -----SSL transactions per second 1000Virtualized contexts 250Module bandwidth in Gbps 4
You can also see the licenses that reside on the Flash disk by entering the following command:
ACE_module5/Admin# dir disk0:236 Oct 17 09:18:26 2006 ACE-SSL-05K-K9.lic <--------235 Oct 17 09:16:58 2006 ACE-VIRT-250.lic <--------
Configuring an ACL to Permit Input Traffic to the ACE
You must configure an ACL to allow the ACE to receive traffic. All traffic to the ACE is blocked until you do so.For example, to configure an ACL that permits all IP trafffic except from the 10.1.1.0 network, enter thefollowing commands:
ACE_module5/Admin(config)# access-list ACL1 extended deny ip 10.1.1.0 255.255.255.0 anyACE_module5/Admin(config)# access-list ACL1 extended permit ip any anyACE_module5/Admin(config)# interface vlan 100ACE_module5/Admin(config-if)# access-group input ACL1
Verifying that the ACE is Sending and Receiving Traffic
You can tell if traffic is reaching the ACE by using the show svclc module number traffic command on theCatalyst 6500 series switch or Cisco 7600 series router. This command displays counters (packets input andpackets output) that increase when the switch or router sends packets to or receives packets from the ACE.
cat6k# show svclc module 5 trafficACE module 5:
Specified interface is up line protocol is up (connected) Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 10Gb/s input flow-control is on, output flow-control is unsupported Last input never, output never, output hang never Last clearing of "show interface" counters 4d02h Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 29
7776 packets output, 746361 bytes, 0 underruns <------- 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out
A trace of the Te?/1 (10-Gbps switch fabric interface, where ? = the module number) interface will show youwhether packets are arriving at the switch fabric interface (SFI).
Another useful command is the show cde health command on the ACE. This command shows the current state ofthe Classification Distribution Engine (CDE). The network processors (NP1 and NP2) are represented by IXP0and IXP1, respectively. You should not observe any drops, errors, or flow control issues in the output of thiscommand. If the Packets Received or the Packets Transmitted counters of the CDE Hyperion Interface are notincreasing, then packets are not coming into or going out of the ACE.
ACE_module5/Admin# show cde health
CDE BRCM INTERFACE======================Packets received 4933Packets transmitted 2437922Broadcom interface CRC error count 0BRCM VOQ status [empty] [not full]BRCM pull status [pulling]
CDE HYPERION INTERFACE======================Packets received 29913371 <-------Packets transmitted 8034 <-------Short packets drop count 0Fifo Full drop count 0Protocol error drop count 0FCS error drop count 0CRC error drop count 0Num times flow control triggered on hyp interface 0Num self generated multicast packets filtered 1880HYP IXP0 VOQ status [empty] [not full]HYP IXP1 VOQ status [empty] [not full]HYP SLOW VOQ status [empty] [not full]HYP tx pull status [pulling]
CDE IXP0 INTERFACE======================Packets received 784985Packets transmitted 27827116Num bad pkts recvd on fast spi channel0 0Num bad pkts recvd on slow spi channel8 0Num bad pkts recvd on fast spi channel2 0Num bad pkts recvd on slow spi channel4 0IXP0 Fast VOQ status [empty] [not full]IXP0 BRCM VOQ status [empty] [not full]IXP0 pull status [pulling]IXP0 spi src status [healthy]IXP0 spi snk status [healthy]
CDE1 SWITCH1 INTERFACE======================Packets received (hyp, ixp0) 4415Packets received (bcm) 1656608
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 30
Packets received (daughter card 0) 0Packets received (daughter card 1) 0Packets Errors received (hyp, ixp0) 0Packets Errors received (bcm) 0Packets Errors received (daughter card 0) 0Packets Errors received (daughter card 1) 0Packets transmitted (ixp1) 2089360Packets transmitted (nitrox) 0Packets Errors transmitted (ixp1) 0Packets Errors transmitted (nitrox) 0
CDE IXP1 INTERFACE======================Packets received 1661023Packets transmitted 2089360Num bad pkts recvd on fast spi channel0 0Num bad pkts recvd on slow spi channel8 0Num bad pkts recvd on fast spi channel2 0Num bad pkts recvd on slow spi channel4 0IXP1 Fast VOQ status [empty] [not full]IXP1 BRCM VOQ status [empty] [not full]IXP1 pull status [pulling]IXP1 spi src status [healthy]IXP1 spi snk status [healthy]
CDE NITROX INTERFACE======================Packets received 0Packets transmitted 0Num bad pkts recvd on fast spi channel0 0Num bad pkts recvd on slow spi channel8 0Num bad pkts recvd on fast spi channel2 0Num bad pkts recvd on slow spi channel4 0NTX Fast VOQ status [empty] [not full]NTX BRCM VOQ status [empty] [not full]NTX pull status [pulling]NTX spi src status [healthy]NTX spi snk status [healthy]== Backplane ==ITASCA_SYS_CNTL1 0x300 data 0x61f0000ITASCA_SYS_CNTL2 0x304 data 0x80c30000
You can also use the show interface command on the ACE to display traffic that is sent and received on theinterface for each VLAN that is configured on the ACE.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 31
ACE_module5/Admin# show interface
bvi2 is administratively down Hardware type is BVI MAC address is 00:18:b9:a6:91:15 Mode : unknown FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set 0 unicast packets input, 0 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 0 unicast packets output, 0 bytes 0 multicast, 0 broadcast 0 output errors, 0 ignored
vlan100 is administratively down Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : unknown FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 0 unicast packets input, 0 bytes 0 multicast, 0 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 0 unicast packets output, 0 bytes 0 multicast, 0 broadcast 0 output errors, 0 ignored
vlan130 is up Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : routed IP address is 10.86.215.134 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor
Traffic that is destined to the ACE itself arrives at the control plane in one of the following ways:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 32
Directly from the console connection• Directly from the supervisor engine connection• Traffic from the SFI that is forwarded by the CDE in the data plane•
Use the following commands to verify that traffic is going to and coming from the control plane.
ACE_module5/Admin# show netio stats
High Priority (Control) Normal Priority (Data)----------------------- ----------------------Net Rx Packets : 224 Net Rx Packets : 2521794Net Rx Bytes : 17528 Net Rx Bytes : 196704169Net Rx Unsupported L2 : 0 Net Rx Unsupported L2 : 0Net Rx Lock Errors : 0 Net Rx Lock Errors : 0Net Rx Interface Miss : 0 Net Rx Interface Miss : 2326290Net Rx No Arp Client : 0 Net Rx No Arp Client : 0Net Rx Alias Drops : 0 Net Rx Alias Drops : 0Net Rx Repl. Errors : 0 Net Rx Repl. Errors : 0Net Rx Repl. If Err : 0 Net Rx Repl. If Errs : 0Net Rx Internal Errs : 0 Net Rx Internal Errs : 0
Net Tx Packets : 0 Net Tx Packets : 5213Net Tx Bytes : 0 Net Tx Bytes : 414073Net Tx Lock Errors : 0 Net Tx Lock Errors : 0Net Tx Bad Context ID : 0 Net Tx Bad Context ID : 0Net Tx No Route Found : 0 Net Tx No Route Found : 0Net Tx No Adjacency : 0 Net Tx No Adjacency : 1Net Tx Invalid If ID : 0 Net Tx Invalid If ID : 0Net Tx If Down : 0 Net Tx If Down : 0Net Tx No Src Addr : 0 Net Tx No Src Addr : 0Net Tx No Encap : 0 Net Tx No Encap : 0Net Tx FIFO Errors : 0 Net Tx Fifo Errors : 0Net Tx No VMAC Errors : 0 Net Tx No VMAC Errors : 0
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 34
This article describes how to troubleshoot basic ACE boot issues.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Understanding ROMMON Mode and the ACE BootConfiguration
1.1 Setting the Boot Method from the ConfigurationRegister
♦
1.2 Booting the ACE from the ROMMON Prompt♦ 1.3 Setting the BOOT Environment Variable♦ 1.4 Displaying the ACE Boot Configuration♦
•
2 Restarting the ACE2.1 Restarting the ACE from the ACE CLI♦ 2.2 Restarting the ACE from the Supervisor Engine♦
•
3 Establishing a Console Connection to the ACE• 4 Troubleshooting ACE Boot Problems•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 35
Understanding ROMMON Mode and the ACE BootConfiguration
You can control how the ACE performs its boot process through either the ACE configuration mode or ROMMonitor (ROMMON) mode. ROMMON is the ROM-resident code that starts executing at power up, reset, orwhen a fatal exception occurs.
Two user-configurable parameters determine how the ACE boots:
Boot field in the configuration register (confreg)• BOOT environment variable•
The ACE enters ROMMON mode if it does not find a valid system image, if the Flash memory configuration iscorrupted, or if the configuration register is set to enter ROMMON mode.
Note: You can manually enter ROMMON mode by restarting the ACE and then pressing the Break key duringthe first 60 seconds of startup. If you are connected to the ACE through a terminal server, you canescape to the Telnet prompt and then enter the send break command to enter the ROMMON mode.
Setting the Boot Method from the Configuration Register
To change the configuration register settings and how the ACE boots from the CLI, use the followingconfiguration mode command:
config-register value
The value argument-supported entries are as follows:
0?ACE boots to the ROMMON prompt. The ACE remains in ROMMON mode at startup.•
1?ACE boots from the system image identified in the BOOT environment variable. If the ACEencounters an error or if the image is not valid, it will try the second image (if one is specified). If thesecond image also fails to boot, the ACE returns to ROMMON mode.
•
For example, to set configuration register to boot the system image identified in the BOOT environment variable,enter the following command:
ACE_module5/Admin(config)# config-register 1
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 36
Booting the ACE from the ROMMON Prompt
If you specify a value of 0 for the config-register command, this configuration register setting forces the ACE toenter the ROMMON mode upon a reload or power cycle of the ACE. The ACE remains in ROMMON mode untilyou identify the location of an image file to boot.
The ACE supports two methods of booting the module from the ROMMON prompt:
To manually change the configuration register setting in ROMMON mode, use the confreg commandfollowed by a value of 0 or 1.
•
To change the boot characteristics using onscreen prompts, use the confreg command without a value.•
To instruct the ACE to manually boot from a particular system image, use the confreg command and specify aconfiguration register value of 1. Identify the name of the system image file that the ACE uses to boot.
For example, to use the confreg command at the ROMMON prompt to instruct the ACE to boot from thec6ace-t1k9-mzg.3.0.0_A2_2_0.bin system image, enter the following command:
To instruct the ACE to automatically boot from the image specified in the BOOT variable, use the confregcommand without specifying a configuration register value to launch the Configuration Summary menu-basedutility. You can then instruct the ACE to boot from the system image identified in the BOOT environmentvariable. See the "Setting the BOOT Environment Variable" section.
For example, to use the confreg command to display the onscreen prompts for changing the boot characteristicsof the ACE and change the configuration register to boot from an image on disk0:, enter the following command:
rommon 4 > confreg
Configuration Summary (Virtual Configuration Register: 0x2000)enabled are:ignore system config infoconsole baud: 9600boot: the ROM monitor
do you wish to change the configuration? y/n [n]: ydisable "ignore system config info"? y/n [n]:change the boot characteristics? y/n [n]: yenter to boot: 0 = ROM Monitor 1 = boot file specified in BOOT variable[0]: 1
Configuration Summary (Virtual Configuration Register: 0x2001)enabled are:ignore system config infoconsole baud: 9600boot: the file specified in BOOT variable
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 37
do you wish to change the configuration? y/n [n]:You must reset/power cycle for new config to take effectrommon 7 > dir disk0:Directory of disk0:
The BOOT environment variable specifies a list of image files from which the ACE can boot at startup. To set theBOOT environment variable, use the boot system image: command. The syntax of this command is as follows:
boot system image:image_name
The image_name argument specifies the name of the system image file. If the file does not exist (for example, ifyou entered the wrong filename), then the filename is appended to the bootstring, and the "Warning: File notfound but still added in the bootstring" message appears. If the file does exist, but is not a valid image, the file isnot added to the bootstring, and the "Warning: file found but it is not a valid boot image" message appears.
For example, to set the BOOT environment variable, enter the following command:
ACE_module5/Admin(config)# boot system image:c6ace-t1k9-mzg.3.0.0_A2_2.0.bin
Displaying the ACE Boot Configuration
To display the current BOOT environment variable and configuration register setting, use the show bootvarcommand in Exec mode. For example, to display the BOOT environment variable settings, enter the followingcommand:
ACE_module5/Admin# show bootvar
BOOT variable = "disk0:c6ace-t1k9-mzg.3.0.0_A2_2_0.bin"Configuration register is 0x1
Restarting the ACE
You can reload the ACE directly from its CLI or reboot it by using the supervisor engine CLI. You may need to
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 38
reboot the ACE from the supervisor engine if you cannot reach the ACE through an external Telnet session or aconsole connection (for example, the ACE is remote).
Restarting the ACE from the ACE CLI
To reboot the ACE directly from its CLI and reload the configuration, use the reload command in Exec mode.The reload command reboots the ACE and performs a full power cycle of both the hardware and software. Thereset process can take several minutes. Any open connections with the ACE are dropped after you enter thereload command.
Caution: Configuration changes that are not written to the Flash partition are lost after a reload. Before rebooting,enter the copy running-config startup-config command in Exec mode to store the current configuration in Flashmemory. If you fail to save your configuration changes, the ACE reverts to its previous settings upon restarting.
When you enter the reload command, the ACE prompts you for confirmation and performs a cold restart of theACE:
ACE_module5/Admin# reload
This command will reboot the systemSave configurations for all the contexts. Save? [yes/no]: [yes]Generating configuration....running config of context Admin savedPerform system reload. [yes/no]: [yes]
Restarting the ACE from the Supervisor Engine
To restart the ACE from the supervisor engine CLI, use the hw-module command. The syntax of this command isas follows:
hw-module module mod_num reset
For example, to use the supervisor engine CLI to reset the ACE located in slot 5 of the chassis, enter the followingcommand:
cat6k# hw-module module 5 resetProceed with reload of module?[confirm]% reset issued for module 5
Establishing a Console Connection to the ACE
In case the ACE becomes unresponsive or you cannot boot the ACE using the reload command from the Admincontext, you can establish a direct serial connection between your terminal (laptop) and the ACE by making aserial connection to the console port on the front of the ACE. The console port is an asynchronous RS-232 serialport with an RJ-45 connector. Any device connected to this port must be capable of asynchronous transmission.Connection requires a terminal configured as 9600 baud, 8 data bits, 1 stop bit, no parity.
Note: Only the Admin context is accessible through the console port; all other contexts can be reached throughTelnet or SSH sessions.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 39
After you connect the terminal to the console port, use any terminal communications application to access theACE CLI. The following procedure uses HyperTerminal for Windows.
To access the ACE by using a direct serial connection, follow these steps:
1. Launch HyperTerminal. The Connection Description window appears.
2. Enter a name for your session in the Name field.
3. Click OK. The Connect To window appears.
4. From the drop-down list, select the COM port to which the device is connected.
5. Click OK. The Port Properties window appears.
6. Set the following port properties:
Baud Rate = 9600• Data Bits = 8• Flow Control = none• Parity = none• Stop Bits = 1•
7. Click OK to connect.
8. Press Enter to access the ACE login prompt.
switch login:
9. If the ACE does not find a valid software image on disk0: or if the ACE is configured to enter ROMMONmode upon booting up, the ROMMON prompt appears.
rommon 1>
Troubleshooting ACE Boot Problems
The ACE module receives power from the chassis back plane and boots up automatically when you insert themodule into the chassis. If your ACE does not boot up when you insert it into the chassis or when you enter thereload Exec mode command from the Admin context, you cannot Telnet to the ACE or establish a session fromthe supervisor engine. In these cases, use the following steps to troubleshoot the issue and boot the ACE:
1. Log in to the Catalyst 6500 series switch or the Cisco 7600 series router and check the status of the ACE byentering the following command:
User Access Verification
Password:cat6k>enablePassword:cat6k# show module 5Mod Ports Card Type Model Serial No.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 40
--- ----- -------------------------------------- ------------------ ----------- 5 1 Application Control Engine Module ACE10-6500-K9 SAD1031044S <------- Module is receiving power
Mod MAC addresses Hw Fw Sw Status--- ---------------------------------- ------ ------------ ------------ ------- 5 0018.b9a6.9114 to 0018.b9a6.911b 1.1 Unknown Unknown Other <------- Firmware and software image status is Unknown
Mod Online Diag Status---- ------------------- 5 Unknown <------- Diagnostics status is Unknown
The first row of information is populated, so you know that the ACE is powered up. The firmware and softwareversions are Unknown and the Status is Other. At this point, you cannot session into the ACE from the supervisorengine.
2. Power cycle the ACE from the supervisor engine to attempt to boot the ACE by entering the followingcommands:
cat6k# config tEnter configuration commands, one per line. End with CNTL/Z.cat6k(config)# no power enable module 5cat6k(config)# power enable module 5
Wait long enough for the ACE to boot up. Try to Telnet or session to the ACE. If you still cannot Telnet orsession to the ACE, continue with Step 3.
3. Establish a console connection to the ACE. For details about establishing a console connection to the ACE, seethe "Establishing a Console Connection to the ACE" section.
rommon 1>
4. Check the ACE configuration register (confreg) by entering the following command:
rommon 2> confregConfiguration Summary (Virtual Configuration Register: 0x1) enabled are: console baud: 9600 boot: the file specified in BOOT variable
A value of 0x1 instructs the ACE to boot from the image in disk0:. A value 0x0 instructs the ACE to boot to theROMMON prompt. If the image specified in the BOOT variable is not in disk0:, then the ACE boots to theROMMON prompt as shown in this example issue.
5. Check the BOOT variable by entering the following command:
rommon 3> setPS1=rommon ! >RELOAD_REASON=reload command by adminBOOT=disk0:c6ace-t1k9-mz.3.0.0_A2_2_0.binARGV0=quiet?=0
6. Ensure that the software image specified in the BOOT variable is present in disk0: by entering the followingcommand:
rommon 4> dir disk0:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 41
31071143 Dec 1 17:01:06 2008 c6ace-t1k9-mzg.A2_2_0.bin 250 Feb 8 20:04:44 2008 scripted_hm.txt 30337516 Jul 31 05:47:42 2008 c6ace-t1k9-mz.A2_1_3.bin 1048640 Aug 8 11:45:06 2008 ACE_FUR_BOOT_ROM.img.rel.2008Apr01_ver121 1048640 Aug 8 13:27:32 2008 ACE_BOOT_ROM.img.rel.2008Apr01_ver121
Usage for image: filesystem 506789888 bytes total used 517210112 bytes free 1024000000 total bytes
7. If the specified image is not in disk0:, then you can boot from another image in disk0: by entering the followingcommand:
rommon 5> boot system disk0:image_name
8. If there is no image on the ACE disk0: to boot from, you can still boot from the supervisor engine. Copy theimage to the supervisor engine's disk0: or disk1:, and then from the supervisor CLI, enter the following command:
9. At the ROMMON prompt on the ACE console, enter the following command to boot the ACE from theEthernet Out-of-Band Channel (EOBC) between the ACE and the Catalyst 6500 series switch or the Cisco 7600series router:
rommon 6> boot eobc:
10. If the ACE is not local or you cannot establish a console connection for any other reason, use the followingprocedure to finish booting the ACE from the supervisor engine with the boot eobc: command:
cat6k# remote login switchTrying Switch ...Entering CONSOLE for Switch
cat6k-sp# svclc console 5Entering svclc ROMMON of slot 5 ...Type "end" to end the session.
rommon 7> boot eobc:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 42
This article describes the ACE system logging facility, how to enable logging, and how to use system messages astroubleshooting tools.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 43
Overview of ACE System Logging
The ACE provides a system logging (syslog) facility that collects and saves system messages and outputs them todestinations that you specify as follows:
Each virtual context generates logs independently from the other virtual contexts. The admin virtual context doesnot log on behalf of the other contexts in the ACE.
The ACE logs the following connection setup and teardown messages in the CP at the connection speed: 106023,302022, 302023, 302024, and 302025. These setup and teardown syslogs are directly forwarded to a remoteserver. Because of the potentially large number of these messages or if you require high-rate system logging ofconnection setup and teardown messages, use the logging fastpath command. This command disables CP syslogsand enables logging of these messages through the DP using a slightly different format and the following syslogIDs: 106028, 302028, 302029, 302030, and 302031, respectively. You can log these messages through the fastpath only to an external syslog server. All other enabled logging destinations are disabled by the logging fastpathcommand.
You can limit the rate at which the ACE generates syslog messages by using the logging rate-limit command.This command allows you to rate limit syslogs based on one of the following criteria:
Time interval• Logging level• Message ID•
Besides logging system messages, the ACE logs access control list (ACL) deny entries.
Note:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 44
Remember to enable logging on the standby ACE in a redundant configuration by entering the loggingstandby command. To log failover information on the standby ACE, you need to set the logging levelto 4.
For details about ACE system message logging, see the Cisco Application Control Engine Module SystemMessage Guide (Software Version A2(1.0)).
Enabling ACE Logging
To enable logging on the ACE module and send syslogs to the monitor, enter the following commands:
ACE_module5/Admin(config)# logging enableACE_module5/Admin(config)# logging monitor 7ACE_module5/Admin(config)# logging trap 7ACE_module5/Admin(config)# no logging message 111008ACE_module5/Admin(config)# no logging message 111009ACE_module5/Admin(config)# logging timestampACE_module5/Admin(config)# do terminal monitor
Use the logging monitor severity_level command only when you are troubleshooting problems on the ACE orwhen there is minimal load on the network. Using this command at other times when the ACE is active maydegrade performance.
Note: logging trap defines the severity sent to the syslog server.
Note: If you do not see syslog messages on the console after enabling logging with the logging enable andlogging monitor 7 commands, log out of the ACE and then log in again.
To enable logging to a syslog server, use the following command syntax:
Message logging: noneBuffered logging: enabled (level - debugging) maximum size 681984Buffer info: current size - 681984 global pool - 1048576 used pool - 1048576 min - 0 max - 681984 cur ptr = 42894 wrapped - yes
Use the logging supervisor command to allow the aggregation of critical syslogs from multiple virtual devices tothe Catalyst 6500 series switch or to the Cisco 7600 series router syslog. For example, enter the followingcommand:
cat6k# show logging...cat6k#17w3d: %TRINITY-7-TRINITY_SYSLOG_DEBUG: %ACE-7-111009: User 'admin' executed cmd: show running.
Logging Severity Levels
The severity_level argument specifies the maximum level for system log messages sent to the console. Theseverity level that you specify indicates that you want syslog messages at that level and messages less than thelevel. For example, if the specified level is 3, the syslog displays level 3, 2, 1, and 0 messages. We recommendthat you use a lower severity level, such as 3, since logging at a high rate may impact the performance of theACE.
After you have enabled system message logging and have specified a destination for the system messages, youcan add more information to the system messages that may be helpful in troubleshooting issues with your ACEmodule. For example, you can do the following:
Add a timestamp• Identify the messages sent to a syslog server• Identify the ACE device ID in messages that are sent to a syslog server•
To add a timestamp to syslog messages, enter the following command:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 46
ACE_module5/Admin(config)# logging timestamp
To identify messages that are sent to a syslog server by severity level, enter the following command:
System Max Queue size: 20080 System Free Queue size for allocation: 19920
In the above example, the ACE has discarded 59 control plane (CP) messages. By default, the syslog messagequeue can hold 80 messages. You can increase the size of the syslog message queue by using the logging queuecommand in configuration mode. Set the queue size before you start collecting syslog messages. When traffic isheavy, messages may be discarded if the queue size is too small. The maximum number of messages that thequeue can hold is 8192.
Displaying the Logging History
To display the ACE logging history, enter the following command from the console:
ACE_module5/Admin# show logging historysyslog_trinity_show_history for context 0: 1(Mar 24 2009 16:39:36): from "KERN" ACE-5-111008:User 'admin' executed the 'logging history 7' command. 2(Mar 24 2009 16:39:48): from "KERN" ACE-5-111008:User 'admin' executed the 'logging console 7' command. 3(Mar 24 2009 16:39:56): from "KERN" ACE-7-111009:User 'admin' executed cmd: do sho logging history 4(Mar 24 2009 16:49:50): from "KERN" ACE-7-111009:User 'admin' executed cmd: do show logging message 5(Mar 24 2009 16:51:35): from "KERN" ACE-4-405001:Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100 6(Mar 24 2009 16:51:36): from "KERN" ACE-4-405001:Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100 7(Mar 24 2009 16:51:40): from "KERN" ACE-4-405001:Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100 8(Mar 24 2009 16:51:41): from "KERN" ACE-4-405001:Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100 9(Mar 24 2009 16:54:46): from "KERN" ACE-7-111009:User 'admin' executed cmd: telnet 10.1.1.130
Displaying Logging Messages
If a particular system message does not appear in the syslog history, check that the message is enabled and thatthe logging level is correct for that message. To display the default logging level and the current status of a systemmessage, all system messages, or disabled system messages, enter the following command:
ACE_module5/Admin# show logging message message_id | all | disabled
For example, to display all disabled system messages in the ACE, enter the following command:
The logging persistence command enables logging to disk0: in Flash memory. The messages are stored in asubdirectory of disk0: called /messages.
To display logging persistence, enter the following command:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 48
ACE_module5/Admin# show logging persistentPersist info: current size - 6626 global pool - 1048576 used pool - 6626 min - 0 max - 189582 cur ptr = 6638 wrapped - no
Mar 24 2009 09:51:31 Admin: %ACE-4-405001: Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100Mar 24 2009 09:51:32 Admin: %ACE-4-405001: Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100Mar 24 2009 09:51:36 Admin: %ACE-4-405001: Received ARP REQUEST collision from 10.1.1.240 00.0c.29.74.51.fa on interface vlan100Mar 24 2009 09:51:37 Admin: %ACE-4-405001: Received ARP RESPONSE collision from 10.1.1.240 00.0b.fc.fe.1b.03 on interface vlan100
Displaying the Logging Rate Limit
To display the logging rate limit, enter the following command:
ACE_module5/Admin# show logging rate-limitRate-limit logging: (min - 0 max 100000 msgs/sec) 100000 messages 1 seconds level 7
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 49
This article describes how the ACE establishes connections and how to troubleshoot connectivity issues with yourACE.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE Connection Handling• 2 Internal Mapping of ACE TCP and UDPFlows
•
3 ACE Connection Table Entries• 4 Tracking Connections Through the ACE• 5 Troubleshooting Connections•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 50
Overview of ACE Connection Handling
This article describes how the ACE handles connections at Layer 4 (L4) and Layer 7 (L7). For L4 connections,the ACE receives a TCP packet from a client and load balances the connection to a server on the first packet (seeFigure 1). The SYN-ACK from the server matches an existing flow and the rest of the connection is handled inthe fast path (hardware accelerated path in the network processors), which is represented here as "shortcut." TheACE completes the TCP handshake . This process applies to the following functions:
Basic load balancing• Source IP sticky• TCP/IP normalization•
Figure 1. Layer 4 Flow Setup
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 51
For L7 flows (for example, L7 load balancing, URL parsing, and generic TCP payload parsing), the ACE acts as aproxy (spoofs the server), intercepts the client's VIP request that matches an L7 rule, and terminates the TCPconnection. See Figure 2. The ACE sends a SYN-ACK to the client in response to the client's TCP SYN. Theclient responds with an ACK to complete the TCP handshake and an L7 request method (for example, HTTP GETor POST).
Figure 2. Layer 7 Flow Setup -- Client Connection
After the ACE receives the L7 information (for example, HTTP GET), it sets up the back-end connection to thereal server based on the load-balancing method and other criteria. See Figure 3.
Figure 3. Layer 7 Flow Setup -- Server Connection
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 52
Finally, the ACE unproxies the connection with the client and splices it together with the back-end connection tothe server. For the life of the HTTP flow, the client communicates directly with the server through the fast path(hardware-accelerated path in the network processors), which is depicted in the figures as "Shortcut." See Figures4.
Figure 4. Layer 7 Flow Setup -- Splicing the Flows Together
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 53
Figure 5 shows how the ACE adjusts the sequence numbers and ACK numbers when it splices the two flowstogether.
Figure 5. Layer 7 Flow Setup -- Adjusting the Sequence and ACK Numbers
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 54
With the persistence rebalance (connection keepalive) command configured, the ACE reproxies and parsessubsequent HTTP 1.1 requests over the same TCP connection. In this case, the ACE again spoofs the server andACKs the HTTP GET as shown in Figure 6. The sequence shown in Figure 2 through Figure 5 repeats for eachnew HTTP 1.1 request over the same TCP connection.
Figure 6. Layer 7 Flow Setup -- Reproxy
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 55
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the ACE fullyterminates the client TCP connection. This connection remains fully proxied because the ACE is acting on behalfof the real server. For SSL termination, the ACE completes an SSL handshake after it establishes the TCPconnection with the server. See Figure 7.
Figure 7. SSL Handshake
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 56
For SSL termination, TCP server reuse, application protocol inspection, and HTTP 1.1 pipelining, the client andserver connections are completely independent and flows are handled in the software, not in the fast path. SeeFigure 8.
Figure 8. Layer 7 Flow Setup -- Full Proxy
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 57
Internal Mapping of ACE TCP and UDP Flows
The ACE maps TCP and UDP flows as two halves of the same flow: one input flow and one output flow. You candisplay the current connections in the ACE by entering the show connections command. See Figure 8.
Figure 8. Internal Flow Mapping
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 58
ACE Connection Table Entries
Understanding ACE?s Conn Table Entries During:
L4 TCP Connection Setup (3 Way Handshake)Normalisation Enabled♦ Normalisation Disabled♦
•
L7 TCP Connection Setup (3 Way Handshake)• TCP Connection Teardown
3 Way Handshake♦ 4 Way Handshake♦ Reset♦
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 59
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 60
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 61
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 62
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 63
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 64
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 65
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 66
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 67
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 68
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 69
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 70
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 71
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 72
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 73
Tracking Connections Through the ACE
You can display the IDs for the request and response connections in the ACE by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-c 9"Connection ID:seq: 9[0x9].6 <------- Request and response connection ID Other ConnID : 3[0x3].4 Proxy ConnID : 0[0x0].0 Next Q : 1124073484[0x4300000c]
192.168.12.15:1985 -> 10.1.1.2:1985 [RX-NextHop: Drop] [TX-NextHop: TX] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 17 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 0 EncapsID:ver : 0:0 TCP ACK delta : 0x0 MSS : 0 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 152 NAT Policy ID : 0 Post NAT hop : 0 Packet Count : 0 Byte Count : 0 TCP Information: (State = 0)
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 74
Window size : 0 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 0 Last seq : 0 timestamp_delta: 0 Last ack : 1ce862 No Trigger : 0 Trigger Status : 0 Timestamp : 26ebc96d TCP options negotiated: Sack:Allow TS:Allow Windowscale: Allow Reserved: Allow Exceed MSS: Allow Window var: Allow
You can display both the front-end and the back-end connection statistics by entering the "-v" (verbose) option ofthe show np command as follows:
ACE_module5/Admin# show np 1 me-stats "-c 9 -v"Connection ID:seq: 9[0x9].2 Other ConnID : 7[0x7].14 Proxy ConnID : 0[0x0].0 Next Q : 0[0x0]
10.1.1.5:23 -> 172.27.16.143:4837 [RX-NextHop: TX] [TX-NextHop: CP] Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No L3 Protocol : IPv4 L4 Protocol : 6 Inbound Flag : 0 Interface Match : Yes Interface MatchID: 7 EncapsID:ver : 3:0 TCP ACK delta : 0x0 MSS : 1260 TOS Stamp : 0 Repeat mode : No ARP Lookup : No TOS Stamp : No TCP Window Check: No ACE ID : 148 NAT Policy ID : 0 Post NAT hop : 4 Packet Count : 347 Byte Count : 24476 TCP Information: (State = 3) Window size : 5840 Window scale : 0 FIN seen : No FIN/ACK seen : No FIN/ACK exp : No Close initiator : No FIN/ACK expval: 5b40000 Last seq : 53768a51 timestamp_delta: 0 Last ack : 658c1f72 No Trigger : 0 Trigger Status : 0 Timestamp : 459f781e TCP options negotiated: Sack:Clear TS:Clear Windowscale: Clear Reserved: Allow Exceed MSS: Deny Window var: Allow Flags: debug: 0 TCP Normalize: Yes Syslog: No Reproxy Request: No Policying Reqd: No Inbound IPSec: No Replicated: No Data Channel: No L7: No Fin Detect: Yes FP Timeout: No Standby: No ConnState: 2 ACA Method: 0 ReqTS: 00000000 RspTS: 00000000
No valid proxy entry.No valid TCB proxy entry.No valid HTTP proxy entry.No valid SSL proxy entry.No valid AI proxy entry.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 76
Troubleshooting Connections
To troubleshoot suspected connectivity issues, follow these steps:
1. Check the ACL hit count by entering the show access-list acl_name command. If the hit count is increasing, goto Step 2. Otherwise, verify that the access list is configured properly to permit traffic.
ACE_module5/Admin# show access-list anyone detailaccess-list:anyone, elements: 1, status: ACTIVE remark :access-list anyone line 8 extended permit ip any any (hitcount=3438) [0x44c2baf1] <------- Hit count
2. Check the service policy hit count by entering the show service-policy detail command. If the hit count is 0,verify that the service policy is active (show service-policy command) and the server farm is up (showserver-farm detail command). If the service policy is large, use the show service-policy policy_name summarycommand for more information as follows:
ACE_module5/Admin# show service-policy VIP summary
3. Check the load-balancing statistics by entering the show stats loadbalance command. If the Layer 4 or Layer 7rejections or the Layer 4 or Layer 7 policy misses are increasing, check the configured class maps for anymisconfiguration.
ACE_module5/Admin# show stats loadbalance
+------------------------------------------+ +------- Loadbalance statistics -----------+ +------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0
Total Layer4 rejections : 3 <-------| Total Layer7 decisions : 0 |------- Failed connections due to traffic not matching the configured class maps
Total Layer7 rejections : 7 <-------| Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0
Total times rserver was unavailable : 10 <------- Failed connections due to no real server' Total ACL denied : 0 Total IDMap Lookup Failures : 0
To clear the load-balancing statistical information stored in the ACE buffer, enter the clear stats loadbalancecommand.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 77
4. If none of the error statistics is increasing, check the connection record by entering the show conn detailcommand and checking the connections for the affected VIP.
ACE_module5/Admin# show conn detail
total current connections : 6
conn-id np dir proto vlan source destination state----------+--+---+-----+----+---------------------+---------------------+------+7 1 in TCP 130 10.1.1.2:1171 10.1.1.134:23 ESTAB [ idle time : 00:00:00, byte count : 60055 ] [ elapsed time: 04:15:29, packet count: 1473 ]9 1 out TCP 130 10.1.1.134:23 10.1.2.74:1171 ESTAB [ conn in reuse pool : FALSE] [ idle time : 00:00:00, byte count : 64880 ] [ elapsed time: 04:15:29, packet count: 1086 ]
5. Display existing ACE connection statistics by entering the following command:
ACE_module5/Admin# show stats connection
+------------------------------------------++------- Connection statistics ------------++------------------------------------------+ Total Connections Created : 628950 Total Connections Current : 7 Total Connections Destroyed: 389 Total Connections Timed-out: 3958Total Connections Failed : 624596 <------- Server did not reply to a SYN within the pending timeout period or it replied with a RST
The Total Connection Failed counter increases when the ACE cannot set up the back-end connection with theserver. To clear the statistical information stored in the ACE buffer, enter the clear stats connection command.
6. Display service policy statistics by entering the following command:
hit count : 22 <------- Shows the hit count dropped conns : 0
7. Display server farm connection statistics by entering the following command:
ACE/Context# show serverfarm HTTPS-FARM detailserverfarm : HTTPS-FARM, type: HOST total rservers : 4 active rservers: 4 description : - state : ACTIVEpredictor : ROUNDROBIN <------- Shows the load-balancing predictor that was used
failaction : - back-inservice : 0 partial-threshold : 0 num times failover : 0 num times back inservice : 0 total conn-dropcount : 0 --------------------------------- ----------connections----------- real weight state current total failures ---+---------------------+--------+---------------------+-----------+----------+--------- rserver: linux-1
192.168.1.11:0 8 OPERATIONAL 0 0 0 <------- Shows connection statistics foreach real server
The Connections Failures counter for a real server in a server farm may increment for one of the followingreasons:
SYN timeout (the three-way handshake fails to complete)• RST received (a client sends an RST to the server)• Internal exception (internal software issue)•
8. Display the statistics for a connection parameter map by entering the following command:
ACE_module5/Admin# show parameter-map CONN_PARAMMAP
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 81
Overview of ACE Remote Access
You can access the ACE remotely using several different protocols as follows:
HTTP• HTTPS• ICMP• KALAP-UDP• SSH• SNMP• Telnet•
These protocols require that you configure a management traffic policy on the ACE and associate that policy withthe interface that you intend to use for management traffic. A management policy allows the classification anddistribution engine (CDE) to classify (match) incoming management traffic to the management policy and toforward that traffic to the control plane (CP). For complete details about remote access, see the Cisco ApplicationControl Engine Module Administration Guide (Software Version A2(1.0)).
Configuring a Management Policy for Remote Access
To configure a management policy that allows remote access to the ACE using ICMP, SSH, or Telnet, enter thefollowing commands:
ACE_module5/Admin(config)# access-list ACL1 extended permit ip any anyACE_module5/Admin(config)# class-map type management match-any MGMT_CLASSACE_module5/Admin(config-cmap-mgmt)# 2 match protocol icmp anyACE_module5/Admin(config-cmap-mgmt)# 3 match protocol ssh anyACE_module5/Admin(config-cmap-mgmt)# 4 match protocol telnet anyACE_module5/Admin(config-cmap-mgmt)# exitACE_module5/Admin(config)# policy-map type management first-match MGMT_POLICYACE_module5/Admin(config-pmap-mgmt)# class MGMT_CLASSACE_module5/Admin(config-pmap-mgmt-c)# permitACE_module5/Admin(config-pmap-mgmt-c)# exitACE_module5/Admin(config-pmap-mgmt)# exitACE_module5/Admin(config)# interface vlan 100ACE_module5/Admin(config-if)# ip address 192.168.12.15 255.255.255.0ACE_module5/Admin(config-if)# service-policy input MGMT_POLICYACE_module5/Admin(config-if)# access-group input ACL1
Troubleshooting Remote Access
If you cannot access the ACE module remotely, follow these steps:
Cisco Application Control Engine (ACE) Troubleshooting Guide
1. Beginning with ACE software release A2(1.1), by default, the ACE CLI is only locally accessible either usingthe ACE console port or through the supervisor by entering the session command. Remote access to the ACE (forexample, Telnet, SSH, and so on) is disabled until you change the admin user account password from the default.Access to the XML API is also disabled until you change the www user account password from the default. TheACE will display these warnings each time you access the CLI using the the console port or the supervisor untilyou change these passwords.
cat6k#session slot 5 processor 0The default escape character is Ctrl-^, then x.You can also type 'exit' at the remote prompt to end the sessionTrying 127.0.0.20 ... Open
ACE_module5 login: adminPassword:Cisco Application Control Software (ACSW)TAC support: http://www.cisco.com/tacCopyright (c) 2002-2009, Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned byother third parties and are used and distributed under license.Some parts of this software are covered under the GNU PublicLicense. A copy of the license is available athttp://www.gnu.org/licenses/gpl.html.Please change the password for admin user.Admin user is allowed to login only from supervisor until the password is changed.User 'www' is disabled. Please change the password to enable the user.
Use the following commands to change the passwords of the admin and www user accounts:
ACE_module5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_module5/Admin(config)# username admin password 0 cisco123 role Admin domain default-domainACE_module5/Admin(config)# username www password 0 cisco123 role Admin domain default-domainACE_module5/Admin(config)# exit
Note that, although the passwords were entered in clear text above, they will be stored in the ACE configurationin an encrypted format:
ACE_module5/Admin# show run | i usernameGenerating configuration....username admin password 5 $1$M7gtcvBC$9ca78Q.ZH5jZpqDVuLnkN0 role Admin domain default-domainusername www password 5 $1$ulc7KHL5$2HlgNTEez03.ElmbiWKyY/ role Admin domain default-domain
2. Ensure that the remote access method protocol (for example, Telnet or SSH) that you are trying to use isconfigured in the management class map and that the management class has been permitted in the managementpolicy. If necessary, correct your ACE configuration. To display your management policy configuration elements,enter the following Exec mode commands:
ACE_module5/Admin# show running-config class-mapGenerating configuration....
class-map type management match-any MGMT_CLASS 2 match protocol icmp any 3 match protocol ssh any 4 match protocol telnet any
ACE_module5/Admin# show running-config policy-map MGMT_POLICYGenerating configuration....
Cisco Application Control Engine (ACE) Troubleshooting Guide
policy-map type management first-match MGMT_POLICY class MGMT_CLASS permit class class-default
3. Ensure that the management policy is applied to the correct interface and that you are using the correct IPaddress for that interface. If necessary, correct your configuration. Enter the following command:
ACE_module5/Admin# show running-config interface
interface vlan 100 ip address 192.168.12.15 255.255.255.0 access-group input ACL1 access-group output ACL1 service-policy input MGMT_POLICY no shutdown
4. Check the status of the management interface by entering the following command:
ACE_module5/Admin# show interface vlan 100
vlan100 is up Hardware type is VLAN MAC address is 00:18:b9:a6:91:15 Mode : routed IP address is 192.168.12.15 netmask is 255.255.255.0 FT status is non-redundant Description:not set MTU: 1500 bytes Last cleared: never Alias IP address not set Peer IP address not set Assigned from the Supervisor, up on Supervisor 115303 unicast packets input, 74570169 bytes 273637 multicast, 521226 broadcast 0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops 12591 unicast packets output, 2120271 bytes 0 multicast, 4604 broadcast 0 output errors, 0 ignored
5. If the interface is down, ensure that the no shutdown command is configured on the interface to enable it. Ifnecessary, correct your configuration. Enter the following command:
ACE_module5/Admin# show running-config interface
6. Ensure that you have not exceeded the allocated resources for management connections or maximummanagement bandwidth by entering the following commands:
ACE_module5/Admin# show resource usage resource mgmt-connections Allocation Resource Current Peak Min Max Denied-------------------------------------------------------------------------------Context: Admin mgmt-connections 2 10 0 100000 0Context: C1 mgmt-connections 0 0 0 100000 0
ACE_module5/Admin# show resource usage resource rate mgmt-traffic Allocation
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 84
Resource Current Peak Min Max Denied-------------------------------------------------------------------------------Context: Admin mgmt-traffic rate 78 3588 0 125000000 0Context: C1 mgmt-traffic rate 0 0 0 125000000 0
7. If necessary, allocate more resources to management connections by entering the following command:
ACE_module5/Admin(config-resource)# limit-resource mgmt-connections minimum number maximum number
8. Ensure that traffic is reaching the CP by entering the following command:
ACE_module5/Admin# show netio stats
High Priority (Control) Normal Priority (Data)----------------------- ----------------------Net Rx Packets : 680 Net Rx Packets : 10799640 <-------Net Rx Bytes : 52902 Net Rx Bytes : 842376636 <-------Net Rx Unsupported L2 : 0 Net Rx Unsupported L2 : 0Net Rx Lock Errors : 0 Net Rx Lock Errors : 0Net Rx Interface Miss : 0 Net Rx Interface Miss : 10470926Net Rx No Arp Client : 0 Net Rx No Arp Client : 0Net Rx Alias Drops : 0 Net Rx Alias Drops : 0Net Rx Repl. Errors : 0 Net Rx Repl. Errors : 0Net Rx Repl. If Err : 0 Net Rx Repl. If Errs : 0Net Rx Internal Errs : 0 Net Rx Internal Errs : 0
Net Tx Packets : 0 Net Tx Packets : 10469 <-------Net Tx Bytes : 0 Net Tx Bytes : 1194879 <-------Net Tx Lock Errors : 0 Net Tx Lock Errors : 0Net Tx Bad Context ID : 0 Net Tx Bad Context ID : 0Net Tx No Route Found : 0 Net Tx No Route Found : 0Net Tx No Adjacency : 0 Net Tx No Adjacency : 0Net Tx Invalid If ID : 0 Net Tx Invalid If ID : 0Net Tx If Down : 0 Net Tx If Down : 0Net Tx No Src Addr : 0 Net Tx No Src Addr : 0Net Tx No Encap : 0 Net Tx No Encap : 0Net Tx FIFO Errors : 0 Net Tx Fifo Errors : 0Net Tx No VMAC Errors : 0 Net Tx No VMAC Errors : 0
Management traffic is considered to-the-ACE traffic or CP traffic. If traffic is reaching the CP, the NormalPriority (Data) Net Rx Packets, Net Rx Bytes, Net TX packets, and Net TX bytes counters should be increasing. Ifnot, contact TAC.
9. If traffic is not arriving at the CP, ensure that traffic is reaching the classification and distribution engine (CDE)from the SFI by entering the following command:
ACE_module5/Admin# show cde health
CDE BRCM INTERFACE
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 85
======================Packets received 10580Packets transmitted 10802845Broadcom interface CRC error count 0BRCM VOQ status [empty] [not full]BRCM pull status [pulling]
CDE HYPERION INTERFACE======================Packets received 12312951 <-------Packets transmitted 17361 <-------Short packets drop count 0Fifo Full drop count 0Protocol error drop count 0FCS error drop count 0CRC error drop count 0Num times flow control triggered on hyp interface 0Num self generated multicast packets filtered 4618HYP IXP0 VOQ status [empty] [not full]HYP IXP1 VOQ status [empty] [not full]HYP SLOW VOQ status [empty] [not full]HYP tx pull status [pulling]
<snip>
If traffic is reaching the CDE, the Packets received and the CDE Hyperion Interface Packets transmitted countersshould be increasing. If not, contact TAC.
10. If packets are not reaching the CDE, ensure that the MSFC in the Catalyst 6500 series switch or the Cisco7600 series router is sending packets to the switch fabric interface (SFI) by entering the following command onthe supervisor engine:
cat6k# show svclc module 5 trafficACE module 5:
Specified interface is up line protocol is up (connected) Hardware is C6k 10000Mb 802.3, address is 0018.b9a6.9114 (bia 0018.b9a6.9114) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 10Gb/s input flow-control is on, output flow-control is unsupported Last input never, output never, output hang never Last clearing of "show interface" counters 1w2d Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 1000 bits/sec, 2 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 912150 packets input, 74727962 bytes, 0 no buffer Received 796374 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 17390 packets output, 2145844 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 86
If the MSFC is sending traffic to the SFI, the packets input and the packets output counters should be increasing.If not, contact TAC.
Troubleshooting Telnet
If you cannot Telnet to the ACE, ensure that you have not reached the maximum connection limit for Telnet byentering the following commands:
ACE_module5/Admin# show telnet
Session ID Remote Host Active Time4254 127.0.0.51 :41985 0: 8:13
The show telnet command output shows only one Telnet session. A maximum of 15 more users can potentiallyTelnet to the Admin context.
To display the maximum number of users allowed to Telnet to a particular context, enter the following command:
ACE_module5/Admin# show telnet maxsessions
Maximum Sessions Allowed is 16
Troubleshooting SSH
If you attempt to connect to the ACE using SSH and receive the following error, follow these steps:
[linux]$ ssh [email protected]_exchange_identification: Connection closed by remote host [linux]$
1. Ensure that SSH is enabled in the management policy by entering the following command:
ACE_module5/Admin# show running-config class-map
class-map type management match-any MGMT_CLASS 2 match protocol http any 3 match protocol https any 4 match protocol icmp any
6 match protocol ssh any <------- SSH is enabled 7 match protocol telnet any 8 match protocol snmp any
switch/Admin# show running-config policy-map MGMT_POLICYGenerating configuration....
policy-map type management first-match MGMT_POLICY class MGMT_CLASS
permit <------- All protocols in the MGMT_CLASS class-map are permitted including SSH
2. Ensure that the SSH key has been generated by entering the following command:
switch/Admin# show ssh key**************************************could not retrieve rsa1 key information
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 87
**************************************could not retrieve rsa key information**************************************could not retrieve dsa key information**************************************no ssh keys present. you will have to generate them**************************************
The show ssh key command output shows that no SSH key has been generated.
3. Generate an SSH key based upon your security requirements by entering the following commands:
ACE_module5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_module5/Admin(config)# ssh key rsa 4096generating rsa key(4096 bits)..............................generated rsa keyACE_module5/Admin(config)# exitswitch/Admin# show ssh keydsa rsa rsa1ACE_module5/Admin# show ssh key**************************************could not retrieve rsa1 key information**************************************rsa Keys generated:Tue Apr 7 15:55:20 2009
bitcount:4096fingerprint:13:96:fe:f0:f7:d7:5f:9c:d7:2a:da:72:8a:93:53:a6**************************************could not retrieve dsa key information**************************************
Now SSH should work.
4. Try connecting to the ACE via SSH again by entering the following command:
[linux]$ ssh [email protected]: Permanently added '192.168.0.210' (RSA) to the list of known hosts.
Password:Cisco Application Control Software (ACSW) TAC support: http://www.cisco.com/tac Copyright (c) 1985-2009 by Cisco Systems, Inc. All rights reserved.The copyrights to certain works contained herein are owned by other third parties and are used and distributed under license.Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html.ACE_module5/Admin#
5. Confirm the SSH session from the ACE CLI by entering the following command:
ACE_module5/Admin# show ssh session-info
Session ID Remote Host Active Time6986 10.76.248.6 :42116 0: 0:46
Troubleshooting KAL-AP
To troubleshoot KAL-AP related issues, follow these steps:
1. Make sure that KAL-AP is enabled under the management policy by entering the following commands:
ACE_module5/Admin# show running-config class-mapGenerating configuration....
class-map type management match-any MGMT_CLASS 2 match protocol http any
Cisco Application Control Engine (ACE) Troubleshooting Guide
Total bytes received : 243956Total bytes sent : 184884Total requests received : 5100Total responses sent : 5100Total requests successfully received : 5100Total responses successfully sent : 5100Total secure requests received : 0Total secure responses sent : 0Total requests with errors : 0Total requests with parse errors : 0Total response transfer errors : 0-----------------------------------------------------
3. Allow secure KAL-AP requests, and add the GSS IP address and the shared secret to the ACE by entering thefollowing commands:
4. Display information about the load VIP by entering the following command:
ACE_module5/Admin# show kalap udp load vip 10.1.1.1
Error: Vip object not found!ACE_module5/Admin#
If the VIP object is not found while displaying the load value as shown above, check whether the VIP gotdownloaded in the configuration manager internal table by entering the following command:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 91
This article describes security access control lists (ACLs) in the ACE, how to configure them, and troubleshootingsteps to follow if you encounter problems with ACLs.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of Security AccessControl Lists
1.1 ACL Types and Uses♦ 1.2 ACL ConfigurationGuidelines
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 92
Overview of Security Access Control Lists
An ACL consists of a series of statements called ACL entries that define the network traffic profile. Each entrypermits or denies network traffic (inbound and outbound) from and to the parts of your network specified in theentry. Each entry also contains a filter element that is based on criteria such as the source address, the destinationaddress, the protocol, and protocol-specific parameters such as ports and so on.
An implicit deny-all entry exists at the end of each ACL, so you must configure an ACL on each interface thatyou want to permit connections. Otherwise, the ACE denies all traffic on the interface.
ACLs allow you to control network connection setups rather than processing each packet. Such ACLs arecommonly referred to as security ACLs. You can configure ACLs as parts of other features (for example,security, Network Address Translation (NAT), server load balancing (SLB), and so on). The ACE merges theseindividual ACLs into one large ACL called a merged ACL. The ACL compiler then parses the merged ACL andgenerates the ACL lookup mechanisms. A match on this merged ACL can result in multiple actions.
Note: You can apply only one extended ACL to each direction (inbound or outbound) of an interface. You canalso apply the same ACL on multiple interfaces. You can apply EtherType ACLs only in the inbounddirection and only on Layer 2 interfaces.
ACL Types and Uses
You can configure the following two types of ACLs in the ACE:
Extended?Control network access for IP traffic (Layer 3 and Layer 4)• EtherType?Control network access for non-IP traffic on Layer 2 interfaces•
The ACE does not explicitly support standard ACLs. To configure a standard ACL, specify the destinationaddress as any and do not specify ports in an extended ACL. For details about configuring an extended ACL, seethe ?Configuring an Extended ACL? section.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 93
ACL Configuration Guidelines
This section describes the guidelines to observe when you configure and use ACLs in your network. It containsthe following topics:
ACL Entry Order• ACL Implicit Deny• Maximum Number of ACL Entries•
ACL Entry Order
An ACL consists of one or more entries. Depending on the ACL type, you can specify the source and destinationaddresses, the protocol, the ports (for TCP or UDP), the ICMP type, the ICMP code, or the EtherType as thematch criteria. By default, the ACE appends each ACL entry at the end of the ACL. You can also indicate thelocation of each entry within an ACL by specifying a line number.
The order of the entries is important. When the ACE decides whether to accept or refuse a connection, the ACEtests the packet against each ACL entry in the order in which the entries are listed. After it finds a match, the ACEdoes not check any more entries. For example, if you create an entry at the beginning of an ACL that explicitlypermits all traffic, the ACE does not check any other statements in the ACL.
Note: If there is a deny statement for packets coming to the Control Plane (CP), then the ACE skips to the nextACL entry.
ACL Implicit Deny
All ACLs have an implicit deny entry at the end of the ACL, so, unless you explicitly permit it, traffic cannotpass. For example, if you want to allow all users to access a network through the ACE except for those users withparticular IP addresses, then you must deny the particular IP addresses in one entry and permit all other IPaddresses in another entry.
Maximum Number of ACL Entries
The ACE supports a maximum of 256,000 ACL entries. Some ACLs use more memory than others, such as anACL that uses large port number ranges or overlapping networks (for example, one entry specifies 10.0.0.0/8 andanother entry specifies 10.1.1.0/24). Depending on the type of ACL, the actual limit that the ACE can supportmay be less than 256,000 entries.
If you use object groups in ACL entries, you enter fewer actual ACL entries, but the same number of expandedACL entries as you did when you entered entries without object groups. Expanded ACL entries count toward thesystem limit. To view the number of expanded ACL entries in an ACL, use the show access-list name command.
If you exceed the memory limitations of the ACE, it generates a syslog message and increments the DownloadFailures counter in the output of the show interface vlan number command. The configuration remains in therunning-config file and the interface stays enabled. The ACL entries stay the same as they were before the failingconfiguration was attempted.
For example, if you add a new ACL with ten entries, but the addition of the sixth entry fails because the ACE runs
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 94
out of memory, the ACE removes the five entries that you successfully entered.
Note: You must allocate sufficient ACL memory resources for each virtual context in the ACE. The ACE doesnot generate a syslog if you exceed the maximum number of ACL entries.
Configuring ACLs
You can configure ACLs in one of two ways:
Using the access-list command in configuration mode• Using the match access-list command in a Layer 3 and Layer 4 class map•
You can permit or deny network connections based on the IP protocol, source and destination IP addresses, andTCP or UDP ports. To configure a non-ICMP extended ACL, enter the following command:
You can also permit or deny network connections based on the ICMP type (for example, echo, echo-reply,unreachable, and so on). To configure an ICMP extended ACL, enter the following command:
You can configure an ACL that controls traffic based on its EtherType. An EtherType is a subprotocol identifier.EtherType ACLs support Ethernet V2 frames; they do not do not support 802.3-formatted frames. To configurean Ethertype ACL, enter the following command:
Note: You can configure an EtherType ACL on a Layer 2 interface in the inbound direction only. If you areoperating the ACE in bridge mode, be sure to configure an ACL on all interfaces that permit BPDUs.Otherwise, a bridge loop may result.
For example, to configure an extended ACL to permit all IP traffic from any source IP address and that is destinedto any IP address on interface VLAN 200, enter the following commands:
ACE_module5/Admin(config)# access-list ACL1 extended permit ip any anyACE_module5/Admin(config)# interface vlan 200ACE_module5/Admin(config-if)# ip address 192.168.1.1 255.255.255.0ACE_module5/Admin(config)# access-group input ACL1
You can apply an ACL to all interfaces in a context at once, subject to the following conditions:
No interface in the context has an ACL applied to it.•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 95
You can globally apply one Layer 2 and one Layer 3 ACL in the inbound direction only.• On Layer 2 bridged-group virtual interfaces (BVIs), you can apply both Layer 3 and Layer 2 ACLs.• On Layer 3 virtual LAN (VLAN) interfaces, you can apply only Layer 3 ACLs.• In a redundant configuration, the ACE does not apply a global ACL to the FT VLAN.•
For example, to apply ACL1 to all interfaces in the Admin context, enter the following command in configurationmode:
The syntax of the match access-list command is as follows:
match access-list acl_name
To configure an ACL match statement in a class map, enter the following commands:
ACE_module5/Admin(config)# class-map match-any L4_CLASSACE_module5/Admin(config-cmap)# match access-list ACL1ACE_module5/Admin(config-cmap)# exitACE_module5/Admin(config)# policy-map multi-match L4_POLICYACE_module5/Admin(config-pmap)# class L4_CLASSACE_module5/Admin(config-pmap-c)#
For more details about ACLs and how to configure them, see the Cisco Application Control Engine ModuleSecurity Configuration Guide.
ACL-Related syslogs
When a packet matches an ACL entry, a syslog message is generated based on the following rules:
All ACL deny entries generate a syslog message unless logging is explicitly disabled using the nologging enable command in configuration mode.
•
An ACL permit entry generates a syslog message only if logging is enabled using the logging enablecommand in configuration mode.
•
All implicit deny entries generate the default deny syslog (%ACE-4-106023).•
To minimize syslog message generation, the ACE uses the flow cache as follows:
For the first packet hit on an ACL entry, the ACE generates a syslog and caches the flow (5-tuple) in theconnection table.
1.
For subsequent hits on the same ACL entry, the ACE checks the cache. If it finds the flow in the cache,the ACE increments a hit counter for this entry in the cache and does not generate a syslog.
2.
After some time (the default is 300 seconds, which is configurable in the ACL entry definition in the CLIas the interval_secs option), the ACE generates a syslog and sets the hit count to 0.
3.
However, if at the expiry of the above time, the hit count is 0, the ACE deletes the cache entry silently. Soby default, a cache entry is aged out 600 seconds after the last hit.
4.
Cisco Application Control Engine (ACE) Troubleshooting Guide
Many ACL issues manifest themselves by all traffic or only certain traffic being denied or permitted access to theACE or out of the ACE. Remember that, initially, all traffic to the ACE is denied until you permit traffic using anACL. Every ACL contains an implicit deny at the end of it, so only traffic that you explicitly permit will haveaccess to the ACE. To troubleshoot ACLs, follow these steps:
1. Verify that your ACL configuration is correct for your network application. Make any required changes to therunning-config file, and then test the configuration. If it is satisfactory, save it to the startup-config file using thecopy runnning-config startup-config command.
For example, to display the ACLs that you have configured in your ACE, enter the following command:
ACE_module5/Admin# show running-config access-listGenerating configuration....
access-list ACL1 remark This ACL permits any IP traffic from any source going to any destination except for ICMP traffic originating from 192.168.12.15 255.255.255.192.access-list ACL1 line 8 extended permit ip any anyaccess-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=0) [0x65af0edd]access-list ANYONE line 8 extended permit ip any any
To verify that the configured ACLs are applied to the correct interfaces and in the right directions (input oroutput), enter the following command:
ACE_module5/Admin# show running-config interfaceGenerating configuration....
interface vlan 100 ip address 10.2.1.1 255.255.255.0 access-group input ANYONE access-group output ANYONE no shutdowninterface vlan 200 ip address 192.168.1.1 255.255.255.0 access-group input ACL1 service-policy input MGMT_POLICY no shutdown
2. Verify that you have allocated sufficient resources for ACLs. To display the allocated resources in your ACE,enter the following command:
For example, to allocate a 10 percent minimum and a maximum of unlimited resources for ACL memory in theAdmin virtual context, enter the following commands:
ACE_module5/Admin(config)# resource myclassACE_module5/Admin(config-resource)# limit-resource acl-memory minimum 10 maximum unlimitedACE_module5/Admin(config-resource)# exitACE_module5/Admin(config)# context AdminACE_module5/Admin(config-context)# member myclass
3. Display the details of an individual ACL by using the show access-list acl_name detail command. Thiscommand displays every entry in the specified ACL, the hit counts for each entry, and a 32-bit hexadecimalMD5-hash value that the ACE computes from the access-list command immediately when you configure an ACL.The ACE includes this hash value in deny syslog messages (106023) to help you identify the ACL entry thatcaused the deny syslog. For example to display the details of the ACL1 access control list, enter the followingcommand:
ACE_module5/Admin# show access-list ACL1 detail
access-list:ACL1, elements: 2, status: ACTIVE remark : This ACL permits any IP traffic from any source going to any destination except for ICMP traffic originating from 192.168.12.15 255.255.255.1.access-list ACL1 line 8 extended permit ip any any (hitcount=9) [0x894c1008] <------- 32-bit hexadecimal MD5-hash valueaccess-list ACL1 line 10 extended deny icmp 192.168.12.15 255.255.255.192 any echo code range 1 1 (hitcount=15) [0x65af0edd]
The format of the deny syslog message is as follows:
%ACE-4-106023: Deny protocol number | name src incoming-interface:src-ip dst outgoing-interface:dst-ip by access-group "acl-name"An IP packet was denied by the ACL.
Explanation: This message displays even if you do not have the log option enabled for an ACL. If apacket hits an input ACL, the outgoing interface will not be known. In this case, the ACE prints the
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 98
outgoing interface as undetermined. The source IP and destination IP addresses are the unmapped andmapped addresses for the input and output ACLs, respectively, when used with NAT.
Recommended Action: If messages persist from the same source address, messages may indicate afoot-printing or port-scanning attempt. Contact the remote host administrators.
An ACL merged list is a large ACL that the CP compiles from multiple security ACL entries and policies. Whenthe ACE executes an ACL merged list, it performs multiple actions on a flow that matches the merged list.
4. Display the actions that the ACE will perform on a flow by entering the show acl-merge merged-listcommand. For example, to display the merged list for VLAN 100, enter the following command:
ACE_module5/Admin# show acl-merge merged-list vlan 100 in non-redundant
All ACEs in merged list 2 Total:18 Non-redundant:12
5. If the acl-memory Denied counter in the output of the show resource usage command is incrementing and thePeak (ACL) memory counter has not exceeded the Max Allocated ACL memory counter, the problem may liewith one of the nodes in the ACL merge tree. The ACL merge tree contains several different kinds of nodes (seethe example output below), each of a different size and each with a maximum limit. If you allocate a minimum of10 percent of the ACE resources to ACL memory, the ACE will guarantee 10% of the maximum number of eachnode. If your configuration causes the ACE to exceed the maximum value of one of these nodes, the ACLresource allocation will fail and the acl-memory Denied counter will increment.
To monitor the ACL merge tree node usage in the ACE, enter the following command:
ACE_module5/Admin# show np 1 access-list resource
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 99
ACL Tree Statistics for Context ID: Admin=========================================ACL memory max-limit: NoneACL memory guarantee: 10.00 %MTrie nodes(used/guaranteed/max-limit): 43 / 26214 / 262143 (compressed) <-------| 3 / 2199 / 21999 (uncompressed) <--------|Leaf Head nodes (used/guaranteed/max-limit): | 39 / 26214 / 262143 <--------------------|---- Maximum number of available nodes in the ACELeaf Parameter nodes (used/guaranteed/max-limit):| 594 / 52428 / 524288 <-------------------|Policy action nodes used: 153memory consumed: 23776 bytes resource-limited 4896 bytes other 28672 bytes total.min-guarantee: 7861043 bytes total, 0 % consumed.max-limit: 78610432 bytes total, 0 % consumed.
ACL Tree Statistics for the linecard====================================MTrie nodes(used): 43 (compressed) 3 (uncompressed) <--------------| (shared): 235929 (compressed) 19800 (uncompressed) |Leaf Head nodes (used/shared): 39 / 235929 <-----------------------|---- Number of used nodes in the ACELeaf Parameter nodes (used/shared): 594 / 471860 <-----------------|Policy action nodes (used/shared): 153 / 261990 <------------------|
You can calculate the percentage of use for each node type by dividing the used nodes value by the maximumnumber of nodes and multiplying the result by 100. If any of these percentages exceeds the maximum value ofallocated ACL memory for the context, increase the max value of allocated ACL memory using thelimit-resource acl-memory command in resource class configuration mode so that that value is greater than orequal to the highest used nodes percentage that you calculated. Alternatively, if you are approaching the limits ofACL resource capacity, you may consider consolidating your ACL configuration.
If the ACL nodes are depleted while the ACE is downloading ACL configurations for an interface, the completeACL merged list for that interface is deleted and no traffic flows through that interface. The ACE increments thedownload failure counter in the output of the show interface command and the ACE logs a system message fromthe configuration manager.
6. To trace a packet through a specific ACL, enter the following command:
ACE_module5/Admin# show np 1 access-list trace vlan 130 in protocol 1 source 172.27.16.23 2000 destination 192.168.12.15 3000
Root 0x2c01b00Src Mtrie (0) offset 1 curr 0x2c01b00 child 0x0 leaf 0x10a840 Dst Mtrie (0) offset 2 curr 0x10a840 child 0x0 leaf 0x3c01330 proto ICMP head node 0x4004880 proto node 0x4004880 src op range port 0/65535 dst op range port 0/65535 lineno 112000 inner match line#:112000inner match line#: 112000
packet matched priority 112000
action node 0x4c02460Action Leaf-nodeversion+aceid 0x99 (version 0 ace_id 153 dirty no)action_flag 0x10 (permit no log no punt_to_cp no capture no bridge yes)path ID 0x0
Cisco Application Control Engine (ACE) Troubleshooting Guide
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 101
This article describes ACE network address translation (NAT), how to configure it, and how to troubleshootissues with NAT that you may encounter.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE Network AddressTranslation
•
2 NAT Configuration Guidelines andRestrictions
•
3 Configuring Dynamic NAT and PAT• 4 Configuring Server-Farm Based DynamicNAT
•
5 Configuring Static NAT and Port Redirection• 6 Configuring SNAT with Cookie and LoadBalancing
•
7 Troubleshooting ACE NAT and PAT•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 102
Overview of ACE Network Address Translation
You can configure the ACE to translate a client source IP address to a routable address in the server's network.This process is called source NAT (SNAT). If you want to preserve the client source IP address, do not configureSNAT.
You can also configure the ACE to translate the private address of a server to a global IP address that is accessibleto clients. This process is called destination NAT (DNAT) and protects the server by hiding its real IP addressfrom the Internet.
Besides translating IP addresses, you can configure the ACE to translate TCP and UDP ports. This process iscalled port address translation (PAT).
The ACE provides the following types of NAT and PAT:
Interface-based dynamic NAT• Interface-based dynamic PAT• Server farm-based dynamic NAT• Static NAT• Static port redirection•
NAT Configuration Guidelines and Restrictions
When you configure NAT and PAT on your ACE, keep in mind the following NAT and PAT guidelines andrestrictions:
If a packet egresses an interface that you have not configured for NAT, the ACE transmits the packetuntranslated.
•
You can configure dynamic NAT or static NAT as an input service policy only; you cannot configure itas an output service policy.
•
When you remove a traffic policy from the last VLAN interface on which you applied the service policy,the ACE automatically resets the associated service-policy statistics. The ACE performs this action toprovide a new starting point for the service-policy statistics the next time that you attach a traffic policy toa specific VLAN interface.
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 103
Configuring Dynamic NAT and PAT
Dynamic NAT is typically used for SNAT. When you configure dynamic NAT and PAT, be sure to configure aninterface for the client-side VLAN and an interface for the server-side VLAN.
The following SNAT configuration example shows the commands that you use to configure dynamic NAT andPAT on your ACE. In this SNAT example, packets that ingress the ACE from the 192.168.12.0 network aretranslated to one of the IP addresses in the NAT pool defined on VLAN 200 by the nat-pool command. The patkeyword indicates that ports higher than 1024 are also translated.
Note: If you are operating the ACE in one-arm mode, omit the client-side interface VLAN 100 andconfigure the service policy on interface VLAN 200.
class-map match-any NAT_CLASS match access-list NAT_ACCESS
policy-map multi-match NAT_POLICY class NAT_CLASS nat dynamic 1 vlan 200
interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown
interface vlan 200 mtu 1500 ip address 172.27.16.2 255.255.255.0 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 pat no shutdown
Configuring Server-Farm Based Dynamic NAT
The following SNAT configuration example shows the commands that you use to configure server farm-baseddynamic NAT on your ACE. In this SNAT example, real server addresses on the 172.27.16.0 network aretranslated to one of the IP addresses in the NAT pool defined on VLAN 200 by the nat-pool command.
Note: If you are operating the ACE in one-arm mode, omit interface VLAN 100 and configure the servicepolicy on interface VLAN 200.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 104
class-map type http loadbalance match-any L7_CLASS match http content .*cisco.comclass-map match-any NAT_CLASS match access-list NAT_ACCESS
policy-map type loadbalance http first-match L7_POLICY class L7_CLASS serverfarm SFARM1 nat dynamic 1 vlan 200 serverfarm primarypolicy-map multi-match NAT_POLICY class NAT_CLASS loadbalance policy L7_POLICY loadbalance vip inservice
interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown
interface vlan 200 mtu 1500 ip address 172.27.16.2 255.255.255.0 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 no shutdown
Configuring Static NAT and Port Redirection
The following DNAT configuration example shows those sections of the running configuration that are related tothe commands necessary to configure static NAT and port redirection on your ACE. Typically, this configurationis used for DNAT, where HTTP packets that are destined to 192.0.0.0/8 and ingress the ACE on VLAN 101 aretranslated to 10.0.0.0/8 and port 8080. In this example, the servers are hosting HTTP on custom port 8080.
access-list acl1 line 10 extended permit tcp 10.0.0.0 255.0.0.0 eq 8080 any
class-map match-any NAT_CLASS match access-list acl1
interface vlan 100 mtu 1500 ip address 192.168.1.100 255.255.255.0 service-policy input NAT_POLICY no shutdown
interface vlan 101 mtu 1500 ip address 172.27.16.100 255.255.255.0 no shutdown
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 105
Configuring SNAT with Cookie and Load Balancing
The following configuration example shows those commands necessary to configure SNAT (dynamic NAT) withcookie load balancing. Any source host that sends traffic to the VIP 20.11.0.100 is translated to one of the freeaddresses in the NAT pool in the range 30.11.100.1 to 30.11.200.1, inclusive. If you want to use PAT instead ofNAT, replace nat dynamic 1 vlan 2021 with nat dynamic 2 vlan 2021 in the L7SLBCookie policy map.
server host http ip address 30.11.0.10 inserviceserverfarm host httpsf rserver http inservice
class-map match-any vip4 2 match virtual-address 20.11.0.100 tcp eq wwwclass-map type http loadbalance match-any L7SLB_Cookie 3 match http cookie JG cookie-value ?.*?
policy-map type loadbalance first-match L7SLB_Cookie class L7SLB_Cookie serverfarm httpsfpolicy-map multi-match L7SLBCookie class vip4 loadbalance vip inservice loadbalance L7SLB_Cookie nat dynamic 1 vlan 2021
interface vlan 2020 ip address 20.11.0.2 255.255.0.0 alias 20.11.0.1 255.255.0.0 peer ip address 20.11.0.3 255.255.0.0 service-policy input L7SLBCookie no shutdowninterface vlan 2021 ip address 30.11.0.2 255.255.0.0 alias 30.11.0.1 255.255.0.0 peer ip address 30.11.0.3 255.255.0.0 fragment min-mtu 68 nat-pool 2 30.11.201.1 30.11.201.1 netmask 255.255.255.255 pat nat-pool 3 30.11.202.1 30.11.202.3 netmask 255.255.255.255 nat-pool 1 30.11.100.1 30.11.200.1 netmask 255.255.255.255 no shutdown
Troubleshooting ACE NAT and PAT
To verify your NAT and PAT configurations and make any necessary corrections, follow these steps:
1. Display your NAT and PAT configurations by entering the following commands:
ACE_module5/Admin# show running-config class-map
class-map match-any L4_CLASS 2 match access-list ACL1
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 106
ACE_module5/Admin# show running-config policy-map
policy-map multi-match NAT_POLICY class NAT_CLASS nat dynamic 1 vlan 200
interface vlan 100 ip Address 192.168.12.2 mtu 1500 service-policy input NAT_POLICY no shutdowninterface vlan 200 ip address 172.27.16.2 255.255.255.0 mtu 1500 access-group input acl1 nat-pool 1 172.27.16.15 172.27.16.24 netmask 255.255.255.0 pat no shutdown
2. Use the show xlate command to verify that dynamic NAT and PAT, and static NAT and port redirection, aretaking place properly.
Dynamic NAT Example
The following example output of the show xlate command shows dynamic NAT (SNAT in this example).When you use Telnet from IP address 172.27.16.5 in VLAN 2020, the ACE translates it to IP address192.168.100.1 in VLAN 2021.
host1/Admin# show xlate global 192.168.100.1 192.168.100.10
NAT from vlan2020:172.27.16.5 to vlan2021:192.168.100.1 count:1
Dynamic PAT Example
The following example shows dynamic PAT. When you use Telnet from IP address 172.27.16.5 port38097 in VLAN 2020, the ACE translates it to IP address 192.168.201.1 port 1025 in VLAN 2021.
host1/Admin# show xlate
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 107
TCP PAT from vlan2020:172.27.16.5/38097 to vlan2021:192.168.201.1/1025
Static NAT Example
The following example shows static NAT. The ACE maps real IP address 172.27.16.5 to IP address192.168.210.1.
host1/Admin# show xlate
NAT from vlan2020:172.27.16.5 to vlan2021:192.168.210.1 count:1
host1/Admin# show conn
total current connections : 2
conn-id dir prot vlan source destination state----------+---+----+----+----------------+----------------+----------+7 in TCP 2020 172.27.16.5 192.168.100.1 ESTAB6 out TCP 2021 192.168.100.1 192.168.210.1 ESTAB
Static Port Redirection (Static PAT) Example
The following example shows static port redirection (DNAT in this example). A host at IP address192.168.0.10:37766 uses Telnet to connect to IP address 192.168.211.1:3030 on VLAN 2021 on theACE. The ACE maps IP address 172.27.0.5:23 on VLAN 2020 to IP address 192.168.211.1:3030 onVLAN 2021.
host1/Admin# show xlate
TCP PAT from vlan2020:172.27.0.5/23 to vlan2021:192.168.211.1/3030Mar 24 2006 20:05:41 : %ACE-7-111009: User 'admin' executed cmd: show xlate
host1/Admin# show conn
total current connections : 2
conn-id dir prot vlan source destination state----------+---+----+----+------------------+------------------+------+6 in TCP 2021 192.168.0.10:37766 192.168.211.1:3030 ESTAB7 out TCP 2020 172.27.0.5:23 192.168.0.10:1025 ESTAB
3. To display the NAT policy and pool information for the current context, enter the show nat-fabric command.The syntax of this command is as follows:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 108
src-nat policy_id mapped_if -- Displays the specified source NAT policy information. To obtain thevalues for the policy_id and mapped_if arguments, view the policy_id and mapped_if fields displayed bythe show nat-fabric policies command.
dst-nat static_xlate_id -- Displays the static address translation for the specified static XLATE ID. Toobtain the value for the static_xlate_id argument, view the static_xlate_id field displayed by the shownat-fabric policies command.
nat-pools -- Displays NAT pool information for a dynamic NAT policy.
implicit-pat -- Displays the implicit PAT policies.
global-static -- Displays global static NAT information when the static command in global configurationmode is configured.
ACE_module5/Admin# show nat-fabric policies
Nat objects:
NAT object Hash Bucket: 9 NAT object ID:2 mapped_if:8 policy_id:1 type:DYNAMIC nat_pool_id:4 Pool ID:4 PAT:1 pool_id:1 mapped_if:8 Ref_count:1 ixp_binding:in all IXPs lower:172.27.16.15 upper:172.27.16.24 Bitmap-ID:40 List of NAT object IDs: 2
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 109
This article describes ACE health monitoring (probes), how to configure it, and how to troubleshoot issues withprobes that you may encounter.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE HealthMonitoring
1.1 Configuring Probes♦ 1.2 Example of a ProbeConfiguration
♦
•
2 Troubleshooting ACE HealthMonitoring
2.1 Troubleshooting anHTTP Probe Error
♦
2.2 Troubleshooting anHTTPS Probe Error
♦
2.3 Troubleshooting anSNMP Probe Issue
♦
2.4 Using the Last StatusCode Field
♦
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 110
Overview of ACE Health Monitoring
This section describes health monitoring on the ACE. The ACE uses probes that you configure to track the stateof a server. By default, no probes are configured in the ACE. Also referred to as out-of-band (OOB) healthmonitoring, the ACE verifies the server response to a probe or checks for any network problems that can preventa client from reaching a server. Based on the server response, the ACE can place the server in or out of serviceand can make reliable load-balancing decisions.
You can also use health monitoring to detect failures for a gateway or a host in high-availability (redundant)configurations. For more information, see the Cisco Application Control Engine Module Administration Guide.
The ACE evaluates the health of a server by marking the probes as follows:
Passed?The server returns a valid response.•
Failed?The server fails to provide a valid response to the ACE and the ACE is unable to reach a server fora specified number of retries.
•
By configuring the ACE for health monitoring, the ACE sends active probes periodically to determine the serverstate. The ACE supports 4096 unique probe configurations, which includes ICMP, TCP, HTTP, and otherpredefined health probes. The ACE can execute only up to 200 concurrent scripted probes at a time. The ACEalso allows the opening of 2048 sockets simultaneously.
You can associate the same probe with multiple real servers or server farms. Each time that you use the sameprobe again, the ACE counts it as another probe instance. You can allocate a maximum of 16,000 probe instances.
Configuring Probes
You can configure health probes on the ACE to actively make connections and explicitly send traffic to servers.The probes determine whether the health status of a server passes or fails by the server's response.
Configuring active probes is a three-step process:
1. Configure the health probe with a name, type, and attributes.
2. Associate the probe with one of the following:
Cisco Application Control Engine (ACE) Troubleshooting Guide
A real server and then associate the real server with a server farm. You can associate a singleprobe or multiple probes with real servers within a server farm.
◊
A server farm. All servers in the server farm receive probes of the associated probe types.◊
3. Place the real server or server farm in service.
Example of a Probe Configuration
The following example shows a running configuration that load balances DNS traffic across multiple real serversand transmits and receives UDP data that spans multiple packets. The configuration uses a UDP health probe.
access-list ACL1 line 10 extended permit ip any any
probe udp UDP interval 5 passdetect interval 10 description THIS PROBE IS INTENDED FOR LOAD BALANCING DNS TRAFFIC port 53 send-data UDP_TEST
rserver host SERVER1 ip address 192.168.10.45 inservicerserver host SERVER2 ip address 192.168.10.46 inservicerserver host SERVER3 ip address 192.168.10.47 inservice
class-map match-all L4UDP-VIP_114:UDP_CLASS 2 match virtual-address 192.168.120.114 udp eq 53
policy-map type loadbalance first-match L7PLBSF_UDP_POLICY class class-default serverfarm SFARM1policy-map multi-match L4SH-Gold-VIPs_POLICY class L4UDP-VIP_114:UDP_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_UDP_POLICY loadbalance vip icmp-reply nat dynamic 1 vlan 120 connection advanced-options 1SECOND-IDLE
interface vlan 120 description Upstream VLAN_120 - Clients and VIPs
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 112
ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4SH-Gold-VIPs_POLICY no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Troubleshooting ACE Health Monitoring
This section describes how to troubleshoot two common probe configuration issues.
Troubleshooting an HTTP Probe Error
In this first scenario, you have configured an HTTP probe, but the real server's health status is displayed asFAILED and the Last disconnect err field indicates that an invalid status code was received as displayed in theshow probe detail command output. You have checked your server and it is up and running. A packet capture onthe server also shows that everything is fine. Where is the issue?
1. Display the probe status details by entering the following command:
ACE_module5/Admin# show probe detail
probe : HTTP_PROBE type : HTTP state : ACTIVE description :---------------------------------------------- port : 80 address : 0.0.0.0 addr type : - interval : 10 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 http method : GET http url : / conn termination : GRACEFUL expect offset : 0 , open timeout : 1 expect regex : - send data : - ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ rserver : SERVER1 192.168.10.45 80 -- 2 2 0 FAILED
Socket state : CLOSED No. Passed states : 0 No. Failed states : 1
No. Probes skipped : 0 Last status code : 200 <------- Last status code from server No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : Received invalid status code <------- Last probe time : Tue Apr 7 16:17:26 2009 Last fail time : Tue Apr 7 16:17:16 2009 Last active time : Never
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 113
The Last disconnect err field indicates that the ACE received an invalid status code. This error means that youhave has not configured the expect status command for the probe.
2. Confirm this finding by entering the following command:
ACE_module5/Admin# show running-config probeGenerating configuration....
probe http HTTP_PROBE interval 10 passdetect interval 10 open 1
3. Correct the problem by entering the following commands:
ACE_module5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_module5/Admin(config)# probe http HTTP_PROBEACE_module5/Admin(config-probe-http)# expect status 200 200 <------- 200 indicates the 200 OK message from the serverACE_module5/Admin(config-probe-http)# end
4. Confirm the configuration by entering the following command:
ACE_module5/Admin# show running-config probeGenerating configuration....
probe http HTTP_PROBE interval 10 passdetect interval 10 expect status 200 200 open 1
5. Display the probe status details again and observe that the server health status value is SUCCESS by enteringthe following command:
ACE_module5/Admin# show probe HTTP_PROBE detail
probe : HTTP_PROBE type : HTTP state : ACTIVE description :---------------------------------------------- port : 80 address : 0.0.0.0 addr type : - interval : 10 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 http method : GET http url : / conn termination : GRACEFUL expect offset : 0 , open timeout : 1 expect regex : - send data : - ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ rserver : SERVER1 192.168.10.45 80 -- 24 15 9 SUCCESS
Socket state : CLOSED
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 114
No. Passed states : 1 No. Failed states : 1 No. Probes skipped : 0 Last status code : 200 No. Out of Sockets : 0 No. Internal error: 0 Last disconnect err : - <------- No error indicated now. The probe is successful. Last probe time : Tue Apr 7 16:21:05 2009 Last fail time : Tue Apr 7 16:17:16 2009 Last active time : Tue Apr 7 16:20:05 2009
Troubleshooting an HTTPS Probe Error
In addition to the methods for Troubleshooting an HTTP Probe Error, use SSL statistics to troubleshoot HTTPSprobe failures. HTTPS probe traffic runs in the Admin virtual context so view the output of the show stats cryptoclient command in that context.
Troubleshooting an SNMP Probe Issue
In this scenario, you have configured an SNMP probe, but the Last disconnect err field indicates that the sum ofthe weights does not add up to the maximum weight value as displayed in the output of the show probe detailcommand.
1. Display the probe status details by entering the following command:
ACE_module5/test# show probe detail
probe : SNMP_PROBE type : SNMP state : ACTIVE description : snmp probe---------------------------------------------- port : 161 address : 0.0.0.0 addr type : - interval : 15 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 version : 2c community : test_comm oid string #1 : .1.3.6.1.2.1.4.3.0 type : ABSOLUTE max value : 1000000000 weight : 10000 threshold : 1000000000 ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ serverfarm : least-loaded, predictor least-loaded real : SERVER1[0] 192.168.10.45 161 -- 0 0 0 INIT
Socket state : CLOSED No. Passed states : 0 No. Failed states : 0 No. Probes skipped : 0 Last status code : 0 No. Out of Sockets : 0 No. Internal error: 30
Last disconnect err : Sum of weights don't add up to max weight value <------- Error condition Last probe time : Never Last fail time : Never Last active time : Never Server load : 16000 <------- Note the server load value
The reason for this error is that the weight command needs to be configured when you have multiple OIDsconfigured for a single probe and from those OIDs if you want to give priority to a specific OID.
Cisco Application Control Engine (ACE) Troubleshooting Guide
The sum of the weights should equal 16000 (see the Server Load field). For a single OID, the weight commanddoes not have any significance.
2. Display the probe configuration by entering the following command:
ACE_module5/Admin# show running-config probe
probe snmp SNMP_PROBE description snmp probe port 161 interval 15 passdetect interval 10 version 2c community TEST_COMM oid .1.3.6.1.2.1.4.3.0 type absolute max 1000000000 weight 10000 <-------
In the above configuration, the weight is configured as 10000 for a single OID. The ACE is expecting anotherOID to be configured in the probe and the sum of both weights should equal 16000.
The configuration is not complete and the ACE is expecting additional parameters in the probe configuration.Because there is not another OID in the configuration, the ACE is not able to calculate the load and that is whythe "Sum of weights don't add up to max weight value" error message appears.
3. Resolve the issue by modifying the probe configuration as follows:
probe snmp SNMP_PROBE description test port 161 interval 15 passdetect interval 60 version 2c community test_comm oid .1.3.6.1.2.1.4.3.0 type absolute max 1000000000 weight 10000 oid .1.3.6.1.2.1.4.10.0 type absolute max 1000000000
weight 6000 <------- 10000 + 6000 = 16000
4. Display the probe status details again by entering the following command:
ACE_module5/test# show probe SNMP_PROBE detail
probe : snmp1 type : SNMP state : ACTIVE description : snmp probe---------------------------------------------- port : 161 address : 0.0.0.0 addr type : - interval : 15 pass intvl : 10 pass count : 3 fail count: 3 recv timeout: 10 version : 2c community : test_comm
oid string #1 : .1.3.6.1.2.1.4.3.0 type : ABSOLUTE max value : 1000000000 weight : 10000 threshold : 1000000000
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 116
oid string #2 : .1.3.6.1.2.1.4.10.0 type : ABSOLUTE max value : 1000000000 weight : 6000 threshold : 1000000000 ------------------ probe results ------------------ associations ip-address port porttype probes failed passed health ------------ ---------------+-----+--------+--------+--------+--------+------ serverfarm : least-loaded, predictor least-loaded real : SERVER1[0] 192.168.10.45 161 -- 4143 0 4143 SUCCESS
Socket state : CLOSED No. Passed states : 1 No. Failed states : 0 No. Probes skipped : 0 Last status code : 0 No. Out of Sockets : 0 No. Internal error: 0 Last disconnect err : - <------- No error indicated now. The probe is successful.
Last probe time : Mon Apr 6 09:12:54 2009 Last fail time : Never Last active time : Sun Apr 5 15:57:28 2009 Server load : 0
Using the Last Status Code Field
Details regarding the last status code field can be provided for nontrivial probes. For example, in the case ofscripted probe PROBENOTICE_PROBE, the status code 30001 means that the probe is successful and the value30002 indicates an error in probe arguments. The last disconnect error for status code 30002 displays "Did notreceive correct response from the server," but the actual issue is related to arguments in the probe configuration,which can be checked by looking at the script for the probe.
ACE_module5/Admin# show probe TEST detail
probe : TEST type : SCRIPTED state : ACTIVE description :---------------------------------------------- port : 0 address : 0.0.0.0 addr type : - interval : 15 pass intvl : 20 pass count : 3 fail count: 3 recv timeout: 10 script filename : PROBENOTICE_PROBE --------------------- probe results -------------------- probe association probed-address probes failed passed health ------------------- ---------------+----------+----------+----------+------- serverfarm : sf1 real : rs2[0] 23.0.0.5 4082 54 4028 SUCCESS
Socket state : RESET No. Passed states : 6 No. Failed states : 5 No. Probes skipped : 8 Last status code : 30001 <------- Indicates success No. Out of Sockets : 0 No. Internal error: 0 Last disconnect err : - Last probe time : Wed Apr 8 04:44:41 2009 Last fail time : Tue Apr 7 12:02:10 2009 Last active time : Tue Apr 7 12:03:45 2009
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 117
This article describes how to troubleshoot Layer 4 (L4) load balancing on the ACE.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE L4 Load Balancing1.1 Classifying L4 Traffic for ServerLoad Balancing
♦
1.2 Example of a Layer 4 Load-BalancingConfiguration
♦
•
2 Troubleshooting L4 Load Balancing on theACE
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 118
Overview of ACE L4 Load Balancing
Load balancing at L4 involves selecting a server in a server farm to service a client request based on the VIPaddress and protocol in the request. You configure a class map to classify (match) interesting traffic arriving at theACE and associate the class map with a policy map to perform an action on the traffic based on the classification.With L4 load balancing, the ACE selects a server based on the first packet it receives in a particular flow. See the"Overview of ACE Connection Handling" section in the Troubleshooting Connectivity article.
For detailed information about ACE load balancing, see the Cisco Application Control Engine Module ServerLoad Balancing Configuration Guide.
Classifying L4 Traffic for Server Load Balancing
You classify inbound network traffic destined to or passing through the ACE based on a series of flow matchcriteria specified by a class map. Each class map defines a traffic classification, which is network traffic that is ofinterest to you. A policy map defines a series of actions (functions) that you want applied to a set of classifiedinbound or outbound traffic.
ACE L3 and L4 traffic policies support the following server load-balancing (SLB) traffic attributes:
Source or destination IP address• Source or destination port• Virtual IP (VIP) address• IP protocol•
The three major steps in the traffic classification process are as follows:
Create a class map using the class-map command and the associated match commands, which comprisea set of match criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocolclassifications.
1.
Create a policy map using the policy-map command, which refers to the class maps and identifies aseries of actions to perform based on the traffic match criteria.
2.
Activate the policy map by associating it with a specific VLAN interface or globally with all VLANinterfaces using the service-policy command to filter the traffic received by the ACE.
3.
Cisco Application Control Engine (ACE) Troubleshooting Guide
Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7policies that the ACE uses for SLB. The figure also shows how you associate the various components of the SLBpolicy configuration with each other.
Figure 1. SLB Flow Diagram
Example of a Layer 4 Load-Balancing Configuration
The following example shows a L4 load-balancing configuration:
access-list ACL1 line 10 extended permit ip any any
rserver host SERVER1 ip address 192.168.252.245 inservice
rserver host SERVER2
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 120
ip address 192.168.252.246 inservice
rserver host SERVER3 ip address 192.168.252.247 inservice
rserver host SERVER4 ip address 192.168.252.248 inservice
rserver host SERVER5 ip address 192.168.252.249 inservice
rserver host SERVER6 ip address 192.168.252.250 inservice
class-map match-all L4WEB_CLASS 2 match virtual-address 192.168.120.112 tcp eq www
policy-map type loadbalance first-match LB_WEB_POLICY class class-default serverfarm SFARM1 backup SFARM2
policy-map multi-match L4WEB_POLICY class L4WEB_CLASS loadbalance vip inservice loadbalance policy LB_WEB_POLICY loadbalance vip icmp-reply active nat dynamic 1 VLAN 120
interface vlan 100 description Upstream VLAN_100 - Clients and VIPs ip address 192.168.120.1 255.255.255.0
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 121
fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4WEB_POLICY no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Troubleshooting L4 Load Balancing on the ACE
To troubleshoot L4 load-balancing issues, follow these steps:
1. Ensure that your load-balancing configuration is correct and that the following conditions exist:
Real servers have valid IP addresses and are in service.◊ Servers are associated with server farms of the same type.◊ A load-balancing policy exists with an associated server farm and is associated with a L4multimatch policy.
◊
An L4 class map contains a valid match virtual-address command and is associated with the L4multimatch policy map.
◊
The L4 policy is applied to the appropriate active interface using a service policy.◊ A static route is configured for the server network.◊
Use the following show commands:
show running-config rserver◊ show running-config serverfarm◊ show running-config policy-map◊ show running-config class-map◊ show running-config interface◊ show ip route◊
2. Check the ACE connectivity. See the "Troubleshooting Connectivity" section.
3. Verify that the L4 VIP class map is referenced in a L4 policy by entering the following command. Also, checkthe following fields:
VIP address and port◊ VIP state◊ Hit count◊ Dropped connections◊
ACE_module5/Admin# show service-policy L4WEB_POLICY detail
Status : ACTIVEDescription: ------------------------------------------Interface: vlan 100
VIP State: INSERVICE <------- VIP state should be INSERVICE curr conns : 0 , hit count : 56
dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded client pkt count : 6297 , client byte count: 1047583 server pkt count : 1238 , server byte count: 1325495 L7 Loadbalance policy : LB_WEB_POLICY class/match : class-default LB action : serverfarm: SFARM1
hit count : 0 <-------|-- Check these counters to see if they are increasingdropped conns : 0 <-------|
The dropped conns counter under a VIP in the output of the show service policy detail command is incrementedwhenever the ACE discards a connection request destined to that VIP. There are several reasons why the ACEdiscards such connection requests. For example:
If all the real servers in the server farm associated with the VIP go down, then the VIP will go down. So,all the incoming connections to that VIP are discarded.
•
If the URL in a connection request to the VIP is unknown, then the connection request is discarded.• If the server to which the ACE load balances the connection does not respond to the request, then, afterthe maximum number of retries, the ACE discards the connection.
•
The dropped conns counter is cumulative and the value may comprise entries from any of the following showcommand counters:
show stats loadbalance•
- Total Layer4 rejections- Total Layer7 rejections- Total Layer4 LB policy misses- Total Layer7 LB policy misses- Total times rserver was unavailable
show stats connection•
- Total Connections Timed-out- Total Connections Failed
The failures counter of the show serverfarm serverfarm_name command•
The Total drop decisions counter of the show stats inspect command•
4. Verify that the L4 policy is applied as a service policy to an active interface by entering the followingcommand:
ACE_module5/Admin# show running-config interfaceGenerating configuration....
Cisco Application Control Engine (ACE) Troubleshooting Guide
5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also,check the IP address, state, and the connection statistics for each real server that is configured in the server farm.
ACE_module5/Admin# show serverfarm SFARM1 detail
serverfarm : SFARM1, type: HOSTtotal rservers : 3active rservers: 3description : -state : ACTIVE <------- Current state of the server farmpredictor : ROUNDROBIN <------- Load-balancing method weight : - autoadjust : MAXLOADfailaction : -back-inservice : 40partial-threshold : 40num times failover : 0num times back inservice : 0total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded--------------------------------- ----------connections----------- real weight state current total failures ---+---------------------+------+------------+----------+----------+--------- rserver: SERVER1
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 124
6. Check the L4 load-balance statistics by entering the following command:
ACE_module5/Admin# show stats loadbalance
+------------------------------------------++------- Loadbalance statistics -----------++------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 0 Total Layer7 decisions : 0 Total Layer7 rejections : 0 Total Layer4 LB policy misses : 0 Total Layer7 LB policy misses : 0 Total times rserver was unavailable : 0 Total ACL denied : 0 Total IDMap Lookup Failures : 0
Note: The ID Map is used to map real servers and server farms between the local and the remote peers in aredundant configuration. The Total IDMap Lookup Failures field increments if the local ACE fails tofind the local ACE to peer ACE ID mapping. A failure can occur if the peer ACE did not send a properremote ID for the local ACE to look up and so the local ACE could not perform a mapping or if the IDMap table was not created.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 125
This article describes how to diagnose and troubleshoot ACE L7 load-balancing issues.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE Layer 7 Load Balancing1.1 Load-Balancing Predictors♦ 1.2 Classifying L7 Traffic for ServerLoad Balancing
♦
•
2 Example of a L7 Load-BalancingConfiguration
•
3 Troubleshooting Layer 7 Load Balancing•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 126
Overview of ACE Layer 7 Load Balancing
Layer 7 server load balancing (SLB) is the process that the load-balancing device uses to decide which servershould fulfill a client request for an L7 service. For example, a client request may consist of a HyperTextTransport Protocol (HTTP) GET for a web page or a File Transfer Protocol (FTP) GET to download a file. Thejob of the load balancer is to select the server that can successfully fulfill the client request and do so in theshortest amount of time without overloading either the server or the server farm as a whole.
The ACE supports the load balancing of the following protocols:
Depending on the load-balancing algorithm?or predictor?that you configure, the ACE performs a series of checksand calculations to determine which server can best service each client request. The ACE bases server selectionon several factors including the source or destination address, cookies, URLs, HTTP headers, or the server withthe fewest connections with respect to load.
For detailed information about ACE load balancing, see the Cisco Application Control Engine Module ServerLoad Balancing Configuration Guide.
Load-Balancing Predictors
The ACE uses the following predictors to select the best server to fulfill a client request:
Application response?Selects the server with the lowest average response time for the specifiedresponse-time measurement based on the current connection count and server weight (if configured).
•
Hash address?Selects the server using a hash value based on either the source or destination IP address orboth. Use these predictors for firewall load balancing (FWLB). For more information about FWLB, seeConfiguring Firewall Load Balancing in the Cisco Application Control Engine Module ServerLoad-Balancing Configuration Guide (Software Version A2(1.0)).
•
Hash content?Selects the server using a hash value based on a content string in the Trusted Third Parties(TTP) packet body.
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
Hash cookie?Selects the server using a hash value based on a cookie name.•
Hash header?Selects the server using a hash value based on the HTTP header name.•
Hash URL?Selects the server using a hash value based on the requested URL. You can specify abeginning pattern and an ending pattern to match in the URL. Use this predictor method to load balancecache servers.
•
Least bandwidth?Selects the server that processed the least amount of network traffic based on theaverage bandwidth that the server used over a specified number of samples.
•
Least connections?Selects the server with the fewest number of active connections based on the serverweight. For the least-connections predictor, you can configure a slow-start mechanism to avoid sending ahigh rate of new connections to servers that you have just put into service.
•
Least loaded?Selects the server with the lowest load based on information obtained from Simple NetworkManagement Protocol (SNMP) probes. To use this predictor, you must associate an SNMP probe with it.
•
Round-robin?Selects the next server in the list of real servers based on the server weight (weightedround-robin). Servers with a higher weight value receive a higher percentage of the connections. This isthe default predictor.
•
Note: The hash predictor methods do not recognize the weight value that you configure for real servers. TheACE uses the weight that you assign to real servers only in the least-connections, application-response,and round-robin predictor methods.
Classifying L7 Traffic for Server Load Balancing
You classify inbound network traffic destined to or passing through the ACE based on a series of flow matchcriteria specified by a class map. Each class map defines a traffic classification, which is network traffic that is ofinterest to you. A policy map defines a series of actions (functions) that you want applied to a set of classifiedinbound or outbound traffic.
ACE traffic policies support the following server load-balancing (SLB) traffic attributes:
Layer 3 and Layer 4 connection information?Source or destination IP address, source or destination port,virtual IP address, and IP protocol
•
Layer 7 protocol information?Hypertext Transfer Protocol (HTTP) cookie, HTTP URL, HTTP header,Remote Authentication Dial-In User Service (RADIUS), Remote Desktop Protocol (RDP), Real-TimeStreaming Protocol (RTSP), Session Initiation Protocol (SIP), and Secure Sockets Layer (SSL)
•
The three major steps in the traffic classification process are as follows:
Create a class map using the class-map command and the associated match commands, which comprisea set of match criteria related to Layer 3 and Layer 4 traffic classifications or Layer 7 protocolclassifications.
1.
Create a policy map using the policy-map command, which refers to the class maps and identifies aseries of actions to perform based on the traffic match criteria.
2.
Activate the policy map by associating it with a specific VLAN interface or globally with all VLANinterfaces using the service-policy command to filter the traffic received by the ACE.
3.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 128
Figure 1 provides a basic overview of the process required to build and apply the Layer 3, Layer 4, and Layer 7policies that the ACE uses for SLB. The figure also shows how to associate the various components of the SLBpolicy configuration with each other.
Figure 1. SLB Flow Diagram
Example of a L7 Load-Balancing Configuration
The following example shows a running configuration that includes multiple class maps and policy maps thatdefine a traffic policy for SLB. In this configuration, when a server farm is chosen for a connection, theconnection is sent to a real server based on one of several load-balancing predictors. The leastconns predictormethod load balances connections to the server that has the lowest number of open connections.
access-list ACL1 line 10 extended permit ip any any
probe tcp TCP interval 5 faildetect 2
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 129
passdetect interval 10 open 3
parameter-map type http PERSIST-REBALANCE persistence-rebalance <---------------------------- Enabled by default in the ACE appliance. Enabled by default in the ACE module in software version A4(1.0) and later.parameter-map type connection PRED-CONNS-UDP_CONN set timeout inactivity 300
rserver SERVER1 ip address 10.1.0.2 inservicerserver SERVER2 ip address 10.1.0.3 inservicerserver SERVER3 ip address 10.1.0.4 inservicerserver SERVER4 ip address 10.1.0.5 inservicerserver SERVER5 ip address 10.1.0.6 inservicerserver SERVER6 ip address 10.1.0.7 inservicerserver SERVER7 ip address 10.1.0.8 inservicerserver SERVER8 ip address 10.1.0.9 inservice
class-map match-all L4PRED-CONNS-UDP-VIP_128:2222_CLASS 2 match virtual-address 192.168.120.128 udp eq 0 class-map match-all L4PRED-CONNS-VIP_128:80_CLASS 2 match virtual-address 192.168.120.128 tcp eq www class-map match-all L4PREDICTOR_117:80_CLASS 2 match virtual-address 192.168.120.117 tcp eq www
policy-map type loadbalance first-match L7PLBSF_PRED-CONNS_POLICY class class-default serverfarm PRED-CONNSpolicy-map type loadbalance first-match L7PLBSF_PRED-CONNS-UDP_POLICY class class-default serverfarm PRED-CONNS-UDPpolicy-map type loadbalance first-match L7PLBSF_PREDICTOR_POLICY class class-default sticky-serverfarm STKY-GRP-43policy-map multi-match L4SH-Gold-VIPs_POLICY class L4PREDICTOR_117:80_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PREDICTOR_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE class L4PRED-CONNS-VIP_128:80_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PRED-CONNS_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE class L4PRED-CONNS-UDP-VIP_128:2222_CLASS loadbalance vip inservice loadbalance policy L7PLBSF_PRED-CONNS-UDP_POLICY loadbalance vip icmp-reply active nat dynamic 1 vlan 120 appl-parameter http advanced-options PERSIST-REBALANCE connection advanced-options PRED-CONNS-UDP_CONN
interface vlan 120
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 131
description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.70 netmask 255.255.255.0 pat service-policy input L4SH-Gold-VIPs_POLICY no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Troubleshooting Layer 7 Load Balancing
To troubleshoot L7 load-balancing issues, use the following steps:
1. Ensure that your load-balancing configuration is correct and that the following conditions exist:
Real servers have valid IP addresses and are in service◊ Servers are associated with server farms of the same type◊ L7 load-balancing policy exists with an associated server farm and that the L7 load-balancingpolicy is associated with a L4 multimatch policy
◊
An L4 class map contains a valid match virtual-address command and is associated with the L4multimatch policy map
◊
The L4 policy is applied to the appropriate active interface using a service policy◊ A static route is configured for the server network◊
Use the following show commands to verify your load-balancing configuration:
show running-config rserver◊ show running-config serverfarm◊ show running-config policy-map◊ show running-config class-map◊ show running-config interface◊ show ip route◊
2. Check the ACE connectivity. See the Troubleshooting Connectivity section.
3. Verify that the L7 load-balancing policy is referenced in the L4 policy by entering the following command.Also, check the following fields:
VIP address, protocol, and port◊ VIP state◊ Hit count◊ Dropped connections◊
ACE_module5/Admin# show service-policy L4WEB_POLICY detail
Status : ACTIVEDescription: ------------------------------------------Interface: vlan 100 service-policy: L4WEB_POLICY class: L4WEB_CLASS
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 132
VIP Address: Protocol: Port: 192.168.120.112 tcp eq 80 <------- VIP address, protocol, and port
loadbalance:L7 loadbalance policy: LB_WEB_POLICY <-------L7 load-balancing policy referenced in the L4 multimatch policy
VIP State: INSERVICE <------- VIP state should be INSERVICE curr conns : 0 , hit count : 56
dropped conns : 14 <------- Number of attempted connections to this VIP that the ACE discarded client pkt count : 6297 , client byte count: 1047583 server pkt count : 1238 , server byte count: 1325495
5. Check the total conn-dropcount field for the primary server farm in the output of the following command. Also,check the IP address, state, and the connection statistics for each real server that is configured in the server farm.
ACE_module5/Admin# show serverfarm SFARM1 detail
serverfarm : SFARM1, type: HOSTtotal rservers : 3active rservers: 3description : -state : ACTIVE <------- Current state of the server farmpredictor : ROUNDROBIN <------- Load-balancing method weight : - autoadjust : MAXLOADfailaction : -back-inservice : 40partial-threshold : 40num times failover : 0num times back inservice : 0total conn-dropcount : 0 <------- Total number of connection attempts to this server farm that the ACE discarded--------------------------------- ----------connections----------- real weight state current total failures ---+---------------------+------+------------+----------+----------+--------- rserver: SERVER1
192.168.252.245:0 10 INSERVICE 0 0 0 <------- Real server IP address, state, and connection statistics
Cisco Application Control Engine (ACE) Troubleshooting Guide
Note: The connection failures counter increments only if the ACE attempts to load balance a connection andthe ACE does not receive a SYN-ACK from the real server in response to a SYN or if the real serverresponds to the SYN with a RST.
6. Check the L7 load-balance statistics by entering the following command:
ACE_module5/Admin# show stats loadbalance
+------------------------------------------++------- Loadbalance statistics -----------++------------------------------------------+ Total version mismatch : 0 Total Layer4 decisions : 0 Total Layer4 rejections : 0Total Layer7 decisions : 0Total Layer7 rejections : 0
Total times rserver was unavailable : 0 Total ACL denied : 0 Total IDMap Lookup Failures : 0
Note: The ID Map is used to map real servers and server farms between the local and the remote peers in aredundant configuration. The Total IDMap Lookup Failures field increments if the local ACE fails tofind the local ACE to peer ACE ID mapping. A failure can occur if the peer ACE did not send a properremote ID for the local ACE to look up and so the local ACE could not perform a mapping or if the IDMap table was not created.
7. If you are having problems with HTTP, check the HTTP statistics and error counters by entering the followingcommand:
8. If you suspect a probe issue, for example, a TCP probe, check the probe statistics and error counters by enteringthe following command:
ACE_module5/Admin# show stats probe type tcp
+------------------------------------------++----------- Probe statistics -------------++------------------------------------------+ ----- tcp probe ---- Total probes sent : 0 Total send failures : 0 Total probes passed : 0 Total probes failed : 0 Total connect errors : 0 Total conns refused : 0 Total RST received : 0 Total open timeouts : 0 Total receive timeout : 0 Total active sockets : 0
9. Check the parameter map statistics for an HTTP parameter map by entering the following command:
ACE_module5/Admin# show parameter-map HTTP_PMAP
Number of parameter-maps : 1
Parameter-map : HTTP_PMAP Type : http server-side connection reuse : disabled case-insensitive parsing : disabled persistence-rebalance : disabled <---------------------------- Enabled by default in the ACE appliance. Enabled by default in the ACE module in software version A4(1.0) and later. header modify per-request : disabled header-maxparse-length : 4096 content-maxparse-length : 4096 parse length-exceed action : drop urlcookie-delimiters : /&#+
10. Clear the L7 load-balancing statistics by entering the following commands:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 135
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 136
This article describes the procedures for troubleshooting redundancy issues with your ACE.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE Redundancy1.1 Redundancy Protocol♦ 1.2 FT VLAN♦ 1.3 Configuration Requirements andRestrictions
♦
1.4 Example of a Redundancy Configuration♦
•
2 Troubleshooting ACE Redundancy• 3 FT Peer and Group Status Details
3.1 FT Group Status Conditions3.1.1 STANDBY_COLD◊ 3.1.2 STANDBY_CONFIG◊
♦
3.2 FT Peer Status Conditions3.2.1 PEER_DOWN◊ 3.2.2 TL_ERROR◊ 3.2.3 FT_VLAN_DOWN◊ 3.2.4 FSM_PEER_STATE_ERROR◊
♦
3.3 About WARM_COMPATIBLE andSTANDBY_WARM
♦
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 137
Overview of ACE Redundancy
Redundancy (or fault tolerance) allows your network to remain operational even if one of the ACEs becomesunresponsive. Redundancy ensures that your network services and applications are always available.
Redundancy provides seamless switchover of flows if an ACE becomes unresponsive or a critical host, interface,or HSRP group (ACE module only) fails. Redundancy supports the following network applications that requirefault tolerance:
Mission-critical enterprise applications• Banking and financial services• E-commerce• Long-lived flows such as FTP and HTTP file transfers•
Redundancy Protocol
You can configure a maximum of two ACE modules (peers) in the same Catalyst 6500 series switch or indifferent chassis for redundancy. You can also configure a maximum of two ACE 4710 appliances forredundancy. Each peer ACE can contain one or more fault-tolerant (FT) groups. Each FT group consists of twomembers: one active context and one standby context. For more information about contexts, see the CiscoApplication Control Engine Module Virtualization Configuration Guide or the Cisco 4700 Series ApplicationControl Engine Appliance Administration Guide (Software Version A3(2.4)). An FT group has a unique group IDthat you assign.
Both ACEs can be active at the same time, processing traffic for distinct virtual devices and backing up each other(stateful redundancy). An Active-Active configuration requires two FT groups and two virtual contexts on eachACE. See Figure 1.
Figure 1. Example of an Active-Active Configuration
The ACE uses the redundancy protocol to communicate between the redundant peers. The election of the activemember within each FT group is based on a priority scheme. The member configured with the higher priority iselected as the active member. If a member with a higher priority is found after the other member becomes active,the new member becomes active because it has a higher priority. This behavior is known as preemption and is
Cisco Application Control Engine (ACE) Troubleshooting Guide
One virtual MAC address (VMAC) is associated with each FT group. The format of the VMAC is:00-0b-fc-fe-1b-groupID. Because a VMAC does not change upon a switchover, the client and server ARP tablesdoes not require updating. The ACE selects a VMAC from a pool of virtual MACs available to it. You canspecify the pool of MAC addresses that the local ACE and the peer ACE use by configuring theshared-vlan-hostid command and the peer shared-vlan-hostid command, respectively. To avoid MAC addressconflicts, be sure that the two pools are different on the two ACEs. For more information about VMACs andMAC address pools, see the Cisco Application Control Engine Module Routing and Bridging ConfigurationGuide.
Each FT group acts as an independent redundancy instance. When a switchover occurs, the active member in theFT group becomes the standby member and the original standby member becomes the active member. Aswitchover can occur for the following reasons:
The active member becomes unresponsive.• A tracked host, interface, or HSRP group fails.• You enter the ft switchover command to force a switchover.•
FT VLAN
Redundancy uses a dedicated FT VLAN between redundant ACEs to transmit flow-state information and theredundancy heartbeat. You must configure this same VLAN on both peer ACEs. You also must configure adifferent IP address within the same subnet on each ACE for the FT VLAN. Cisco recommends twoport-channeled 1-Gigabit Ethernet links for the FT VLAN. For the appliance, when you configure theft-port-vlan command, the ACE modifies the associated Ethernet port or port-channel interface to a trunk port.
Note: Do not use the FT VLAN for any other network traffic, including HSRP traffic and data.The two redundant ACEs constantly communicate over the FT VLAN to determine the operating status of eachACE. The standby member uses the heartbeat packet to monitor the health of the active member. The activemember uses the heartbeat packet to monitor the health of the standby member. Communications over theswitchover link include the following data:
Redundancy protocol packets• State information replication data• Configuration synchronization information• Heartbeat packets•
For multiple contexts, the FT VLAN resides in the system configuration file. Each FT VLAN on the ACE has oneunique MAC address associated with it. The ACE uses these device MAC addresses as the source or destinationMACs for sending or receiving redundancy protocol state and configuration replication packets.
Note: The IP address and the MAC address of the FT VLAN do not change at switchover.
Configuration Requirements and Restrictions
Follow these requirements and restrictions when configuring the redundancy feature:
Cisco Application Control Engine (ACE) Troubleshooting Guide
Redundancy is not supported between an ACE module and an ACE appliance operating as peers.Redundancy must be of the same ACE device type and software release.
•
In bridged mode (Layer 2), two contexts cannot share the same VLAN.• To achieve active-active redundancy, a minimum of two contexts and two FT groups are required on eachACE.
•
When you configure redundancy, the ACE keeps all interfaces that do not have an IP address in the Downstate. The IP address and the peer IP address that you assign to a VLAN interface should be in the samesubnet but should be different IP addresses. For more information about configuring VLAN interfaces,see the Cisco Application Control Engine Module Routing and Bridging Configuration Guide or theCisco ACE 4700 Series Appliance Routing and Bridging Configuration Guide
•
FT Interfaces are put into an automatic trunk status and, for the module, the Catalyst 6500 series switchneeds to be set to trunk the specific VLAN you are using for the FT interface.
•
Example of a Redundancy Configuration
The following example shows a running-configuration file that defines fault tolerance (FT) for a single ACEoperating in a redundancy configuration. You must configure a maximum of two ACEs (peers) for redundancy tofail over from the active ACE to the standby ACE.
Note: All FT parameters are configured in the Admin context.This configuration addresses the following redundancy components:
A dedicated FT VLAN for communication between the members of an FT group. You must configurethis same VLAN on both peers.
•
An FT peer definition.• An FT group that is associated with the Admin context.• A critical tracking and failure detection process for an interface.•
access-list ACL1 line 10 extended permit ip any any
class-map type management match-any L4_REMOTE-MGT_CLASS 2 match protocol telnet any 3 match protocol ssh any 4 match protocol icmp any 5 match protocol http any 7 match protocol snmp any 8 match protocol https any
policy-map type management first-match L4_REMOTE-MGT_POLICY class L4_REMOTE-MGT_CLASS permit
interface vlan 100 ip address 192.168.83.219 255.255.255.0 peer ip address 192.168.83.230 255.255.255.0 alias 192.168.83.200 255.255.255.0 access-group input ACL1 service-policy input L4_REMOTE-MGT_POLICY no shutdown
ft interface vlan 200 ip address 192.168.1.1 255.255.255.0 peer ip address 192.168.1.2 255.255.255.0 no shutdown
Cisco Application Control Engine (ACE) Troubleshooting Guide
This section describes the methods and CLI commands that you can use to troubleshoot redundancy issues in yourACE.
1. Ensure that the software versions and licenses installed in the two ACEs are identical. A software or licensemismatch may generate the following syslog message:
%ACE-1-727006: HA: Peer is incompatible due to error str. Cannot be Redundant.
To verify the software (SRG) and license compatibility of the FT peer, enter the following command:
ACE_5/Admin# show ft peer status
Peer Id : 1State : FSM_PEER_STATE_MY_IPADDRMaintenance mode : MAINT_MODE_OFFSRG Compatibility : COMPATIBLELicense Compatibility : COMPATIBLEFT Groups : 1
If the software or license is incompatible, install the appropriate software image or license in the peer to correctthe problem.
2. Ensure that any SSL certificates (certs) and keys that exist in the active ACE are also configured in the standbyACE. SSL certs and keys are not synchronized automatically from the active to the standby. Use the cryptoexport and crypto import commands to accomplish this task. This requirement also applies to scripts andscripted probes. Failure to keep the active and standby configurations identical will cause configurationsynchronization to fail and may cause the standby ACE to enter the STANDBY_COLD state.
3. The ACE sends heartbeat packets via UDP over the FT VLAN between peers. When heartbeats are notreceived during the specified interval (the interval and count are configurable), the ACE notifies the HA processoron the CP by sending a Peer_Down interprocess communication protocol (IPCP) message. If a peer is down orunreachable, you may receive one of the following syslog messages:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 141
%ACE-1-727001: HA: Peer IP address is not reachable. Error: error str
%ACE-1-727002: HA: FT interface interface name to reach peer IP address is down. Error: error str
4. Verify connectivity between the peers over the FT VLAN. If a peer device is physically up but connectivity isthe problem, you may end up with two active devices. If connectivity is lost due to the peer going down, rebootthe peer to restore redundancy between the two devices.
5. Display heartbeat statistics, including missed heartbeats, by entering the following command:
ACE_5/Admin# show ft statsHA Heartbeat Statistics------------------------
Number of Heartbeats Sent : 0Number of Heartbeats Received : 0Number of Heartbeats Missed : 0Number of Unidirectional HB's Received : 0Number of HB Timeout Mismatches : 0Num of Peer Up Events Sent : 0Num of Peer Down Events Sent : 0Successive HB's miss Intervals counter : 0Successive Uni HB's recv counter : 0
6. Provide an alternate path for the ACE to check the peer's status in case of missed heartbeats and configure aquery interface using the followng commands:
ACE_5/Admin# configEnter configuration commands, one per line. End with CNTL/Z.ACE_5/Admin(config)# ft peer 1ACE_5/Admin(config-ft-peer)# query-interface vlan 100
If the query interface is configured, upon receiving a PEER_DOWN message from the heartbeat process, theACE data plane attempts to ping the peer using the Query VLAN. If the ping fails, the standby transitions to theACTIVE state. If the ping is successful, the standby transitions to the STANDBY_COLD state. To recover fromthe STANDBY_COLD state, reboot the standby.
7. Each peer uses a VMAC that is dependent on the FT group number. If you are using multiple ACE modules inthe same chassis, be careful when you configure the same FT groups in more than one module.
Display the VMAC for an FT group by entering the following command:
LastChange: 0 (Thu Jan 1 00:00:00 1970)iflookup index: 100vlan-vmac index:0Next Shared IF: 0Lock: Unlocked, seq 5Lock errors: 0Unlock errors: 0No. of times locked: 5No. of times unlocked: 5Current/last owner: 0x40a7fc
8. If the members of an FT group are unable to reach the ACTIVE or the STANDBY_HOT state, there may be acontext name mismatch for the same FT group. You may receive the following syslog message:
%ACE-1-727003: HA: Mismatch in context names detected for FT group FTgroupID. Cannot be redundant.
Be sure that the context names within the same FT group are identical on both ACEs.
9. Check the FT group configuration on both devices. Make sure that both devices are associated with the samecontext. Enter the following command:
ACE_5/Admin# show running-config ft
10. Verify the FT peer status and configuration by entering the following command:
ACE_5/Admin# show ft peer detail
Peer Id : 1State : FSM_PEER_STATE_COMPATIBLEMaintenance mode : MAINT_MODE_OFFFT Vlan : 100FT Vlan IF State : DOWNMy IP Addr : 10.1.1.1Peer IP Addr : 10.1.1.2Query Vlan : 110Query Vlan IF State : DOWNPeer Query IP Addr : 172.25.91.202Heartbeat Interval : 300Heartbeat Count : 20Tx Packets : 318573Tx Bytes : 66301061Rx Packets : 318540Rx Bytes : 66272840Rx Error Bytes : 0Tx Keepalive Packets : 318480Rx Keepalive Packets : 318480TL_CLOSE count : 0FT_VLAN_DOWN count : 0PEER_DOWN count : 0SRG Compatibility : COMPATIBLELicense Compatibility : COMPATIBLEFT Groups : 3
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 143
11. Verify the FT group status and configuration by entering the following command:
ACE_5/Admin# show ft group detail
FT Group : 1No. of Contexts : 1Configured Status : in-serviceMaintenance mode : MAINT_MODE_OFFMy State : FSM_FT_STATE_ACTIVEMy Config Priority : 110My Net Priority : 110My Preempt : EnabledPeer State : FSM_FT_STATE_STANDBYPeer Config Priority : 100Peer Net Priority : 100Peer Preempt : EnabledPeer Id : 1Last State Change time : Thu Apr 2 00:00:00 2009Running cfg sync enabled : EnabledRunning cfg sync status : Running configuration sync has completedStartup cfg sync enabled : EnabledStartup cfg sync status : Running configuration sync has completedBulk sync done for ARP: 0Bulk sync done for LB: 0Bulk sync done for ICM: 0
For information on troubleshooting the FT group status, see the "FT Group Status Conditions"
FT Peer and Group Status Details
This section describes how to diagnose unexpected status conditions for the FT group and FT peer. Thisinformation may enable you to troubleshoot an issue directly or help you to provide additional information to yourCisco support representative.
FT Group Status Conditions
This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT groupstatus.
STANDBY_COLD
An FT group status of STANDBY_COLD may appear when:
Config sync fails (including, incr-sync and bulk-sync), or• FT VLAN is down while the query interface is up•
Config Sync Failure
In configuration synchronization fails, the peers are not correctly exchanging configuration information. Thisfailure can be identified as follows:
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 144
Output of the show ft peer detail command shows that the peer state is "Compatible".1. Entering show ft group detail shows that the FT group is in "Standby Cold" mode and entering cfg syncstatus shows the reason for the failure. For incr-sync failure, the output shows exactly which commandresulted in an execution error on the standby. For a bulk-sync failure, the reason is "Error on Standbydevice when applying configuration file replicated from active".
2.
To further investigate a bulk-sync failure, perform these steps on the standby device:3.
For software version A2(2.0) and earlier and version A2(1.3) and earlier releases, from theAdmin context, enter show ft history cfg_cntlr and grep for "error:" to find any CLI commandsthat caused execution errors.
◊
For later releases, enter show ft config-error ctx_name to view the failed CLI commands.◊
To work around a bulk sync failure, perform these steps to remove the CLI commands that triggered the error (asidentified from the preceding analysis) and then retrigger the bulk sync operation, as follows:
Retrigger bulk sync by disabling config sync with the no ft auto-sync running command.1. Re-enable config sync with ft auto-sync running.2.
If the problem persists, repeat the above sequence until you eliminate the CLI command that triggered theproblem.
FT VLAN Down with Query Interface Up
This condition can be identified by:
Entering show ft peer detail, which shows a peer state of FT_VLAN_DOWN.1. Entering show ft stats, which shows that heartbeats are being missed.2.
In this case, check the physical connectivity of the device. It might be a physical port or cable issue.
STANDBY_CONFIG
If a device appears to be stuck in the STANDBY_CONFIG state:
Run show ft history cfg_cntlr to determine whether the peer devices successfully exchangednotifications regarding configuration synchronization.
1.
Grep for the keywords MTS_OPC_REQ_CFG_DNLD_STATUS andMTS_OPC_CFG_DNLD_STATUS.
2.
If one or both of the messages are missing, an error occurred in the synchronization exchange process.
Note that once it is stuck in the STANDBY_CONFIG state, configuration mode will be disabled on both theactive and standby devices. It can be stuck in this state for up to 4 hours, after which a timeout period expires.
FT Peer Status Conditions
This section describes how to diagnose and troubleshoot unexpected status conditions applicable to the FT peer.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 145
PEER_DOWN
If the peer status shows PEER_DOWN:
Check whether IP addresses for the local and peer device are configured correctly on both.1. Verify that pinging or telneting to the peer IP address works. If ping fails, check whether the interface isup (show interface). If so, the interface VLAN is probably not allocated to the ACE module on thesupervisor (which suggests a configuration issue on the supervisor).
2.
Enter show arp to see if the FT peer IP address is resolved. (If ARP is not resolved and ping/telnet alsofailed, it might be an encapsulation issue requiring support).
3.
Enter show conn on both sides to see if HA connections have been set up. If connections have not beenset up, check the HA DP manager log (show ft history ha_dp_mgr). Setup may have failed for variousreasons. If this is the case, contact Cisco technical support.
4.
Enter show ft stats on both devices to see if heartbeats are being sent or received. If the Number ofHeartbeats Missed counter is incrementing, the heartbeat packets could be getting dropped. Enter shownp 1 me-stats -sfp to see if heartbeat packets are being received and forwarded to X-Scale, as indicatedby the counter Packets forward to XScale. If this counter is not incrementing, provide the information toCisco technical support.
5.
TL_ERROR
This state may occur when the telnet connection used to exchange configuration information between the peerscannot be established but heartbeat packets are exchanged successfully. To identify this issue:
Verify that heartbeats are flowing by checking the statistics, show ft stats.1. Attempt to connect by telnet or to ping the FT peer. The telnet connection attempt will likely fail.2. Run show arp to see if the FT peer IP address can be resolved.3.
If show arp indicates that the address is not resolvable and the ping or telnet connect attempts fail, it is likely anencapsulation issue on the ACE.
FT_VLAN_DOWN
This state typically occurs when the FT VLAN goes down while the query interface is up. If the heartbeatexchange fails and the query interface is determined to be up based on an ICMP message check, the status isFT_VLAN_DOWN.
To verify, attempt to connect to the FT VLAN Peer IP address by ping or telnet.
If running show ft stats indicates that heartbeats are being missed, it is likely a physical connectivity issue, suchas the physical port or cable failure.
FSM_PEER_STATE_ERROR
This indicates a Software Relationship Graph (SRG) version inconsistency between the peers. See the relationshipgraph table in the following section.
Cisco Application Control Engine (ACE) Troubleshooting Guide
While peers in a redundant configuration are designed to operate with identical versions of the software, whenyou are upgrading or downgrading the software in the ACEs, it is possible for the peers to temporarily employdifferent software versions. The WARM_COMPATIBLE and STANDBY_WARM redundancy states helpminimize the operational impact of CLI compatibility issues between the peers, and allow failovers to occur on abest-effort basis during such transitions.
When you upgrade or downgrade the ACE software in a redundant configuration with different software versions,the STANDBY_WARM and WARM_COMPATIBLE states allow the configuration and state synchronizationprocess between the peers to continue on a best-effort basis. This basis allows the active ACE to synchronizeconfiguration and state information with the standby even though the standby may not recognize or understandthe CLI commands or state information.
In the STANDBY_WARM state, as with the STANDBY_HOT state, configuration mode is disabled on thestandby ACE and configuration and state synchronization continues. A failover from the active to the standbybased on priorities and preempt can still occur while the standby is in the STANDBY_WARM state. However,while stateful failover is possible for a WARM standby, it is not guaranteed. In general, modules should beallowed to remain in this state only for a short period of time.
When redundancy peers run different software versions, the SRG compatibility field shown by the show ft peerstatus command output displays WARM_COMPATIBLE instead of COMPATIBLE. When the peer is in theWARM_COMPATIBLE state, the FT groups in the standby transition to the STANDBY_WARM state instead ofthe STANDBY_HOT state.
The following software version combinations tables indicate whether the SRG compatibility field will displayWARM_COMPATIBLE (WC) or COMPATIBLE (C):
ACE Module: C = COMPATIBLE / WC = WARM_COMPATIBLEActive(Column)/Standby(Row) A2(1.5) A2(1.6) A2(2.0) A2(2.1) A2(2.2) A2(3.0) A2(2.3)A2(1.5) C WC C C WC WC WCA2(1.6) WC C C WC WC WC WCA2(2.0) C C C C C C CA2(2.1) C WC C C WC WC WCA2(2.2) WC WC C WC C WC WCA2(3.0) WC WC C WC WC C WCA2(2.3) WC WC C WC WC WC CACE Appliance: C = COMPATIBLE / WC = WARM_COMPATIBLEActive(Column)/Standby(Row) A3(1.0) A3(2.0) A3(2.1) A3(2.2) A3(2.3) A3(2.4)A3(1.0) C C C C WC WCA3(2.0) C C C C WC WCA3(2.1) C C C C WC WCA3(2.2) C C C C WC WCA3(2.3) WC WC WC WC C WCA3(2.4) WC WC WC WC WC C
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 147
This article describes the process and CLI commands for troubleshooting SSL in the ACE.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE SSL Troubleshooting1.1 Example of an SSLTermination Configuration
♦
1.2 Example of an SSL InitiationConfiguration
♦
•
2 Troubleshooting ACE SSL•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 148
Overview of ACE SSL Troubleshooting
Secure Sockets Layer (SSL) runs over TCP. After the TCP three-way handshake completes and the ACE hasproxied the connection, the SSL handshake takes place. For information about proxied connections, see theTroubleshooting Connectivity article. See Figure 1 for an illustration of the SSL handshake.
Figure 1. SSL Handshake
The ACE supports the following SSL configurations (see Figure 2):
SSL termination (ACE acts as an SSL server)• SSL initiation (ACE acts as a client)• End-to-end SSL (SSL termination plus SSL initiation)•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 149
Figure 2. SSL Configurations
Before you begin to troubleshoot potential SSL issues, be sure that the following conditions exist:
You have configured basic SLB and SSL on your ACE. For details about configuring SLB, see the CiscoApplication Control Engine Module Server Load-Balancing Configuration Guide or the Cisco ACE 4700Series Appliance Server Load-Balancing Configuration Guide. For details about configuring SSL, see theCisco Application Control Engine Module SSL Configuration Guide or the Cisco ACE 4700 SeriesAppliance SSL Configuration Guide.
•
If you are running multiple ACEs in a redundant configuration, be sure that you have copied the SSLcertificates (certs) and keys to the standby ACE. Certs and keys are not replicated in a redundantconfiguration from the active ACE to the standby ACE. Also, ensure that the configurations on the activeand the standby are identical, including the same licenses and software versions.
•
Be sure that the certs and keys are no larger than 4096 bits and that they are of an RSA type supported bythe ACE. For details about configuring SSL, see the Cisco Application Control Engine Module SSLConfiguration Guide or the Cisco ACE 4700 Series Appliance SSL Configuration Guide. The ACEsupports the following RSA key pair sizes:
•
512 (least security)◊
768 (normal security)◊
1024 (high security, level 1)◊
1536 (high security, level 2)◊
2048 (high security, level 3)◊
4096 (high security, level 4) - For software release A2(2.4) and later in the ACE module andsoftware release A3(2.6) and later in the ACE appliance, you can use 4096-bit SSL certificates inchaingroups and authgroups. You can also import public certificates and keys that are 4096 bits in
◊
Cisco Application Control Engine (ACE) Troubleshooting Guide
Server certs are valid, installed, and have not expired•
Example of an SSL Termination Configuration
The following example shows a running-configuration file of the ACE acting as an SSL proxy server; terminatingSSL or TLS connections from a client and then establishing a TCP connection to an HTTP server. When the ACEterminates the SSL or TLS connection, it decrypts the cipher text from the client and transmits the data as cleartext to the HTTP server.
access-list ACL1 line 10 extended permit ip any any
probe http GEN-HTTP port 80 interval 50 faildetect 5 expect status 200 200
rserver SERVER1 ip address 10.1.0.11 inservicerserver SERVER2 ip address 10.1.0.12 inservicerserver SERVER3 ip address 10.1.0.13 inservicerserver SERVER4 ip address 10.1.0.14 inservicerserver SERVER5 ip address 10.1.0.15 inservicerserver SERVER6 ip address 10.1.0.16 inservicerserver SERVER7 ip address 10.1.0.17 inservicerserver SERVER8 ip address 10.1.0.18 inservice
policy-map multi-match L4_SSL-VIP_POLICY class L4_SSL-TERM_CLASS loadbalance vip inservice loadbalance policy L7_SSL-TERM_POLICY loadbalance vip icmp-reply ssl-proxy server SSL_PSERVICE_SERVER connection advanced-options TCP_PARAM
interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 152
fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.80 netmask 255.255.255.0 pat service-policy input L4_SSL-VIP_POLICY no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Example of an SSL Initiation Configuration
The following example shows a running-configuration file of the ACE acting as an SSL proxy client, initiatingand maintaining an SSL connection between itself and an SSL server. The ACE receives clear text from an HTTPclient, and then encrypts and transmits the data as cipher text to the SSL server. On the reverse side, the ACEdecrypts the cipher text that it receives from the SSL server and sends the data to the client as clear text.
access-list ACL1 line 10 extended permit ip any any
probe http GEN-HTTP port 80 interval 50 faildetect 5 expect status 200 200
rserver SERVER1 ip address 10.1.0.11 inservicerserver SERVER2 ip address 10.1.0.12 inservicerserver SERVER3 ip address 10.1.0.13 inservicerserver SERVER4 ip address 10.1.0.14 inservicerserver SERVER5 ip address 10.1.0.15 inservicerserver SERVER6 ip address 10.1.0.16 inservicerserver SERVER7 ip address 10.1.0.17 inservicerserver SERVER8 ip address 10.1.0.18 inservice
interface vlan 120 description Upstream VLAN_120 - Clients and VIPs ip address 192.168.120.1 255.255.255.0 fragment chain 20 fragment min-mtu 68 access-group input ACL1 nat-pool 1 192.168.120.70 192.168.120.80 netmask 255.255.255.0 pat service-policy input L4_SSL-VIP_POLICY no shutdown
ip route 10.1.0.0 255.255.255.0 192.168.120.254
Troubleshooting ACE SSL
To troubleshoot SSL issues, follow these steps:
1. For the ACE module, check the health of the Nitrox-II (crypto module) and ensure that it has not becomeunresponsive. Stop all traffic, and then enter the following command:
ACE_module5/Admin# show crypto hardware
Figure 3. Example of the Show Crypto Hardware Command Output for an Unresponsive Crypto Module
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 155
STX1 is a count of the number of packets transmitted by the Nitrox-II and IMX1 is the number of packetsreceived by the Nitrox-II. On a normal system, these values should be the same once traffic has stopped. If thevalues are not the same, the Nitrox-II has become unresponsive.
The Nitrox-II uses 0x500 TX buffers to transmit packets and 0x200 RX buffers to receive packets. If the [TR]XBuffers used count ever exceeds the amount available, the Nitrox-II has become unresponsive.
The available cores field shows which of the 22 cores of the Nitrox-II are active. When no traffic is flowing, thereshould be no numbers following the Using: statement. If there are, as in the sample output above, then that core (0in this case) is hung, and the Nitrox-II has become unresponsive.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 156
For the POM count, there are two numbers, A(B). The "A" value is the number of outstanding packets to thePacket Order Manager, while the "B" value, counts the number of packets that have been processed in the lastsecond. When no traffic is flowing, both of these values should be 0. If no traffic is flowing, and the value of "A"is nonzero as shown above, then there are outstanding requests to the POM that are not being processed, becausethe Nitrox-II has become unresponsive.
2. Ensure that appropriate ports are designated for PAT in an SSL termination configuration. By default,connections to the real server from the ACE will inherit the destination port from the client to VIP connection sothat a connection to port 443 on the VIP will go to port 443 on the real server, unless otherwise specified in theserver farm configuration. This will cause problems if you are using ACE to offload SSL between the client andthe VIP and send clear-text traffic to the real servers. The following example demonstrates a port definition in aserver farm configuration:
Handshake Timeouts: 0total transactions: 0SSLv3 active connections: 0SSLv3 connections in handshake phase: 0SSLv3 conns in renegotiation phase: 0SSLv3 connections in data phase: 0TLSv1 active connections: 0 TLSv1 connections in handshake phase: 0TLSv1 conns in renegotiation phase: 0TLSv1 connections in data phase: 0
7. Display SSL server statistics by entering the following command:
ACE_module5/Admin# show stats crypto server
+----------------------------------------------++---- Crypto server termination statistics ----++----------------------------------------------+SSLv3 negotiated protocol: 0TLSv1 negotiated protocol: 0SSLv3 full handshakes: 0SSLv3 resumed handshakes: 0SSLv3 rehandshakes: 0TLSv1 full handshakes: 0TLSv1 resumed handshakes: 0TLSv1 rehandshakes: 0SSLv3 handshake failures: 0SSLv3 failures during data phase: 0TLSv1 handshake failures: 0TLSv1 failures during data phase: 0
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 159
Handshake Timeouts: 0total transactions: 0SSLv3 active connections: 0SSLv3 connections in handshake phase: 0SSLv3 conns in renegotiation phase: 0SSLv3 connections in data phase: 0TLSv1 active connections: 0TLSv1 connections in handshake phase: 0TLSv1 conns in renegotiation phase: 0TLSv1 connections in data phase: 0
9. Display session cache statistics for the current context by entering the following command:
switch/Admin# show crypto sessionSSL Session Cache Stats for Context------------------Number of Client Sessions: 0Number of Server Sessions: 0
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 162
This article describes how to troubleshoot performance issues with your ACE.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of TroubleshootingPerformance Issues
•
2 Troubleshooting Performance Issues•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 163
Overview of Troubleshooting Performance Issues
Before you begin to troubleshoot ACE performance issues, check and record the following items:
1. Be sure that the correct licenses are installed in your ACE.
2. Record the number of flows that you are sending to the ACE.
3. Record the performance of a single flow.
4. Identify the type of traffic: unidirectional (UDP, management) or bidirectional (TCP, HTTP, SSL, and so on)
5. Identify the ACE context that is receiving the traffic.
6. Enter the following Exec mode commands and save the output to a file:
clear stats all• show clock• show tech-support• show clock•
7. Be familiar with your application setup.
Troubleshooting Performance Issues
To troubleshoot performance issues with your ACE, follow these steps:
1. Display the resources allocated to each resource class in the ACE by entering the following command:
ACE_module5/Admin# show resource allocation---------------------------------------------------------------------------Parameter Min Max Class---------------------------------------------------------------------------
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 165
syslog rate 0 0 0 100000 0
Note: All bandwidth values are in units of bytes per second. To convert to bits per second (bps), multiply thedisplayed bandwidth value by eight. The ACE reserves 1 Gbps of bandwidth for management(to-the-ACE) traffic.
3. From the supervisor CLI, check the connectivity to the back plane by entering the following command:
cat6k# show fabric status slot channel speed module fabric status status 2 0 8G OK OK 3 0 8G OK OK 4 0 8G OK OK
5 0 8G OK OK <-------Shows 8 Gbps connectivity to the chassis back plane 6 0 20G OK OK 8 0 8G OK OK
4. Check the fabric utilization by entering the following command:
5. Display the load of the network processors (NPs) in terms of packets and connection processing for eachmicroengine (ME) by entering the following command:
Note: All show np commands must be entered for both NP1 and NP2 to obtain the total combined results.NPs operate safely at any percentage of utilization. As ME functions within the NPs approach 100percent, the traffic load is stressing the system close to its architectural limits. Any ME function thatreaches 100 percent utilization can cause back pressure and lead to dropped packets or droppedconnections.
6. Monitor the CDE queues and ensure that the Fifo Full drop count counter is not incrementing by entering thefollowing command:
ACE_module5/Admin# show cde health | include FifoFifo Full drop count 0
Backpressure is the mechanism that the ACE uses to slow the system down if queues start to fill up internally.Queues that can be affected and create backpressure are as follows:
FIFOs for the CDE, NPs, and the Crypto Module• Internal queues for each ME•
It is possible that some packets that are received by the ACE could be dropped internally if backpressure isapplied.
7. Monitor the Fastpath micro engine queues and ensure that the FastQ Transmit Backpressure, the SlowQTransmit Backpressure, the Drop: Transmit Backpressure, and the Drop: Next-Hop queue full counters are notincrementing by entering the following command:
ACE_module5/Admin# show np 1 me-stats "-s fp" | include BackpressureFastQ Transmit Backpressure: 0SlowQ Transmit Backpressure: 0Drop: Transmit Backpressure: 0
ACE/Admin# show np 1 me-stats "-s fp" | include queueDrop: Next-Hop queue full: 0
8. Monitor the TCP micro engine queues and ensure the Drops due to FastTX queue full, Drops due to Fastpathqueue full, Drops due to HTTP queue full, Drops due to SSL queue full, Drops due to AI queue full, and Dropsdue to Fixup queue full are not incrementing by entering the following command. If TCP receives backpressure, itcan drop packets, fail to ACK packets, and fail to properly track the next packet in the TCP connection.
ACE/Admin# show np 1 me-stats "-s tcp" | include queueDrop reproxy msg queue full: 0Drops due to FastTX queue full: 0Drops due to Fastpath queue full: 0Drops due to HTTP queue full: 0Drops due to SSL queue full: 0Drops due to AI queue full: 0Drops due to Fixup queue full: 0
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 167
The control plane (CP) processor processes all CP traffic (ARP, HSRP, ICMP to VIPs, routing, syslogs, SNMP,probes, and so on) and handles configuration management to parse the CLI for syntactical errors and enforceconfiguration dependencies and requirements before pushing the configuration to the data plane.
9. Display a three-way moving average of the CP processor utilization (updated every five seconds) by enteringthe following command:
ACE_module5/Admin# show processes cpu | inc utilCPU utilization for five seconds: 81%; one minute: 15%; five minutes: 10%
The ACE allocates data-plane memory to guarantee concurrent connection support for basic Layer 4 connections(such as TCP, UDP, IPsec), Layer 7 connections (proxied flows, typically for application aware load balancing orinspection, and SSL connection when using SSL acceleration). The ACE can support the maximum bidirectionalconcurrent connection limit regardless of the features enabled.
Table 1. Concurrent Connection Support
Connection Type ACE Module LimitLayer 4 4,000,000Layer 7 512,000
The state for both directions (client-to-VIP/ACE and server-to-ACE) of a TCP connection is maintained withdistinct connection objects.
10. Display the connection table by entering the following command:
ACE_module5/Admin# show conn
total current connections : 6
conn-id np dir proto vlan source destination state----------+--+---+-----+----+---------------------+---------------------+------+1 1 in TCP 130 161.44.67.242:2856 10.86.215.134:23 ESTAB2 1 out TCP 130 10.86.215.134:23 161.44.67.242:2856 ESTAB4 1 in TCP 130 161.44.67.242:2837 10.86.215.134:23 ESTAB3 1 out TCP 130 10.86.215.134:23 161.44.67.242:2837 ESTAB4 2 in TCP 130 161.44.67.242:2857 10.86.215.134:23 ESTAB3 2 out TCP 130 10.86.215.134:23 161.44.67.242:2857 ESTAB
Note: You can add the detail command option to provide the following additional fields: connection idle time,elapsed time of the connection, byte count, and packet count for each connection object.
The total current connections counter is also maintained in the output of the following command:
switch/Admin# show stats connection
+------------------------------------------++------- Connection statistics ------------++------------------------------------------+ Total Connections Created : 124 Total Connections Current : 6
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 168
Total Connections Destroyed: 62 Total Connections Timed-out: 58 Total Connections Failed : 0
Note: The Total Connections Current counter counts the number of used connection objects, not the number ofTCP flows. The number of TCP flows can be roughly determined as half the number of connectionobjects minus any UDP connections. The Total Connections Current counter is always up to date andthe maximum value can be 8,000,000.
Because of the Cisco ACE Module?s architecture, with distinct paths for new and established connections, thenumber of existing concurrent connections does not heavily impact the rate at which new connections can be setup. Nevertheless, a very large number of concurrent connections will eventually affect the performance of thesystem in setting up new connections.
11. Use the command "tcp wan-optimization rtt 0" for slow connections.
The ACE module architecture includes a mechanism where connections can be moved to the fastpath in order toincrease performance for a given connection. The LB decision is made in the software (proxy) and then moved tothe fastpath (unproxy). In a persistence rebalance scenario, the proxy/unproxy can occur Many times on a givenconnection. It is possible that if a packet enters the system during the transition Between the proxy and unproxystates, a packet may not be forwarded as expected and a retransmission may be relied upon. This can affectperformance. As a workaround, it is possible to configure the ACE such that fastpath forwarding is prohibitedThis can be accomplished by configuring a parameter map with the following:
"tcp wan-optimization rtt 0"
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 169
This article describes the ACE system limits and performance numbers for various resources and configurationobjects.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 ACE Performance Numbers and ResourceLimits
1.1 ACE Appliance Data Sheet♦ 1.2 ACE Module Data Sheets♦ 1.3 SLB-Related Limits♦ 1.4 Security-Related Limits♦ 1.5 Management-Related Limits♦
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 170
ACE Performance Numbers and Resource Limits
For the most current performance numbers for the ACE products, always refer to the data sheets for the ACEappliance and the ACE module.
ACE Appliance Data Sheet
ACE appliance data sheet
ACE Module Data Sheets
ACE10/ACE20 module data sheet
ACE30 module data sheet
If you have any questions or concerns related to ACE performance, please contact your Cisco account team forguidance.
SLB-Related Limits
Scalability Numbers The scalability numbers provided here are intended to provide guidelines related toconfiguration scalability. The scalability numbers, however, are based on basic configurations. In order to obtainscalability numbers specific to your deployment, testing with your feature combination is strongly recommended.If there are any questions or concerns related to ACE performance, please contact your Cisco account team forguidance.
SLB-Related Object ACE ModuleSystem Limit
ACE ModuleContext Limit
ACE ApplianceLimit
AdditionalInformation
ARP Entries 32,768 32,768 32,768
Bridge Table Entries 32,768 32,768 32,768A few are reserved forL2 interafces,redundancy, and so on.
Scalability Numbers The scalability numbers provided here are meant to provide guidelines related toconfiguration scalability. The scalability numbers, however, are based on basic configurations. In order to obtainscalability numbers specific to a particular customer, testing with that customer?s feature combination is stronglyrecommended before any commitment on ACE performance is made to the customer. If there are any questions orconcerns related to ACE performance, please contact your Cisco account team for guidance.
This number is strictlyenforced in A220,A214, and A322
Management-Related Limits
Scalability Numbers The scalability numbers provided here are meant to provide guidelines related toconfiguration scalability. The scalability numbers, however, are based on basic configurations. In order to obtainscalability numbers specific to a particular customer, testing with that customer?s feature combination is stronglyrecommended before any commitment on ACE performance is made to the customer. If there are any questions orconcerns related to ACE performance, please contact your Cisco account team for guidance.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 173
Management-RelatedObject
ACE ModuleSystem Limit
ACE ModuleContext Limit ACE Appliance Additional Information
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 174
This article describes how to manage and control the ACE system resources.
Guide ContentsMain ArticleOverview of ACE TroubleshootingUnderstanding the ACE Module Architecture and Traffic FlowPreliminary ACE TroubleshootingTroubleshooting ACE Boot IssuesTroubleshooting with ACE LoggingTroubleshooting ConnectivityTroubleshooting ACE Appliance Ethernet PortsTroubleshooting Remote AccessTroubleshooting Access Control ListsTroubleshooting Network Address TranslationTroubleshooting ACE Health MonitoringTroubleshooting Layer 4 Load BalancingTroubleshooting Layer 7 Load BalancingTroubleshooting RedundancyTroubleshooting SSLTroubleshooting CompressionTroubleshooting Performance IssuesACE Resource LimitsManaging ACE ResourcesShow Counter Reference
Contents
1 Overview of ACE Resources• 2 Managing ACE Resources
2.1 ACE Resource Planning♦ 2.2 Creating a Resource Class for ResourceManagement
♦
2.3 Allocating Resources Within aResource Class
♦
2.4 Changing the Resource Allocation of aResource Class
♦
2.5 Displaying the ACE ResourceAllocation and Usage
♦
•
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 175
Overview of ACE Resources
Resource classes allow you to manage context access to ACE resources, such as concurrent connections orbandwidth rate. The ACE is preconfigured with a default resource class that it applies to the Admin context andany user context upon creation. The default resource class is configured to allow a context to operate within arange that can vary from no resource access (0 percent) to complete resource access (100 percent).
When you use the default resource class with multiple contexts, you run the risk of oversubscribing ACEresources because the ACE permits all contexts to have full access to all of the resources on a first-come,first-served basis. When a resource is utilized to its maximum limit, the ACE denies additional requests made byany context for that resource.
To avoid oversubscribing resources and to help guarantee access to a resource by any context, the ACE allowsyou to create customized resource classes that you associate with one or more contexts. A context becomes amember of the resource class when you make the association. Creating a resource class allows you to set limits onthe minimum and maximum amounts of each ACE resource that a member context is entitled to use. You definethe minimum and maximum values as percentages of all resources. For example, you can create a resource classthat allows its member contexts access to no less that 25 percent of the total number of SSL connections that theACE supports.
You can limit and manage the allocation of the following ACE resources:
ACL memory• Buffers for syslog messages and TCP out-of-order (OOO) segments• Concurrent connections (through-the-ACE traffic)• Management connections (to-the-ACE traffic)• Proxy connections• Set resource limit as a rate (number per second)• Regular expression (regexp) memory• SSL connections• Sticky entries• Static or dynamic network address translations (Xlates)•
By default, when you create a context, the ACE associates the context with the default resource class. The defaultresource class provides resources of a minimum of 0 and a maximum of unlimited for all resources except stickyentries. For stickiness to work properly, you must explicitly configure a minimum resource limit for sticky entriesby using the limit-resource command.
For more information about managing ACE resources, see the Cisco Application Control Engine ModuleVirtualization Configuration Guide (Software Version A2(1.0)).
Cisco Application Control Engine (ACE) Troubleshooting Guide
You can allocate system resources to multiple contexts by creating and defining one or more resource classes andthen associating the contexts with a resource class. This section contains the following topics:
ACE Resource Planning• Creating a Resource Class for Resource Management• Allocating Resources within a Resource Class• Changing the Resource Allocation of a Resource Class•
ACE Resource Planning
When you plan the initial resource allocations for the virtual contexts in your configuration, allocate only theminimum required or estimated resources. The ACE protects resources that are in use, so to decrease a context'sresources, those resources must be unused. Although it is possible to decrease the resource allocations in realtime, it may require additional management overhead to clear any used resources before reducing them.Therefore, it is considered a best practice to initially keep as many resources in reserve as possible and allocatethe unused reserved resources as needed.
To address scaling and capacity planning, we recommend that new installations do not exceed 60 to 80 percent ofthe ACE's total capacity. To accomplish this goal, create a reserved resource class with a guarantee of 20 to 40percent of all the ACE resources and configure a virtual context dedicated solely to ensuring that these resourcesare reserved. Then, you can efficiently distribute such reserved resources to contexts as capacity demands forhandling client traffic increase over time.
Creating a Resource Class for Resource Management
You can create a resource class to allocate and manage system resources by one or more contexts. The ACEsupports a maximum of 100 resource classes. After you create and configure the resource class, use the membercommand in context configuration mode to assign a resource class to the context (see the "Associating a Contextwith a Resource Class" section). To create a resource class, use the resource-class command in configurationmode. The syntax of the command is as follows:
resource-class name
For the name argument, enter an unquoted text string with no spaces and a maximum of 64 alphanumericcharacters.
For example, to create the RC1 resource class, enter the following command:
To remove the resource class from the configuration, enter the following command:
host1/Admin(config)# no resource-class RC1
When you remove a resource class from the ACE, any contexts that were members of that resource class
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 177
automatically become members of the default resource class. The default resource class allocates a minimum of0.00 percent to a maximum of 100.00 percent of all ACE resources to each context. You cannot modify thedefault resource class.
Allocating Resources Within a Resource Class
You can allocate all resources or individual resources to all member contexts of a resource class. For example,you can allocate only concurrent connections or sticky table memory or management traffic. To allocate systemresources to all members (contexts) of a resource class, use the limit-resource command in resource-classconfiguration mode.
Note: The limit that you set for individual resources when you use the limit-resource command overrides thelimit that you set for all resources when you use the limit-resource all command.
If you lower the limits for one context (context A) in order to increase the limits of another context (context B),you may experience a delay in the configuration change because the ACE will not lower the limits of context Auntil the resources are no longer being used by the context.
For example, to allocate 20 percent of all resources (minimum and maximum) to all member contexts of theresource class, enter the following command:
(config-resource)# limit-resource all minimum 20% maximum equal-to-min
To restore resource allocation to the default values of 0 percent minimum and 100 percent maximum for allresources to all member contexts, enter the following command:
(config-resource)# no limit-resource all
Table 1 lists the managed system resources of the ACE. You can limit these resources per context or for allcontexts associated with the resource class by using the limit-resource command. See the "Allocating Resourceswithin a Resource Class" section.
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 178
Rate---Bandwidth 4 gigabits per second (Gbps)
You can upgrade the ACE maximum bandwidth to 8 Gbps or 16 Gbps bypurchasing a separate license from Cisco. For more information, see theCisco Application Control Engine Module Administration Guide (SoftwareVersion A2(1.0)).
---Connections (any kind) 325,000 connections per second (CPS)---MAC miss 2000 packets per second (PPS)---Management traffic 1 Gbps---SSL transactions 1000 transactions per second (TPS), upgradeable to 15000 TPS with a
separate license. For more information, see the Cisco Application ControlEngine Module Administration Guide (Software Version A2(1.0)).
---Syslog For traffic going to the ACE (control plane), 5000 messages per second
For traffic going through the ACE (data plane), 350,000 messages per secondRegular Expression Memory 1,048,576 bytesSticky Entries 4,194,304 entriesXlates (network and port addresstranslation entries)
524,286 translations
Cisco Application Control Engine (ACE) Troubleshooting Guide
Changing the Resource Allocation of a Resource Class
If you (as the global Admin) need to change the resource allocation in a resource class of which two or more usercontexts are members, you may do so at any time by entering the appropriate CLI commands. (For details aboutallocating resources, see the "Allocating Resources Within a Resource Class" section.) However, the shift inresources between the contexts does not take place immediately unless the appropriate resources are available toaccommodate the change. In most cases, to effect a change in resource allocation, you must inform the contextadministrators involved to ensure that the new resource allocation is possible.
For example, suppose that context A is using 100 percent of the available resources of the class and you want toallocate 50 percent of the resources to context A and 50 percent of the resources to context B. Although the CLIaccepts your resource allocation commands, context B cannot allocate 50 percent of the resources until context Adeallocates 50 percent of its resources.
In this case, you must perform the following:
Inform the Context A administrator to start deallocating resources•
Inform the Context B administrator to start allocating resources after the Context A administrator releasesthe resources
•
Note: As resources are released from other contexts, the ACE assigns the resources to resource-starvedcontexts (contexts where the resource-class minimum allocations have not been met).
Displaying the ACE Resource Allocation and Usage
To view the current resource allocation in your ACE, enter the following command:
ACE_mdule5/Admin# show resource allocation---------------------------------------------------------------------------Parameter Min Max Class---------------------------------------------------------------------------
acl-memory 0.00% 100.00% default
syslog buffer 0.00% 100.00% default
conc-connections 0.00% 100.00% default
mgmt-connections 0.00% 100.00% default
proxy-connections 0.00% 100.00% default
bandwidth 0.00% 100.00% default
connection rate 0.00% 100.00% default
inspect-conn rate 0.00% 100.00% default
Cisco Application Control Engine (ACE) Troubleshooting Guide
07/26/11 180
syslog rate 0.00% 100.00% default
regexp 0.00% 100.00% default
sticky 0.00% 100.00% default
xlates 0.00% 100.00% default
ssl-connections rate 0.00% 100.00% default
mgmt-traffic rate 0.00% 100.00% default
mac-miss rate 0.00% 100.00% default
throughput 0.00% 100.00% default
To view the current resource usage, enter the following command:
Note: All bandwidth values are in bytes per second. To convert to bits per second (bps), multiply the values byeight. The ACE guarantees 1 Gbps of bandwidth for management traffic. So, the total bandwidth for a4-Gbps ACE license is actually 5 Gbps. Throughput is still 4 Gbps.
To display the data plane resource allocation and usage and to cross-check the output of the above twocommands, enter the following command:
The Admin context has a context ID of 0. To display the resource allocation and and usage statistics for anothercontext, change the "0" in the "-L<context_id>" parameter to the context ID of another context.
Cisco Application Control Engine (ACE) Troubleshooting Guide