-
Troubleshooting Guidefor Cisco Unified Customer Voice
Portal7.0(2)
January 2011
Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan
Jose, CA 95134-1706USAhttp://www.cisco.comTel: 408 526-4000800
553-NETS (6387)Fax: 408 527-0833
-
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN
THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE.ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE
ACCURATE BUT ARE PRESENTEDWITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION
OFANY PRODUCTS.THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE
ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKETTHAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS
REFERENCE. IF YOU ARE UNABLE TO LOCATE THESOFTWARE LICENSE OR
LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.The
Cisco implementation of TCP header compression is an adaptation of
a program developed by the University of California, Berkeley (UCB)
aspart of UCBs public domain version of the UNIX operating system.
All rights reserved. Copyright 1981, Regents of the University of
California.NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT
FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED"AS IS" WITH ALL
FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL
WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING,WITHOUT LIMITATION,
THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT OR ARISINGFROM A COURSE OF DEALING, USAGE, OR TRADE
PRACTICE.IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY
INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES,INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING
OUT OF THE USE OR INABILITY TO USETHIS MANUAL, EVEN IF CISCO OR ITS
SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.Cisco and the Cisco Logo are trademarks of Cisco Systems,
Inc. and/or its affiliates in the U.S. and other countries. A
listing of Cisco's trademarkscan be found at
http://www.cisco.com/go/trademarks. Third party trademarks
mentioned are the property of their respective owners. The use
ofthe word partner does not imply a partnership relationship
between Cisco and any other company. (1005R)Any Internet Protocol
(IP) addresses used in this document are not intended to be actual
addresses. Any examples, command display output, andfigures
included in the document are shown for illustrative purposes only.
Any use of actual IP addresses in illustrative content is
unintentional andcoincidental.Copyright 2011 Cisco Systems, Inc.
All rights reserved.
http://www.cisco.com/go/trademarks
-
Table of Contents
Preface
...........................................................................................................................................................1Purpose
.....................................................................................................................................................1Audience
....................................................................................................................................................1Organization
..............................................................................................................................................1Related
Documentation
.............................................................................................................................2Conventions................................................................................................................................................3Obtaining
Documentation and Submitting a Service
Request...................................................................4Documentation
Feedback...........................................................................................................................4
1. Troubleshooting
Overview...........................................................................................................................5Problem
Solving
Process...........................................................................................................................5
Guidelines to Assist in Isolating
Problems.............................................................................................5Problem
Solving
Tasks...............................................................................................................................6Troubleshooting
Checklist.........................................................................................................................11
2. Symptoms and
Solutions..........................................................................................................................13Overview...................................................................................................................................................13Call
Server................................................................................................................................................13VXML
Server............................................................................................................................................38Reporting
Server......................................................................................................................................54Audio........................................................................................................................................................60Transfer
and
Connect...............................................................................................................................77Unified
CM................................................................................................................................................91Unified
ICME............................................................................................................................................91Content
Services
Switch..........................................................................................................................98General
System
Issues............................................................................................................................98H.323
Troubleshooting............................................................................................................................105SIP
Troubleshooting................................................................................................................................109SNMP
Troubleshooting...........................................................................................................................117Installation
Troubleshooting....................................................................................................................118Gateway
Troubleshooting.......................................................................................................................124Cisco
Unified Presence
Server..............................................................................................................128Operations
Console
(OAMP)..................................................................................................................129Cisco
Security Agent
(CSA)...................................................................................................................130Support
Tools..........................................................................................................................................131Backup
and
Restore...............................................................................................................................131
Appendix A. Error
Messages......................................................................................................................135Infrastructure
Error
Messages................................................................................................................135IVR
Error
Messages...............................................................................................................................141VMS
Error
Messages.............................................................................................................................145Reporting
Server Error
Messages..........................................................................................................149ICM
Error
Messages...............................................................................................................................151SIP
Error
Messages...............................................................................................................................154
Appendix B. Event and Cause
Codes.........................................................................................................157IVR
Service Error
Codes........................................................................................................................157Unified
ICME Error
Codes......................................................................................................................159SIP
Codes..............................................................................................................................................161
Appendix C. Troubleshooting Radvision
Errors...........................................................................................163
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
i
-
Appendix D. Accessing and Interpreting SNMP
Data.................................................................................165
Appendix E. Internal Interfaces
Description................................................................................................169
Appendix F. Trace Bits
Description..............................................................................................................171
Appendix G.
Debugging..............................................................................................................................175SIP
IOS
Debugging................................................................................................................................175
SIP IOS Troubleshooting
Commands................................................................................................176H.323
IOS
Debugging............................................................................................................................176
Index
...........................................................................................................................................................179
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
ii
-
Preface
Purpose
This manual describes how to troubleshoot common issues for the
Cisco Unified CustomerVoice Portal (Unified CVP) product. The
document also explains the tools and techniques thatcan assist in
the troubleshooting process.
Audience
This document is intended for Contact Center managers, Unified
CVP system managers, VoIPtechnical experts, and IVR Service
application developers.
Organization
The document is divided into the following chapters.
DescriptionChapter
Describes various methods of approaching troubleshooting
tasks.Chapter 1, TroubleshootingOverview
Describes particular issues, any associated error messages, the
cause of the issue, aswell as the solution.
Chapter 2, Symptoms andSolutions
Provides information on Unified CVP error messages.Appendix A,
Error Messages
Provides information on Unified CVP error codes.Appendix B,
Event and CauseCodes
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
1
-
DescriptionChapter
Describes how to isolate Unified CVP errors from those of
Radvision. This appendixaims at distinguishing these errors, along
with an explanation of each error code,and the appropriate action
to take before involving Radvision TAC.
Appendix C, TroubleshootingRadvision Issues
Provides information on SNMP severity levels and their
descriptions.Appendix D, Accessing andInterpreting SNMP Data
Summarizes the major interfaces between the Unified CVP product
components andother Cisco products.
Appendix E, Internal InterfacesDescription
Lists the common trace bits, their corresponding values and the
description for eachUnified CVP subsystem type. Each value is
unique within each Unified CVPsubsystem.
Appendix F, Trace BitsDescription
Describes SIP troubleshooting for some problems that may occur,
and describessome useful command line commands for H.323 IOS
troubleshooting.
Appendix G, Debugging
Related Documentation
Note: Planning your Unified CVP solution is an important part of
the process in setting upUnified CVP. Cisco recommends that you
read the Cisco Unified Customer Voice Portal Release7.x Solution
Reference Network Design (SRND) guide before configuring your
Unified CVPsolution. With Unified 7.x, the Planning Guide for Cisco
Unified Customer Voice Portal hasbeen incorporated into the SRND
guide.
Unified CVP provides the following documentation:
Cisco Security Agent Installation/Deployment for Cisco Unified
Customer Voice Portalprovides installation instructions and
information about Cisco Security Agent for the UnifiedCVP
deployment.We strongly urge you to read this document in its
entirety.
Cisco Unified Customer Voice Portal Release 7.x Solution
Reference Network Design (SRND)provides design considerations and
guidelines for deploying contact center voice responsesolutions
based on Cisco Unified Customer Voice Portal (Unified CVP) 7.x
releases.
Configuration and Administration Guide for Cisco Unified
Customer Voice Portal describeshow to set up, run, and administer
the Cisco Unified CVP product, including
associatedconfiguration.
Element Specifications for Cisco Unified CVP VXML Server and
Unified Call Studio describesthe settings, element data, exit
states, and configuration options for Elements.
Installation and Upgrade Guide for Cisco Unified Customer Voice
Portal describes how toinstall Unified CVP software, perform
initial configuration, and upgrade.
Operations Console Online Help for Cisco Unified Customer Voice
Portal describes how touse the Operations Console to configure
Unified CVP solution components.
Port Utilization Guide for Cisco Unified Customer Voice Portal
describes the ports used ina Unified CVP deployment.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
2
Preface
Related Documentation
-
Programming Guide for Cisco Unified CVP VXML Server and Unified
Call Studio describeshow to build components that run on the Cisco
Unified VXML Server.
Reporting Guide for Cisco Unified Customer Voice Portal
describes the Reporting Server,including how to configure and
manage it, and discusses the hosted database.
Say It Smart Specifications for Cisco Unified CVP VXML Server
and Unified Call Studiodescribes in detail the functionality and
configuration options for all Say It Smart pluginsincluded with the
software.
User Guide for Cisco Unified CVP VXML Server and Unified Call
Studio describes thefunctionality of Call Studio including creating
projects, using the Call Studio environment,and deploying
applications to the Unified CVP VXML Server.
For additional information about Unified ICME, see the Cisco web
site Cisco web site
(http://www.cisco.com/en/US/products/sw/custcosw/ps1001/tsd_products_support_series_home.html)listing
Unified ICME documentation.
Conventions
This manual uses the following conventions:
DescriptionConvention
Boldface font is used to indicate commands, such as user
entries,keys, buttons, and folder and submenu names. For
example:
boldface font
Choose Edit > Find.
Click Finish.
Italic font is used to indicate the following:italic font
To introduce a new term. Example: A skill group is acollection
of agents who share similar skills.
For emphasis. Example: Do not use the numerical
namingconvention.
A syntax value that the user must replace. Example:
IF(condition, true-value, false-value)
A book title. Example: See the Cisco CRS Installation Guide.
Window font, such as Courier, is used for the following:window
font
Text as it appears in code or that the window displays.Example:
Cisco Systems,Inc.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
3
Preface
Conventions
http://www.cisco.com/en/US/products/sw/custcosw/ps1001/tsd_products_support_series_home.html
-
DescriptionConvention
Angle brackets are used to indicate the following:< >
For arguments where the context does not allow italic, suchas
ASCII output.
A character string that the user enters but that does not
appearon the window such as a password.
Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, submitting a service
request, and gatheringadditional information, see the monthly
What's New in Cisco Product Documentation, whichalso lists all new
and revised Cisco technical documentation, at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Subscribe to the What's New in Cisco Product Documentation as a
Really Simple Syndication(RSS) feed and set content to be delivered
directly to your desktop using a reader application.The RSS feeds
are a free service and Cisco currently supports RSS version
2.0.
Documentation Feedback
You can provide comments about this document by sending email to
the following address:
mailto:[email protected]
We appreciate your comments.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
4
Preface
Obtaining Documentation and Submitting a Service Request
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.htmlmailto:[email protected]
-
Troubleshooting OverviewWhen troubleshooting a telephony or IP
network environment, it is important to define thespecific
symptoms, identify all potential problems that could be causing the
symptoms, andthen systematically eliminate each potential problem
from most likely to least likely until thesymptoms disappear.
This chapter contains the following topics:
Problem Solving Process, page 5 Problem Solving Tasks, page 6
Troubleshooting Checklist, page 11
Problem Solving Process
The following steps provide some guidelines to assist in
isolating a particular issue.
Guidelines to Assist in Isolating Problems
Step 1 Analyze the problem and create a clear problem statement.
Define symptoms and potentialcauses.
Step 2 Gather the facts that you need to help isolate possible
causes.
Step 3 Consider possible causes based on the facts that you
gathered.
Was anything recently added, removed, or modified?
Is it a reproducible event?
Does it occur at a particular time of day, or day of week?
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
5
Chapter 1
-
Have there been any changes made to the domain, network, or
security policies?
Step 4 Create an action plan. Begin with the most likely problem
and devise a plan in which youmanipulate only one variable at a
time.
Step 5 Implement the action plan, performing each step while
testing to see whether the symptomdisappears.
Step 6 Analyze the results to determine whether the problem has
been resolved. If it has, the processis complete.
Step 7 If the problem has not been resolved, create an action
plan based on the next most probablecause on your list, or contact
the Cisco Technical Assistance Center (TAC), or your
CiscoPartner.
Note: Only change one variable at a time. If that does not
resolve the issue, undo that changeand move on to the next step of
your plan.
Problem Solving Tasks
In addition to the process discussed above, there are other
areas to consider if a problem ariseswithin the Unified CVP
environment. The information below provides some
additionalinformation to assist in isolating or resolving a
particular issue.
Verify that the software versions comply with the Hardware and
System Software Specificationfor Unified Customer Voice Portal
document. A version incompatibility might not necessarilybe the
cause of the issue, but can be investigated and considered as a
possibility. This documentis available at:
http://www.Cisco.com/en/US/products/sw/custcosw/ps1006/prod_technical_reference_list.html
Review the Unified CVP logs for possible error or warning
events. Use the Support Toolsutility since it has the ability to
retrieve and process trace logs from most components, andto set or
reset trace levels on those components. To access the Support
Tools, open theOperations Console, and click Support Tools from the
Tools menu. The Configuration andAdministration Guide for Cisco
Unified Customer Voice Portal and Cisco Support ToolsUser Guide for
Cisco Unified Software provide additional information about Support
Tools.
Note: Users can execute some UNIX equivalent commands, such as
more, grep, diff, tail,tee on Unified CVP servers during
troubleshooting. These commands are part of the SupportTools
installation. The PATH environment variable allows these commands
to be accessiblevia command line.
Unified CVP captures two different types of logs: events and
errors. When a Unified CVPcomponent generates an event, it is
logged in the Unified CVP log. When a log statement iswritten to
the log file and the level is at ERROR or above, it is also written
to error log file.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
6
Chapter 1: Troubleshooting Overview
Problem Solving Tasks
http://www.Cisco.com/en/US/products/sw/custcosw/ps1006/prod_technical_reference_list.htmlhttp://www.Cisco.com/en/US/products/sw/custcosw/ps1006/prod_technical_reference_list.html
-
Note: This file is created on demand. If there are no ERROR
level (or above) logs, you donot see this file.
Below are the locations of the log files for Unified CVP. In the
listings below, CVP_HOMEis the location that was selected for the
installation of your Unified CVP software, such asC:\Cisco\CVP.
CVP_HOME\logs\
Call Server and/or Reporting Server logs.
CVP_HOME\logs\OAMP\
Operations Console logs.
CVP_HOME\logs\VXML\
VXML Server logs.
CVP_HOME\logs\SNMP\
SNMP Agent logs.
CVP_HOME\logs\ORM\
Unified CVP Resource Manager logs.
Note: The Operations Console Resource Manager (ORM) is a service
that is installedduring the installation of the Operations Console.
When installing the Operations Console,the two Windows services
that are installed are Unified CVP OpsConsoleServer andUnified CVP
Resource Manager. The Resource Manager resides on each Unified
CVPserver and it allows the Operations Console to communicate with
that server and performfunctions on that server, such as starting,
stopping, or monitoring.
Virtually all of the Unified CVP logs start a new log file at
12:00AM every night, with thedate as part of the filename. The VXML
Server logs do not necessarily start a new file at12:00AM. The
format of the date is yyyy-mm-dd. These logs also start a new log
file whenthey reach the predefined size limit of 10 MB and have a
number as part of the filenameextension. The number indicates which
log it was for that day. When the log directory sizereaches the
maximum number of megabytes (20,000MB) for disk storage for log
files, theold files are purged as necessary. Both the Log File Size
and Log Directory Size areconfigurable in the device Infrastructure
tab of the Operations Console.
Check that the network is functioning properly. For the initial
connectivity tests, the followingcommand line tools may be
helpful:
ping - tests whether a host is reachable across an IP network.
The ping command offersseveral parameters or flags that can be set
to produce various outputs. To view all of theavailable parameters,
type ping /? at the command prompt.
C:\> ping 10.86.132.150
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
7
Chapter 1: Troubleshooting Overview
Problem Solving Tasks
-
Pinging 10.86.132.150 with 32 bytes of data:
Reply from 10.86.132.150: bytes=32 time=5ms TTL=122
Reply from 10.86.132.150: bytes=32 time=4ms TTL=122
Reply from 10.86.132.150: bytes=32 time=4ms TTL=122
Reply from 10.86.132.150: bytes=32 time=4ms TTL=122
Ping statistics for 10.86.132.150: Packets: Sent = 4, Received =
4, Lost = 0 (0% loss),Approximate round trip times in milliseconds:
Minimum = 4ms, Maximum = 5ms, Average= 4ms
C:\>
traceroute - determines the packet route that is taken to the
target host across an IP network.The command in a Windows
environment is: tracert. The tracert command offers
severalparameters or flags that can be set to produce various
outputs. To view all of the availableparameters, type tracert /? at
the command prompt.
C:\> tracert 10.1.1.6
Tracing route to 10.1.1.6 over a maximum of 30 hops
---------------------------------------------------
1 2 ms 3 ms 2 ms 10.1.2.1
2 25 ms 83 ms 88 ms 192.168.11.1
3 25 ms 79 ms 93 ms 10.1.1.6
Trace complete.
netstat - lists the active incoming and outgoing network
connections on a system. Thenetstat command offers several
parameters or flags that can be set to produce variousoutputs. To
view all of the available parameters, type netstat /? at the
command prompt.
C:\>netstat -a -n
Active Connections
Proto Local Address Foreign Address State
TCP 0.0.0.0:42 0.0.0.0:0 LISTENING
TCP 0.0.0.0:88 0.0.0.0:0 LISTENING
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING
TCP 0.0.0.0:389 0.0.0.0:0 LISTENING
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
8
Chapter 1: Troubleshooting Overview
Problem Solving Tasks
-
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING
TCP 0.0.0.0:593 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1038 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1041 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1048 0.0.0.0:0 LISTENING
TCP 0.0.0.0:1723 0.0.0.0:0 LISTENING
TCP 0.0.0.0:3268 0.0.0.0:0 LISTENING
TCP 10.99.99.1:53 0.0.0.0:0 LISTENING
TCP 10.99.99.1:139 0.0.0.0:0 LISTENING
TCP 10.99.99.1:389 10.99.99.1:1092 ESTABLISHED
TCP 10.99.99.1:1092 10.99.99.1:389 ESTABLISHED
TCP 10.99.99.1:3604 10.99.99.1:135 TIME_WAIT
TCP 10.99.99.1:3605 10.99.99.1:1077 TIME_WAIT
UDP 0.0.0.0:135 *:*
UDP 0.0.0.0:445 *:*
UDP 0.0.0.0:1087 *:*
UDP 10.99.99.1:53 *:*
UDP 10.99.99.1:137 *:*
UDP 10.99.99.1:138 *:*
Using the Windows Services control panel, verify that all
appropriate services are runningfor that server. To view the
Service control panel, go to Start and select Run. In the
Runwindow, type services.msc and click OK.
Check the Application and System logs in the Windows Event
Viewer for warning or errorevents. To start the Event Viewer, click
Start and select Run. In the Run window, typeeventvwr.exe and click
OK. This launches the Event Viewer console, which displays thethree
logging options (Application, Security, and System).
A device may be in some state other than Up, such as Down, or
Not Reachable. Check theOperations Console to verify the status of
a particular device. To check the status:
From the Operations Console, select System > Control
Center.
Click the Device Pool tab in the left frame of the page.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
9
Chapter 1: Troubleshooting Overview
Problem Solving Tasks
-
In the right frame of the page, click the General tab if it is
not already selected.
Locate the device you are interested in and check the status of
that device in the Statuscolumn.
Check the call statistics from the Operations Console to try to
determine any abnormalities.The reported values from the various
components might provide some information that willassist you in
isolating the issue.
Note: The Operations Console online help describes the device
status fields in detail. Fromthe console, choose System >
Control Center and then select the Device Pool tab and choosea
device pool. You can also view a particular type of device by
selecting the Device Typetab and choosing a device type. The status
of some devices such as, IOS devices, UnifiedCM, ICM servers, and
SIP proxy servers display as N/A (Not Applicable) because
theOperations Console does not monitor these device types.
You can access call statistics either by:
Choosing System > Control Center, selecting a Call Server,
clicking Statistics icon inthe toolbar, and then selecting the
various tabs (Unified ICME, SIP, IVR, or Infrastructure).
Choosing Device Management > CVP Call Server, selecting a
Call Server, clicking theStatistics icon in the toolbar, and then
selecting the various tabs (Unified ICME, SIP,IVR, or
Infrastructure).
The examples above list the procedure for viewing the statistics
for a Call Server. The statisticsfor other devices, such as the
Reporting Server, VXML Server, and so forth can be viewedin the
same manner.
Below are some of the initial areas to check in the
Infrastructure tab for abnormal values orthresholds.
Check for licensing issues in the Infrastructure tab. The
Licensing Statistics column isbroken up into Realtime, Interval,
and Aggregate sections. Check the value being reportedfor Port
Licenses Available against the value of Maximum Port Licenses
Used.
Check the level of the Current Port Licenses State. This reports
on the threshold level ofport license usage. There are four levels:
safe, warning, critical and failure. An administratormay set the
required percentage of port licenses in use needed to reach a given
thresholdlevel, with the exception of the failure level which is
reached when the number of portschecked out is equal to the total
number of licenses ports.
Check the Uptime statistic, as well as the memory and thread
pool statistics for possibleissues. If the Uptime statistic is low,
then that indicates that the component was recentlyunavailable for
some period of time. High memory or thread pool statistics may
indicatea low available resource issue.
Below are some of the initial areas to check in the SIP tab for
abnormal values or thresholds.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
10
Chapter 1: Troubleshooting Overview
Problem Solving Tasks
-
In the Interval Statistics section, check the value of the
Failed SIP Transfers (Pre-Dialog).This is the total number of
failed transfers on the first Unified CVP transfer in the
laststatistics aggregation interval. A SIP dialog is established
after the first Unified CVPtransfer is completed. The metric does
not include rejections due to the SIP service beingout of service.
The metric includes failed transfers that were made after a label
was returnedfrom the Unified ICME in a CONNECT message.
Check the value of the Failed SIP Transfers (Post-Dialog). The
total number of failedtransfers on second and subsequent Unified
CVP transfers in the last statistics aggregationinterval. After SIP
dialog is established, Unified CVP performs SIP transfers
usingre-INVITE messages. The metric does not include rejections of
SIP REFER transfers. Themetric includes failed transfers that were
made after a label was returned from the UnifiedICME in a CONNECT
message.
Below are some of the initial areas to check in the IVR tab for
abnormal values or thresholds.
Check the Active Calls statistic. This is the number of active
calls being serviced by theIVR service currently. Refresh the page
to determine if the calls are increasing.
Check the Average, Maximum, and Minimum Call Latency statistics.
These report theamount of time in milliseconds it took for the IVR
Service to complete the processing ofa New Call Request or a
Request Instruction Request during the time interval. Longdurations
may indicate a network or load problem.
Also, verify that your network environment has been correctly
configured. The table belowshows the speed and duplex settings for
Unified CVP.
Server/Gateway NICSpeed/Duplex Setting
Switch Port Speed/DuplexSetting
Server/Gateway NIC SpeedCapability
Ethernet Switch SpeedCapability
Auto / AutoAuto / Auto1000 Mb1000 Mb
100 Mb / Full100 Mb / Full100 Mb1000 Mb
100 Mb / Full100 Mb / Full100 Mb100 Mb
100 Mb / Full100 Mb / Full1000 Mb100 Mb
If you have exhausted all the common causes and actions
including those outlined in thisdocument or others that you have
identified in your environment, contact the Cisco
TechnicalAssistance Center (TAC) or your Cisco Partner.
Troubleshooting Checklist
Complete this checklist to assist in isolating the issue, or to
provide information to your supportpartner or Cisco Technical
Support.
1. What is the version of Unified CVP that is currently running?
Include any Hotfix or ServiceRelease (SR) information.
2. Is this a new installation or an upgrade?
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
11
Chapter 1: Troubleshooting Overview
Troubleshooting Checklist
-
3. If this is an upgrade, what version was previously
installed?
4. When did the problem occur?
5. What are the observed symptoms, and the conditions under
which these symptoms occur?
6. Was anything changed or updated in hardware, software, or
network components prior tothe first occurrence of the observed
symptoms?
7. Describe the related call flow. Some examples include: Public
Switched Telephone Network(PSTN) originated or IP Phone originated.
If it is a transfer, is it Unified CVP dequeue oragent originated?
Is the call inbound ACD, inbound IVR Service treatment, inbound
queuetreatment, outpulse transfer, or agent transfer?
8. Is the problem reproducible?
9. What Unified CVP deployment model used for deployment?
10. What is the call transfer method used?
11. Are you able to capture a screen shot of the error or
failure? If Yes, save it to a file andattach to a case.
Network Topology
Complete this checklist to assist in isolating the issue, or to
provide information to CiscoTechnical Support.
1. Has auto-negotiate been disabled on all PCs, routers, and
switch ports?
Note: Duplex/speed mismatch between a device and its
corresponding port on the switchis the single most common problem
for network latency.
2. Is a network topology diagram available?
3. Which type of IP Gateway is being used in this Unified CVP
solution?
4. Save the output from a show tech, show stack, and a show
Gatekeeper endpoint fromthe Gateway.
5. Which type of Gatekeeper is being used in this Unified CVP
solution?
6. On which server are the recorded media files located, and
what is the path to those files?
7. Are Content Service Switches (CSS) used for failover?
8. Collect and provide versions of IOS, Digital Signal Processor
(DSP) , applications, andEngineering Special (ES)/patch levels in
the environment.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
12
Chapter 1: Troubleshooting Overview
Troubleshooting Checklist
-
Symptoms and Solutions
Overview
This chapter contains problem descriptions and steps to diagnose
and resolve each problem.Problem descriptions are grouped by
Unified CVP component.
Call Server
This section contains problems associated with the Call
Server.
Device Becomes Corrupted
Symptom:
A device has become corrupted. The user receives a specific
error message that indicates whatdevice has been corrupted.
Message:
"Device with IP Address: {1} and Hostname: {2} is in an
inconsistent state. Please Save andDeploy the properties of this
Server again to fix the problem. If problem persists, please
contactyour administrator for more details."
"Device with IP Address: {1} and Hostname: {2} is not ready at
this time. Please retry theoperation at a later time."
"Device with IP Address: {1} and Hostname: {2} is in an
inconsistent state and cannot beproperly deleted at this time.
Please retry the operation later or try deleting again for
forcefuldelete."
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
13
Chapter 2
-
Cause:
The most likely cause is that the Unified CVP software was
reinstalled.
Action:
The following steps outline the process to resolve the issue of
software being reinstalled on aserver.
1. Identify the affected devices from the Control Center of the
Operations Console. Theaffected devices are:
- devices with the same IP addresses, and
- devices associated to the devices identified with the same IP
address.
2. Go to the Edit page of each device and click
Save&Deploy.
3. Restart all affected devices from Control Center in the
Operations Console.
If the procedure above does not resolve the issue, then you must
clean up the device byperforming the following steps.
Device Clean Up
1. Delete the device from the Operations Console.
2. Reinstall the software on that device.
3. Recreate the device from the Operations Console.
Each Operations Console Server will be identified by a unique
identifier. The identifier isgenerated at the time of creating a
first device through that Operations Console Server. Oncegenerated,
the identifier will then be stored in the Operations Console Server
database and inthe Resource Manager Configuration file on every
device that includes the Operations ConsoleServer. The purpose of
this unique identifier is to prevent management of a device from
multipleOperations Consoles.
The unique identifier gets imported whenever the import
functionality of an Operations Consoledatabase is invoked.
Importing a database which was exported from a different
OperationsConsole Server is not a supported feature unless the
Operations Console is newly installed orhas never been used
before.
If a user uses an Operations Console Server that is already
managing at least one device toimport a database (from a different
Operations Console Server or an empty database which wasnever used
to create a device) the following side effects will occur:
User may not be able to manage devices which were managed by the
same Operations Consoleserver prior to the import. This is because
the unique identifiers will now be different betweenthe Operations
Console Server and the previously managed device. User may not be
able to
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
14
Chapter 2: Symptoms and Solutions
Call Server
-
create an IOS device such as Gatekeeper(s), Gateway(s), and
CSS(s) from the OperationsConsole.
An Operations Console Server that is managing at least one
device must not import an emptydatabase that was previously
exported from any Operations Console Server. This will cause
amismatch in the unique identifiers used since the Operations
Console database and the OperationsConsole Server have a unique
identifier and the empty database has no identifier. If a
customerrequires replacing the existing database with an empty
database, the recommended way is toreinstall the Operations Console
Server software. If this step is not followed, the customer canno
longer manage devices that were previously managed by the same
Operations Console Server.
SSL Recovery
Symptom:
The Operations Console is unable to communicate with the
Operations Console ResourceManager (ORM) service on a device.
Message:
Call Server device with IP Address: and Hostname: operation
failed. Devicecould not be reached because of a failure in
negotiating security certificates. Please refer todocumentation for
configuring security before trying the operation again.
Cause:
If the user checks the secure communications checkbox in
Operations Console deviceconfiguration without first configuring
security certificates as outlined in Chapter 6 of theConfiguration
and Administration Guide for Cisco Unified Customer Voice Portal,
the OperationsConsole will not be able to communicate with the
Operations Console Resource Manager onthe device.
Action:
The solution is to edit the orm_jmx.properties in the conf
directory of the device. Change theproperty
com.sun.management.jmxremote.ssl to false to put the device back
intonon-secure mode. Next, uncheck the checkbox in the Operations
Console and restart theOperations Console Resource Manager service
on the device. This will put both the OperationsConsole and the
device Resource Manager back into a non-secure mode. The user can
thenconfigure security certificates before checking the secure
communications checkbox.
Note: Changes in security settings require that the Operations
Console service and the ResourceManager service be restarted.
Resource Manager Service Fails to Start
Symptom:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
15
Chapter 2: Symptoms and Solutions
Call Server
-
Since the default log setting is ERROR for the Resource Manager
Wrapper, the log file shouldprovide exception details of the
failure. The wrapper log file does indicate there is a failure
inResource Manager service startup, but the details of the event
are not logged.
Message:
None.
Cause:
The wrapper log was set to the wrong level.
Action:
The change is unable to be made from Support Tools, so the
change requires manual steps tochange this setting. The log file to
look for the exception is%CVP_HOME%\logs\orm_wrapper.log.
1. Stop the 'Cisco CVP Resource Manager' service.
2. Edit %CVP_HOME%\conf\wrapper.conf file to change the
property'wrapper.logfile.loglevel' value from 'ERROR' to
'INFO'.
3. Save and close the file.
4. Start the 'Cisco CVP Resource Manager' service.
If the Resource Manager service fails to start, the ORM logs
(orm_wrapper.log) collected fromSupport Tools will show the
detailed information of the event.
Call Server Is Not Reachable
Symptom:
After a forced delete of the Reporting Server, the Call Server
state did not change from theDown state to the Partial or Up
state.
Message:
Opsconsole Control Center: Call Server status is down.
Cause:
The Call Server messageAdapter.properties file is corrupt.
Action:
1. From the command line, run %CVP_HOME%\bin\tac\reimage.bat on
the Call Server.
2. Restart the Operations Console Resource Manager (ORM)
service.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
16
Chapter 2: Symptoms and Solutions
Call Server
-
3. Log in to the Operations Console, and choose Device
Management > CVP Call Server.
4. Click Save & Deploy to save the changes and apply them to
the Call Server.
Note: All co-located devicesincluding VXML Server, Reporting
Server, and VideoMedia Servermust be deployed after reimage.bat is
run. You must repeat these stepsfor each co-located device.
Note: Optionally, you can reinstall the CVP device.
Call Server Cannot Communicate with Co-located Reporting
Server
Symptom:
The environment:
Call Server 1 (primary) & Reporting Server 1.
Call Server 2 (secondary)
In the above deployment, when the Operations Console Resource
Manager (ORM) on CallServer 1 is down, the ORM on Call Server 2 is
unable to communicate with the ReportingServer.
Message:
None.
Cause:
This is not a supported model for deployment.
Action:
A co-located Reporting Server can never scale to support
multiple Call Servers and must notbe configured in that manner.
Call Server or VoiceXML Service Cannot Be Restarted
Symptom:
After applying license files, the Call Server service or the
VoiceXML service are unable to berestarted. Both services can be
stopped, but receive a warning message when trying to
restartthem.
Message:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
17
Chapter 2: Symptoms and Solutions
Call Server
-
Windows could not start the Unified CVP Call Server/VXML Server
on Local Computer. Formore information, review the System Event
Log. If this is a non-Microsoft service, contact theservice vendor,
and refer to service-specific error code 0.
Cause:
The problem is due to the side effect of the first version (1.0)
of the MS06-040 patch (KB921883):
Action:
Apply the new version of the Microsoft security patch
KB921883.
See Also
Refer to the Microsoft Security Bulletin KB921883 that is
available on the Microsoft web site.Go to:
http://support.microsoft.com/
Connection with PIM Dropped, Call Server Going Out of
Service
Symptom:
The TCP connection between ICM VRU PG and CVP CallServer ICM
subsystem was dropped.
Calls are dropped during load conditions.
Wireshark can not capture H.323 messages when a call processing
load is running on theCall Server.
Message:
The following errors are written to the CVP error log file when
either the connection betweenthe VRU PIM and CVP drops for an
unknown reason, or the VRU PIM is intentionally shutdown for
maintenance:
LOGMSG_ICM_SS_PIM_SHUTDOWN: Lost socket connection to VRU PIM.
Transitioningto partial service.
LOGMSG_ICM_SS_STATE: Waiting for VRU PIM Connection.
Cause:
The TCP Chimney feature is an offload engine that was introduced
in Windows Server 2003Scalable networking Pack.
Action:
Disable the TCP Chimney feature.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
18
Chapter 2: Symptoms and Solutions
Call Server
http://support.microsoft.com/
-
1. Open Registry Editor, and
locateHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
2. Set the EnableTCPChimney parameter to 0 .
3. Reboot the machine (required).
H.323 Service Cannot Access Call Server
Symptom:
Some HTTP requests from the Unified CVP Voice Browser to the
Call Server are being blocked.
Message:
Blocked by port blocking rule VoiceBrowser.exe Prevent IRC
communication.
Blocked by port blocking rule tomcat5.exe Prevent IRC
communication.
Cause:
VirusScan is blocking ports in the range of 6666-6669, which are
used in IRC communications.The application does not have to be
listening on these ports for it to be blocked. If the applicationis
assigned a client port in that range, the attempt to connect to a
server using that port is blockedby VirusScan on the server
machine. The connect attempt fails.
Action:
Check the VirusScan logs for error messages. access the
logs,VirusScan Console -> Task ->View Log You might find
entries like these: 10/10/2006 3:15:01 AM Blocked by port
blockingrule VoiceBrowser.exe Prevent IRC communication
10.86.129.138 10/10/2006 8:49:01 PMBlocked by port blocking rule
tomcat5.exe Prevent IRC communication 10.86.129.211.
To view VirusScan logs
1. Go to Start > Programs > Network Associates >
VirusScan Console.
2. Click on the Task menu and select View Log.
You might find entries such as:
Blocked by port blocking rule VoiceBrowser.exe Prevent IRC
communication.
Blocked by port blocking rule tomcat5.exe Prevent IRC
communication.
INFO: Retrying communication with Call Server localhost:8000
INFO: Retrying communication with Call Server localhost:8000
Voice Browser to Call Server localhost:8000 connection
unsuccessful.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
19
Chapter 2: Symptoms and Solutions
Call Server
-
Voice Browser cannot access any Call Servers and is no longer
accepting calls.
Voice Browser is out of service.
Exclude the Unified CVP processes from port blocking when using
McAfee VirusScan on asystem running Unified CVP Call Server. If a
different anti-virus product is used, do theequivalent exclusions
for the port blocking rules for that product.
Instructions for McAfee VirusScan on Unified CVP Call
Server.
1. Open the VirusScan Console.
2. Click Access Protection.
3. Click Properties under the Task menu.
4. Click the Port Blocking tab.
5. Check both Prevent IRC Communication rules.
6. Click the first Prevent IRC Communication rule and click
Edit.
7. Enter VoiceBrowser.exe,tomcat5.exe to the Excluded Process
field.
8. Click OK.
9. Click the second Prevent IRC Communication rule and click
Edit.
10. Enter VoiceBrowser.exe,tomcat5.exe to the Excluded Process
field.
11. Click OK.
Resetting System Clock Stops IVR Service Requests
Symptom:
Resetting the system clock on the Call Server causes the IVR
Service to stop functioning.
Message:
None
Cause:
Changing the time of the system clock on the Unified CVP Call
Server causes the IVR Serviceto stop accepting calls.
Action:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
20
Chapter 2: Symptoms and Solutions
Call Server
-
Do not reset the Windows system clock on a machine running
Unified CVP. Resetting theWindows system clock is not supported on
a Call Server.
IVR Service CALL_RESULT Returns Error Code Other than 0
Symptom:
The IVR Service sends a CALL_RESULT error code other than a zero
(0) to the Call Serverlog.
Message:
Below is an example of one of the messages:
4765560: 10.86.129.211: Aug 18 2007 06:09:59:484
-0400:%TEMP:CVP7.0_IVR-7-TRACE_CALL: {Thrd=http-8000-3}
VBServlet:service: Requestfrom 10.86.129.20:
{CALL_ID=979679B9-2DD811DB-B72D0014-6944B762,MSG_TYPE=CALL_RESULT,
CALL_SEQ_NUM=2, ERROR_CODE=32}
Cause:
The problem is due to an error with the Run Script Request from
Unified ICME.
Action:
Check the Error Code that was received in the Event Codes and
Reasons appendix of thisdocument, and take the appropriate action.
For additional information and a complete list of thecause codes,
refer to the Cause Codes and Debug Values section of the Cisco IOS
VoiceTroubleshooting and Monitoring Guide.
Call Server Is Unreachable
Symptom:
The Call Server is unreachable.
Message:
UNREACHABLE
Cause:
A configured component is unreachable in any one of the
following three conditions:
Operations Console is unable to connect to the Operations
Console Resource Manager thatis co-located with the Call Server.
For example, the Operations Console Resource Manageris down.
The Operations Console Resource Manager on the Call Server has
not received a state eventfrom the controller of that
component.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
21
Chapter 2: Symptoms and Solutions
Call Server
-
The Operations Console Resource Manager is unable to connect to
the Call Server so it isnot receiving state events from the central
controller. For example, the Call Server is down.
Action:
In this example, all three services (Unified ICME, IVR, and SIP)
have been configured. Thecentral controller reports that IVR
Service and SIP are IN_SERVICE, but it does not report thestatus of
the Unified ICME Service to the Operations Console for some reason.
When thisoccurs, Operations Console reports the status of Unified
ICME as UNREACHABLE. OperationsConsole aggregates the status of the
various components of the device to arrive at device status.It sees
that two of the components (IVR Service and SIP) are in SERVICE,
but Unified ICMEis UNREACHABLE. Operations Console shows the status
of the Call Server asUNREACHABLE even though IVR Service and SIP
are actually in service.
Check the network environment for connectivity, as well as the
actual state of the server.
Received ERROR_SCRIPT_NAME = 6 on IVR Service
Symptom:
The ERROR_SCRIPT_NAME = 6 error code is received on the IVR
Service.
Message:
ERROR_SCRIPT_NAME = 6
Cause:
The VRU Script Name data that is passed from the Unified ICME to
the IVR Service does notcontain the expected components. These
components are micro-application name, media filename, and media
file type.
Action:
In the Unified ICME Network VRU Script List tool, locate the
appropriate script and resolvethe configuration error.
See Also
For more information, refer to the Micro-Application
configuration section of the Unified CVPConfiguration and
Administration Guide.
Invalid Path or File Name
Symptom:
Note: This is a Call Server issue that is found within the Video
Media Server (VMS), which isa part of the Call Server.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
22
Chapter 2: Symptoms and Solutions
Call Server
-
Sometimes, a file is read from the VMS movies directory with a
name that does not validate tothe standard accepted character set.
These files are marked with the "File could not beread" status.
Message:
File could not properly be read from the file system.
Cause 1:
An invalid status for a media file stating that the "file could
not be read", generallyimplies that the file has an invalid path or
file name.
Action 1:
Verify and correct the path or file name.
Cause 2:
An invalid status for a video media file (that is a 3gp or an
XML file) stating that the "filecould not be properly read",
generally implies that the file has an invalid file name orpresents
within an invalid path.
Action 2:
Verify and correct the path or file name.
The file names can contain alphanumeric characters, underscore,
spaces, and the followingspecial characters:
!$&().,'-;@[]^_`{}~+=. File names cannot lead with a period (.)
or a space.
Only path names with alphabetical, numeric, spaces, and
!$&().,'-;@[]^_`{}~+= characters arepermitted; path names
cannot lead or end with a period (.) or a space.
Dialogue Failure Event Error in the Call Server Log Files
Symptom:
Callers hear critical error media and Dialogue Failure Event
errors appear in the Call Serverlog files.
Message:
None.
Cause:
If a call reaches the end of ICM script processing without being
queued or released, the UnifiedICME sends a Dialogue Failure Event
message to the Call Server. The Call Server, in turn,sends an error
message to the H.323 Service.
Action:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
23
Chapter 2: Symptoms and Solutions
Call Server
-
The author of the script must ensure that each path in the
script ends with one of the followingScript Editor nodes: Release,
Label, Skill Group, or Queue to Skill Group.
Call Server Variable Data Is Invalid Error
Symptom:
The Call Server is reporting an error.
Message:
Variable Data is Invalid.
Cause:
The ICM script might truncate values with leading zeros or zeros
after decimals or roundsvalues.
Action:
Put quotes around numbers in a Script Editor Set node so it is
processed as a string. This isespecially important if:
Leading zeros are present (example: dates).
Trailing zeros are present after a decimal point (example:
currency).
The number is very large (example: a number normally expressed
through exponentialnotation).
Unable Add IOS Device Because Device Is Already Managed
Symptom:
When attempting to add a device to the Operations Console, an
error message is received.
Message:
Gateway device with IP Address: and Hostname: cannot be
created.The device is already managed by another Operations
Console.
Cause:
This occurred because a database was imported from a different
Operations Console. EachOperations Console Server is identified by
a unique identifier. The identifier is generated at thetime of
creating a first device through that Operations Console Server.
Once generated, theidentifier will then be stored in the Operations
Console Server database and in the ResourceManager Configuration
file in every device that includes the Operations Console Server.
The
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
24
Chapter 2: Symptoms and Solutions
Call Server
-
purpose of this unique identifier is to prevent multiple
Operations Console Servers fromconfiguring the same device.
The side effects of importing a database to an Operations
Console Server that is managing atleast one device, if the database
to be imported is an exported database from a different
OperationsConsole Server or is an empty database which was never
used to create a device are:
1. User may not be able to manage devices which were managed by
the same OperationsConsole server prior to the import.
2. User may not be able to create an IOS device such as
Gatekeeper(s), Gateway(s), andCSS(s) from the Operations
Console.
Importing a database which was exported from a different
Operations Console Server is not asupported feature. In this case
the import may succeed but will have various side effects
includingthe above two side effects.
Action:
Importing a database which was exported from a different
Operations Console Server is not asupported feature. The solution
is to never import an empty database to an Operations
ConsoleServer, if the server is managing at least one device prior
to import. If customers require replacingthe existing database with
an empty database, the customer is to reinstall the Operations
ConsoleServer software. Doing so, the customer can no longer manage
devices that were previouslymanaged by the same Operations Console
Server.
VRU Application Error in Call Server Log
Symptom:
The call might not go through the Call Server, so a prompt is
not played. An error or warningmessage might appear in the Call
Server log.
Message:
VRU APPLICATION ERROR: Assigning the tag 7 to the non-existing
ECC variableuser.microapp.error_code.
Cause:
The ECC Variable is not configured on the Unified ICME and/or
NAM software, or the definedlength is not the same on both
devices.
Action:
Add the identical ECC variable definition to the Unified ICME
and/or NAM.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
25
Chapter 2: Symptoms and Solutions
Call Server
-
ECC Variable Contents Truncated after Passing through Call
Server
Symptom:
The contents of an ECC Variable configured in Script Editor are
truncated after passing throughthe Call Server. For example, the
Text-To-Speech (TTS) text that is spoken to the caller is notthe
complete text that was configured. Another example of this behavior
is where there aremedia fetch failures and the URL to the media
file is only a subset of the expected URL.
Message:
None.
Cause:
The cause of this problem may be that the length of the ECC
Variable value that was set inScript Editor is longer than the
maximum length of the ECC Variable configured at setup time.
Action:
The solution is to make the maximum length of the ECC Variables
longer. This is done usingthe Unified ICME Configuration Manager.
If you do alter the maximum length of ECC Variables,you need to
restart the Call Server after making the change.
Note: In a NAM/ICM environment, the length needs to be identical
on all NAM and UnifiedICME environments or the variable will not
pass.
"Have You Configured Your Call Server?" Error
Symptom:
The Call Server is running, but the an error message is logged
in the log file.
Message:
Have you configured your Call Server?
Cause:
The Call Server was started from the Services Control Panel, but
it was not configured fromthe Operations Console.
Action:
Configure the Call Server from the Operations Console. After
completing the configuration,restart the Call Server from the
Services Control Panel.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
26
Chapter 2: Symptoms and Solutions
Call Server
-
Erratic Behavior When Everything Is Configured Properly
Symptom:
General erratic behavior when everything is configured properly.
Sometimes TTS plays,sometimes it does not, or the call stops when
the VXML Server is running in script node inUnified ICME. Other
behaviors include calls succeed in Unified ICME, but no TTS plays,
orthe caller hears silence, but no errors are reported in the log.
Unavailable TTS errors are reported.
Message:
None.
Cause:
If everything appears to be configured properly, and behavior is
generally erratic, check theconfiguration of your Ethernet
link.
Action:
Do the following:
Check the Gateway log files.
Check the VXML Server application logs.
Verify the network settings. The chart below shows the speed and
duplex settings for UnifiedCVP.
Table 1: Speed and Duplex Settings
Server/Gateway NICSpeed/Duplex Setting
Switch Port Speed/DuplexSetting
Server/Gateway NIC SpeedCapability
Ethernet Switch SpeedCapability
Auto / AutoAuto / Auto1000 Mb1000 Mb
100 Mb / Full100 Mb / Full100 Mb1000 Mb
100 Mb / Full100 Mb / Full100 Mb100 Mb
100 Mb / Full100 Mb / Full1000 Mb100 Mb
IVR Subsystem Goes into Disabled State When Call Server Service
Starts
Symptom:
When the Call Server service is started, the IVR subsystem goes
into a Disabled state.
Message:
Cause 1:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
27
Chapter 2: Symptoms and Solutions
Call Server
-
The Unified CVP license in use is a non-expiring license (no
expiration date).
Action 1:
Use a Unified CVP license that has an expiration date
Cause 2:
If the Unified CVP license in use has an expiration date, the
Unified CVP Call Server serviceneed to be recycled.
Action 2:
Recycle the Unified CVP Call Server Service. After recycling,
check to see if the IVR subsystemis not in a Disabled state. If it
is disabled, the process needs to be repeated a few times.
IVR Subsystem Returns a 500 Rejection on VXML Fetch from
Gateway
Symptom:
IVR Subsystem returns a 500 rejection on VXML fetch from the
gateway as seen in the gatewayor CVP logs.
Message:
Cause:
Incorrect setting of parameter "Max Length of DNIS" on ICM
service configuration page ofOperations Console.
Action:
Set parameter to the same length as the DNIS part of the label
returned by ICM.
Full Video Call Disconnected When Routed Through IVP
Symptom:
Full video call is disconnected when it is routed through
IVP.
Message:
Cause:
Access number DN prefixes are not added to the iCONTACT
application and/or the iCONTACTxml configuration.
Action:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
28
Chapter 2: Symptoms and Solutions
Call Server
-
See Chapter 17 in the Configuration and Administration Guide for
Cisco Unified CustomerVoice Portalfor information on modifying the
Access Number in the iCONTACT xml file.
SIP Code 488 or 603 Returned on Full Video Call Rejection
Symptom:
SIP code 488 or 603 is returned on a full video call rejection
from IVP. A Fast Busy is heard.
Message:
Cause:
MCU or MVP/EMP unit on IVP is not registering or is in
service.
Action:
Follow the steps in the Radvision documentation to place the MCU
or MVP in service.
Full Video Warm Transfer with Queuing Not Working
Symptom:
SIP trunk on outgoing call to VRU label is not getting routed to
IVP.
Message:
Cause:
Action:
Set the SIP trunk on the route pattern for the ICME VRU label to
point to IVP instead of CVP.
CVP Warm Transfer With Queuing Get Disconnected
Symptom:
CVP warm transfer with queuing gets disconnected after a few
seconds when the transfer iscomplete. CUCM logs and/or sniffer
trace show a reinvitation collision scenario. This onlyoccurs with
IP originated callers, not TDM callers.
Message:
Cause:
This problem is visible when CVP and CUPS are using proxy and
UDP on the outgoing direction,and the CUCM SIP trunk is using TCP
on the outgoing direction.
Action:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
29
Chapter 2: Symptoms and Solutions
Call Server
-
Set UDP on the CUCM SIP trunk security profile or set CVP and
CUPS to use TCP.
Hold Movie not Visible to Caller on Hold
Symptom:
Note: This tip applies to full video only.
The caller does not see a hold movie when agent places the
caller on hold.
Message:
Cause 1:
Mis-configuration of ECC variables that control the hold movie
to be played to caller.
Action 1:
Make sure the user.microapp.media_server and
user.cvpmovies_bg_media ECC variable areconfigured correctly via
the "Set Variable" node in the ICM script: for
example,user.microapp.media_server = "rtsp://10.86.129.233"
user.cvpmovies_bg_media="app/hold.3gp"
Cause 2:
The default hold movie is not configured in
the/home/rv/ivp/applications/iCONTACT/conf/iCONTACTConfiguration.xml
file.
Action 2:
Verify if a default hold movie is configured in
the/home/rv/ivp/applications/iCONTACT/conf/iCONTACTConfiguration.xml
file as follows:
agent
rtsp:///en-us/app/hold.3gp
Specify the RTSP URL (as shown above) or as a file:// URL (if
file stored locally on IVP
box). (e.g. file:///home/rv/movies/local/hold.3gp). 1
Full Video Calls Drop upon Dialing the DNIS
Symptom:
1) A Linux server in which the IVP software from Radvision is
loaded is refered to as an IVP system. IVP systems are used as part
of the Unified CVP Release7.x solution.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
30
Chapter 2: Symptoms and Solutions
Call Server
-
Upon dialing the Dialed Number Identification Service (DNIS),
full video calls are immediatelydropped.
Message:
none
Cause 1:
The connection between Unified CVP and the Interactive Video
Platform (IVP) system fromRadvision (used as part of the Unified
CVP solution) may not be established.
Action 1:
Verify the following requirements:
The Call Server status is functioning (up) in the OAMP control
center.
A valid connection exists between Unified CVP and IVP via Diag
Servlet. To verify, clickon Dump IVR subsystem and look for the
following ContactClient statements:
ContactClientManager: video enabled: true
ContactClient count: 1
ContactClient [192.168.150.170]:9981,
connectionId:iCONTACT-Connection-1
Look for following log message in CVP log: 2472:
192.168.150.173: Sep 24 200711:35:25.500 -0400:
%CVP_7_0_IVR-1-VIDEO_CONNECTION_UP: Connection toiContact server
[192.168.150.170]:9981 is UP [id:3100]
Cause 2:
The access numbers and/or prefix numbers have not been
configured for iCONTACT in IVPManager and/or iCONTACT configuration
file.
Action 2:
1. Configure the access numbers and/or prefix numbers for
iCONTACT using IVP Manager.
2. Configure the same access numbers and/or prefix numbers in
the iCONTACT configurationfile.
3. Restart IVP after making these changes.
Undecipherable Message Is Displayed When the VMS Subsystem Is
Down
Symptom:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
31
Chapter 2: Symptoms and Solutions
Call Server
-
When playing a video to a caller, the DarwinStreamingServer.exe
is terminated. The video stopsplaying and an error message is
recorded in the VMS Error Log.
Message:
340: 10.30.30.97: Oct 25 2007 16:59:46.267
-0400:%CVP_7_0_VMS-3-VMS_SUBSYSTEM_STREAMING_SERVER_DOWN: The
StreamingServer State: UNKNOWN. [id:11512]
Cause:
Note: There are three internal states outputted by the VMS
Darwin Streaming Server keep-alivethread which help drive the
overall state of the VMS. These are internal states, not
necessarilyrepresentative of the state of the VMS as a whole.
The appropriate error message for an agent to see when VMS
detects Darwin is down is "VMSSubsystem is down". You may see a
different error message if the streaming server polleris in between
cycles when the VMS Subsystem goes down. This error message is
accurate asit is the output from the keep-alive thread, which
tracks the VMS Subsystem through threestates: IN_SERVICE,
PARTIAL_SERVICE and UNKNOWN. These are not the states that theVMS
Subsystem uses.
Action:
To see the appropriate error message, wait for one full polling
cycle (default: 30 sec) to ensurethat the proper error message
appears.
Connection between Unified CVP and IVP cannot be established
when setting up TLS
Symptom:
Note: This tip applies to full video only.
When the Transport Layer Security (TLS) is being set up, the
connection cannot be establishedbetween Unified CVP and IVP
Message:
SSL session handshake failed (is the server SSL enabled?).
Log messages constantly show connection being made and then
broken.
Cause 1:
The TLS setting is enabled on the Call Server but not in
iContact. Enable TLS setting on CallServer and IVP are not in
sync.
Action 1:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
32
Chapter 2: Symptoms and Solutions
Call Server
-
Verify that both Call Server and iContact are configured for
TLS. Call Server configuration isenabled via OAMP. iContact
configuration is via iContactConfiguration.xml. Both Call Serverand
IVP must be restarted if their settings for enabling/disabling TLS
are changed.
Cause 2:
The required procedures to import/export keys between Call
Server and IVP has not completedor is not accurate.
Action 2:
Verify that the proper keys are imported and exported. Refer to
the TLS configurationdocumentation for procedural Call Server/IVP
information.
Only One Call Server Is Receiving Calls
Symptom:
Note: This tip applies to full video only.
When multiple Call Servers are connected to a single iCONTACT,
only one of the Call Serversis receiving calls.
Message:
None
Cause:
iCONTACT configuration may not be set to distribute calls to
Call Servers via round-robinmechanism.
Action:
Open the iCONTACT configuration file and verify that the
notification type is set to roundrobin i.e Round-robin.
Calls Are Disconnecting
Symptom:
There are a number of abnormally disconnected calls.
Message:
Calls are disconnecting.
Cause:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
33
Chapter 2: Symptoms and Solutions
Call Server
-
The Radvision IVP Linux interface, as well as Radvision MVP/MCU,
does not autonegotiateinterface link speed/duplex mode
properly.
Action:
On the Radvision IVP Linux server, use the ethtool utility to
manually set the current interfacelink speed (MB) and duplex
mode.
1. At the command prompt type, ethtool -s eth0 speed 100 duplex
full
2. To permanently change the interface link speed/duplex mode
settings (that is, to survivea reboot), in the
/etc/sysconfig/network-scripts directory add the following line to
theifcfg-eth0 file:
ETHTOOL_OPTS="speed 100 duplex full autoneg off "
Non-English Video Text Overlay is Garbled in Full Video
Symptom:
Message:
Cause 1:
The "user.micropp.charset" ECC variable is not set correctly for
the target language.
Action 1:
Verify that the correct value is set in the
"user.microapp.charset" ECC variable for the targetlanguage in the
ICM script. The supported values values are specified in the
Configuration andAdministration Guide for Cisco Unified Customer
Voice Portal.
Cause 2:
The Radvision MCU is not configured for the target language.
Action 2:
Upgrade the Radvision MCU to the target language so that the
proper fonts are loaded. If theMCU is already configured for the
target language, restart the MCU and EMP
Movie Media of Video Micro-Application Cannot be Interrupted
Symptom:
Note: This tip applies to full video only.
Movie media of video micro-app cannot be interrupted by a DTMF
barge-in or when the agentbecomes available.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
34
Chapter 2: Symptoms and Solutions
Call Server
-
Message:
None.
Cause:
This can happen if the micro-application movie media only has an
audio-stream and does nothave any video stream.
Action:
In the Network VRU script micro-app configuration, change the
Show/Display option in theConfiguration to A for audio-only.
Media Poller timing issues
Symptom:
Metadata files are modified from within the administrative web
UI or from within the file systemwhile the media poller is polling
in the background.
Message:
none
Cause:
The media poller feature is used to automatically add new video
resources from the file system,but can result in some rare timing
issues.
The media poller begins its polling and cycles through the
c:\foo directory. At this time, if anadministrator adds and saves a
new file c:\foo\bar.xml using the administrative web UI,
theadministrator (and possibly agents) see bar.xml displaying in
their web UI. At this point themedia poller cycle ends as
c:\foo\bar.xml was not present when that directory was polled.
Thisis because c:\foo\bar.xml is considered to be deleted by the
media poller and is removed frommemory. Subsequent refreshes of the
administrative or agent web UIs show c:\foo\bar.xml asmissing. The
following media poller cycle picks up c:\foo\bar.xml more
permanently.
Action:
This is a subtle, timing issue. In the end, the media poller
performs its function and gets backto synchronizing with the file
system after another polling cycle. The VMS media poller
processwill never delete or modify existing files on the system. It
will only poll for their existence andcurrent status.
Invalid Thumbnail Images on VMS Server
Symptom:
Invalid images in the VMS movie library causing unintended
consequences on the VMS system.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
35
Chapter 2: Symptoms and Solutions
Call Server
-
Message:
none
Cause:
none
Action:
Be sure to adhere to the following guidelines for thumbnail
images on the VMS system. Validthumbnail image pixel size is 120 x
80 (width x height).Thumbnail images must have the samename as the
XML file, with the exception of the file extension. Supported file
extensions are:.jpg, .gif, and .png.
Note: Large images use more network bandwidth and will impact
Video Media Serverperformance when they are displayed to agents.
Images that are larger or smaller than120 x 80will be resized for
display only. The content on the image itself does not change on
the filesystem.
CALL Server Cannot Connect to IVP Server
Symptom:
Note: This tip applies to full video only.
Call Server cannot get a connection to the IVP server.
Message:
The following is seen in the iCONTACT log: 2007-11-16
18:41:09,320 (iContactStartupHandler)- Can not start iContact
application server. The following is seen in the Call Server
logCVP_7_0_Infrastructure-3-SOCKET_SETUP: Failed getting socket for
host: 10.86.129.104port: 9981 Exception: java.net.ConnectException:
Connection refused: connect
Cause:
The most likely cause is that the iCONTACT XML configuration
file has an XML syntax error.
Action:
Fix any XML syntax errors in the iCONTACT configuration file and
restart the IVP serverusing the IVP Manager.
VMS Administrator Replaces Video Content While Agent is
Previewing
Symptom:
Note: An administrator should be aware that
deleting/replacing/moving video files while callsare being served
can result in unpredictable behavior. VMS is limited in that it
cannot control
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
36
Chapter 2: Symptoms and Solutions
Call Server
-
how the Darwin Streaming Server or the IVP server will react to
file changes. However, theresults are very minor. At worst, the
currently playing video stops, which is not fatal to thesystem.
VMS Administrator replaces a .3gp video file with new content,
but retains the same filenamewhile at the same time a VMS Agent is
Previewing the initial video. The video displaying inthe VMS Agent
QuickTime window freezes.
Message:
none
Cause:
none
Action:
VMS Agent must close the QuickTime video window, reselect the
video, and then Preview.The new video content will now be
available.
VMS Administrator Replaces Video Content While Agent is Pushing
Initial Video to Caller
Symptom:
Note: An administrator should be aware that
deleting/replacing/moving video files while callsare being served
can result in unpredictable behavior. VMS is limited in that it
cannot controlhow the Darwin Streaming Server or the IVP server
will react to file changes. However, theresults are very minor. At
worst, the currently playing video stops, which is not fatal to
thesystem.
VMS Administrator replaces a .3gp video file w\new content, but
retains the same filenamewhile a VMS Agent is Pushing the initial
video to a Caller. The video displaying in the Callerswindow
stops.
Message:
none
Cause:
none
Action:
VMS Agent must reselect the video and then Push it to the
Caller. The new video content willdisplay to the Caller.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
37
Chapter 2: Symptoms and Solutions
Call Server
-
Unable to Process Playback Request
Symptom:
An agent may see this error in the Cisco Unified CVP Video
Desktop if the agent attempts aplayback request after a call has
ended. However, if the agent sees this during an active call,
itmeans that the Video Desktop is still trying to send requests for
a previous call.
Message:
Unable to process playback request. The call has been lost.
Cause:
Action:
To be able to send playback requests for the current call:
1. Close the Video Desktop browser window.
2. Click the Video Desktop button in CTIOS to open the Video
Desktop with the new callinformation.
3. Playback requests should now work correctly.
VXML Server
Ended Session Request Error
Symptom:
This error message appears when an HTTP request initiated by the
root document is made bythe browser for a session that the VXML
Server has already ended. The way the VXML Serverknows what session
the request is asking for is by looking at the cookie included in
the requestfor the sessions ID. A request from the root document
indicates either a VXML error occurredor the caller hung up. In a
typical call, all requests made for a call session come in series
andso there is never a request that comes after the VXML Server
believes the call has ended. Thiserror message would appear if a
request telling the VXML Server to end the session is followedby a
new request initiated by the root document referencing the same
cookie.
Message:
SERVER ERROR: There has been an error with the http session: a
request was made due to anevent caught in the root document that
refers to a session that has already ended.
Cause 1:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
38
Chapter 2: Symptoms and Solutions
VXML Server
-
This is a very rare situation as normal operation would not
cause any further requests to be madeafter the request that ended
the session (it is expected that the session-ending request is the
lastone). Note that for this error to appear, the new request must
come from the root document, inmany cases indicating an error
encountered by the H.323 Service. There could be several
raresituations where this could occur. The first begins with the
end of call event activated by thefinal request to VXML Server
taking a long time to run. This would cause the browser to makea
new request timing out the session. This next request would have to
be made before the initialrequest returned, but after the end of
call event completed, causing the VXML Server to endthe session
before the new request was received. In order for this situation to
occur, the timingwould have to be very precise.
Action 1:
For the first situation, the developer must ensure that their
end of call events do not have thepossibility of taking longer to
execute than the H.323 Services fetch timeout VoiceXMLproperty.
Another possible solution if the end of call event is a Java class
would be to executethe class in a separately spawned thread. The
problem with this would be that the spawnedthread should not
attempt to access content in the session due to the possibility
that it could beinvalidated while the thread was executing.
Cause 2:
Another situation that could cause this would be some network
overhead or slowdown causingthe final response for a call to fail
to arrive at the browser, precipitating a time out to occur.Since
VXML Server functioned without problem and ended the session, this
new request wouldcause the error to occur. Note that some Call
Servers and operating systems under high HTTPload can queue HTTP
requests and potentially even lose requests if the queue is too
large. Theadministrator should investigate this queue if their
system is expected to support high load (forexample, on Apache
Tomcat, this is known as acceptcount).
Action 2:
For the second situation, it may be an indication of excessive
network, operating system, orCall Server load. For the network,
perform diagnostics to ensure proper operation. For theoperating
system, check the CPU and memory utilization to ensure they are not
maxed out. Andfor the Call Server, check the configuration to
ensure the HTTP request queue is not too small(for example, on
Apache Tomcat, the acceptcount setting can be configured in the
conf/server.xmlconfiguration file). Finally, the administrator may
decide to direct less load on the system.
The administrator may also choose to ignore these issues if they
occur rarely since there are nobad consequences for the call
session that encounters this error.
Severity: Due to the fact that this error message appears only
in very rare situations, as well asthe fact that even if it does
occur, the caller does not encounter any issues with their call,
thiserror message is benign.
Ending Session Request Error
Symptom:
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
39
Chapter 2: Symptoms and Solutions
VXML Server
-
This error message appears when an HTTP request initiated by the
root document is made bythe browser for a session that VXML Server
has started the process of ending but that processis not complete.
This is similar to the previous error, but where in that situation
the request wasmade after the session had completed, this situation
involves the request occurring while theVXML Server was ending the
session but before it completed.
Message:
SERVER ERROR: An HTTP request was made due to an event caught in
the root documentto a session that has been slated for
invalidation.
Cause:
One very rare situation that could cause this error to occur
under standard call volume involvesa perfect timing of events. This
error would not be produced if the extra request arrives whilean
end of call event was executing. However if it did take a long time
to execute, promptingthe browser to time out the request by having
the root document make a new request, and thatrequest came at
exactly the time the end of call event completed but before VXML
Server wasable to finish the session ending process, this error
would occur. Since this process normallyoccurs rapidly, the chances
that this would occur during this period is very small.
This error would also be encountered if the network, operating
system, or Call Server on whichVXML Server is installed is
sufficiently overloaded such that either HTTP requests are
beingqueued up too much, HTTP requests are being dropped or the
Call Server is so overloaded thatthe normally trivial process of
ending a session takes a non-trivial amount of time.
Severity: The severity for this error is very low because there
is no effect on the caller sincethis occurs after the call is
complete. Additionally, the situations in which this can occur
duringnormal operation are extremely rare.
Action:
For the first situation, the developer must ensure that their
end of call events do not have thepossibility of taking longer to
execute than the H.323 Services fetch timeout VoiceXMLproperty.
Another possible solution if the end of call event is a Java class
would be to executethe class in a separately spawned thread. As
long as the end of call class completed its executionbefore the
session invalidation delay specified in the conf/global_conf.xml of
the installationdirectory, there would be no risk of the class
referring to data within the session after it wasinvalidated.
For the second situation, it may be an indication of excessive
network, operating system, orCall Server load. For the network,
perform diagnostics to ensure proper operation. For theoperating
system, check the CPU and memory utilization to ensure they are not
maxed out. Andfor the Call Server, check the configuration to
ensure the HTTP request queue is not too small(for example, on
Apache Tomcat, the acceptcount setting can be configured in the
conf/server.xmlconfiguration file). Finally, the administrator may
decide to direct less load on the system.
The administrator may also choose to ignore these issues if they
occur rarely since there are nobad consequences for the call
session that encounters this error.
Troubleshooting Guide for Cisco Unified Customer Voice Portal
7.0(2)
40
Chapter 2: Symptoms and Solutions
VXML Server
-
Session Does Not Contain Call Information
Symptom:
This is a rare error message that is displayed after end session
request error. In this situation,VXML Server has completed its
process of ending a session, however the Call Server has
notcompleted its process for ending the session when a new HTTP
request is received referencingthat session. This would yield a
legitimate session from the Call Servers standpoint but aninvalid
session by VXML Servers standpoint.
Message:
SERVER ERROR: The session is not associated with a Server
Configuration.
Cause:
The situations where this would be encountered are identical to
those listed in the previous errorcondition except the timing of
the second request would occur after the standard end of
callprocess taken by VXML Server completed but before the Call
Server actually removed thesession. The timing for this situation
is actually tighter than the previous situation and thereforewould
occur even more rarely. While there is a chance that this could
occur under standard callvolume, it would more likely occur only
under extreme load to the network, operating system,or network.
Severity: The severity for this error is very low because there
is no effect on the caller since