Tools and Processes for Testing VoIP Chris Bajorek, Director CT Labs .

Tools and Processes Tools and Processes for Testing VoIPfor Testing VoIP

Chris Bajorek, DirectorChris Bajorek, DirectorCT LabsCT Labs

www.ct-labs.comwww.ct-labs.com

About the SpeakerChris Bajorek, Director and Founder, CT Labs

Chris Bajorek is a 25-year veteran of computer telephony and converged communications. Bajorek has led the company to its industry-leading position in testing services which include real-world performance testing, interoperability verification, and usability and quality analysis. Customers include first-tier enterprise and carrier-grade next-generation network product manufacturers.

Prior to founding CT Labs, Bajorek founded Telephone Response Technologies, Inc. (TRT), which developed and sold turnkey voice response and unified messaging products as well as award-winning toolkits for rapid development of voice-based applications. Prior to TRT he worked for Integrated Office Systems and Time and Space Processing where he performed pioneering work on voicemail and digital voice communications products.

Bajorek holds a B.S.E.E. from Cal Poly, San Luis Obispo.

For Today’s Talk: Taking a Developer’s Perspective to VoIP

Test• Much of CT Labs’ business is with R&D and

QA groups of VoIP product manufacturers• Would like to provide a window into some of

our VoIP test experiences, includingo Common VoIP test mythso Testing tips and suggestionso Focus on voice quality testing—hot area for VoIP

Myths around VoIP Deployment

• Voice quality is a given• VoIP is easy to deploy • VoIP is inexpensive to deploy• All VoIP-enabled phones are created equal• Once you have your VoIP network set up,

you can leave it alone

VoIP Requires a Lifecycle Approach

• Lack of a proper lifecycle will:o Drive Costs Upo Reduce VoIP Reliability / Availabilityo Risk Complete Failure of Deployment

Step 1

Pre-Deployment

Assess,Design,Baseline

Step 2

Install&

Cutover

Step 3

Manage:Identify,

Diagnose,Repair

Step 4

Optimize

Should design new VoIP

products with this in mind

VoIP Troubleshooting Areas – The Big Picture

• Call Processing (i.e. call connectivity, service availability)

• Voice Quality• Interoperability / Feature Interaction• Configuration / Registration• Routing • Security• Applications (conferencing, IVR, voicemail, …)

Troubleshooting example• Symptom: sporadic call “failures”

• Common causes :o Gateway and switch

mis-configurationo Interoperability issues

between equipmento Capacity limitationso Performance issues and

delays triggering timeoutso “Feature interaction” issues such as conflicting

call-forwarding settings

VoIP Deployment Segments

1. Residential (Voice over Broadband)2. Enterprise3. Next-Gen Network Carriers and Service

Providers

All three areas are quite active now…

VoIP Products, by Segment(products that “touch” the media stream)

• Residentialo Analog terminal adapters, VoIP softphones,

residential routers

• Enterpriseo IP PBXs, IP Contact Centers, VoIP phones &

softphones, firewalls/ALGs, media servers (conferencing, voice mail)

• Next-Gen Carriers and Service Providerso Session border controllers, media servers, media

gateways, transcoding/VQ enhancement processors

VoIP Testing Areas of Focus• Service reliability

o i.e. Availability of service, Call connectivity

• Voice qualityo Includes measurement of VQ, latency, levels, echo can.,

etc.

• “Phone” featureso CLASS features, such as call park, transfer, etc.

• VoIP Access to enhanced serviceso Voice mail, conferencing, IVR, etc.

• Each of these areas has its own set of testing challenges, but one thing is clear: all relate to the end-user Quality Experience and must be validated

Active versus Passive VoIP Testing

• Active testso Involves driving real 2-way calls thru the VoIP networko Benefits: more accurate, uses mature standards

(PESQ, etc.) for automated quality assessmento Negatives: consumes network resources

• Passive testso Involves passive evaluation of call-based packet flowso Ignores (or models) VoIP endpoint-specific behaviors

to network conditions

Post-Deployment, Passive Testing is Key

• Deployed VoIP networks should:o Continuously monitor passive VQ, call completion

rates, network packet loss, jitter, & latencyo Set alarming thresholds for VoIP call performance

that degrades below adaptive-corrective levels

• Assumption: Pre-deployment tests resulted in…o Clean bill of network healtho Baseline characterization of network during peak,

off-peak times

Passive Monitoring “Embedded Components” for Product

Developers• Products incorporating these can quickly

adapt to changing IP network conditionso Real-time access to estimated MOS, round-trip

latencyo Access to level and echo information for estimate

of MOS-Conversational Quality

• VQMon – from Telchemy (www.telchemy.com)

• PsyVoIP -- from Psytechnics (www.psytechnics.com)

A few things about Codecs• Waveform codecs

o Produces waveform as identical as possible to the original (G.711 PCM, G.726 ADPCM)

• Source codecso Uses a model of how speech is generatedo Can significantly alter the time-domain waveform

while sounding very similar to the input (G.729a/729, G.723.1)

A few things about Codecs

• Hybrid codecso Combine techniques from waveform and source

codecso Uses different modes and bit rates depending on

network conditionso AMR

• Bit rate: 4.75-12.2 kbps MIPS complexity: 15-20

o AMR-WB / G.722.2 (wideband—7kHz signal bw)• Bit rate: 6.6-28.3 kbps MIPS complexity: 38 (incl. VAD and CNG)

• Why knowledge of codec method(s) is useful for VQ analysis

Devices that can affect a User’s “VoIP Experience”

• IP PBXs

• IP Phones & VoIP endpoints

• Media Gateways

• IVR / Voice portals

• SBCs (Border Controllers)

• Media Servers• Firewalls/ALGs• Messaging Servers• Conference Bridges

Voice Quality versus Intelligibility

• Voice quality: the “acceptability” of speech• Intelligibility: the “clarity” of speech

o Subjective tests: Diagnostic Rhyme Test, Modified Rhyme Test

o Higher frequencies more important for intelligibility, a good benefit of wideband codecs

• Lower quality affects intelligibility but not necessarily vice versa

Voice Quality Measurement – A Hot Topic

• What is considered the “gold standard” way to measure voice quality?o Answer: with humans, and the more of them in a

listening session the better the resolution of the resulting quality scores

• However, conducting a live-listener test is not as easy or cheap as you may think…

MOS Subjective Testing

• It’s a Standard: ITU-T P.800 (1996)• The technique rates quality using “absolute

category rating” method (ACR) 5-grade scale: 5=excellent 4=good 3=fair 2=poor 1=bad

MOS Subjective Testing

• How it’s doneo Requires use of a group of 32-64 “naive” listenerso Standardized male, female, and child phrases are

usedo Calibrating “reference” degraded conditions are

intermixed with actual sampleso The identical speech sample sets are played to all

listenerso Listeners judge the quality of each phrase using ACR

scale

MOS Subjective Testing

• Strengthso Provides the definitive answer to “which sounds

best?”

• Weaknesseso High cost, especially when many different test

conditions or sample sets must be evaluatedo Takes time to schedule test and get results

Objective VQ Standards

• All automated VQ measurement techniques are designed to estimate the way humans perceive voice quality

• PSQM P.861 (1996)o PSQM+ handled higher distortion levels than

PSQM

• PESQ P.862 (2001)o Solved variable delay (“alignment”) problem of

PSQM

What PESQ VQ Testing is designed for

• PESQ is a way to quickly and cost-effectively estimate the effects of one-way speech distortion and noise on speech quality

• PESQ is “endpoint-agnostic” – can be used for VoIP-to-VoIP, VoIP-to-PSTN calls, etc.

• PESQ can be used for VQ assessment of wideband codecs if your test platform supports it (if not, 3.1kHz signal bandwidth applies)

PESQ Narrowband vs. WidebandEffect of using PESQ Narrowband

to Score Wideband Signals

00.5

11.5

22.5

33.5

44.5

5

Self 5 kHz filtered 3 kHz filtered8kHz Signal BW Reference Compared to

PE

SQ

Sc

ore

PESQ WB

PESQ NB

What PESQ VQ Testing is not designed for

• PESQ does not evaluate the effects of loudness loss, fixed latency, sidetone, or echo as related to two-way caller interactions

• PESQ can not safely be used to declare a VQ “winner” when the PESQ score differential is small (i.e. <.25)o “Opposite conclusion” errors are very possible, so the

bigger the score differential the better o Especially true when comparing samples with more than a

single changed “variable”

Objective VQ Testing

• Strengthso Provides excellent estimate of voice qualityo Tests can be performed quicklyo Tests are very repeatable

• Weaknesseso Not good for reliably resolving small differences

in quality scores

• Must look at all the metrics of VoIP calls exactly as transmitted on the network

• Packet Loss ?• Jitter ?• Delay ?• Voice Quality ?

Jitter distribution graph

Troubleshooting VQ Issues

Measurement is critical for problem resolution

Tip: How to test end-to-end VQof VoIP phones

• #1: It’s usually not enough to evaluate VQ by just looking at the packet streams (i.e. E-model)

• #2: Must evaluate quality all the way to the phone’s earpiece and microphone wireso So can evaluate the proper operation of the phone’s

internal “VoIP gateway”, including automatic gain (AGC), voice activity detection (VAD), comfort noise generation (CNG), echo cancellation, codecs, jitter buffer management, and packet loss concealment algorithms.

o In other words, there is much that can go wrong.

Tip: How to test end-to-end VQof VoIP phones

• #3: Must evaluate under expected LAN/WAN impairment conditionso Packet loss, Jitter, Latencyo Effective bandwidth of IP connection

• i.e. Broadband versus Dialup

• #4: Don’t forget interoperability testing against other VoIP deviceso Verify VQ against other expected manufacturer’s

devices

Testing end-to-end VQ of VoIP phones

• The automated VQ testo Important for verifying VQ under many conditions o Vary one dimension at a time during subsequent

test runs

• The manual VQ “real user” testo Conduct 2-way calls with real users who are

familiar with potential echo cancellation and other 2-way effects

o Include handset and speakerphone test calls

Testing end-to-end VQ of VoIP phones

• Test setup exampleso Softphone to softphone testo VoIP Phone to VoIP Phone test (in lab)o VoIP Phone to PSTN call testo Variations on these themes easily set up

• Wideband codecs used? If so, be sure to verify that all test equipment in the audio/media signal path can support 8 kHz.

Testing Softphone-to-Softphone

Internet to VoIP Network

Workstation #1+ VoIP Client A

Workstation #2+ VoIP Client B

Empirix NetEmWAN emulator

CT Labs Sound Studio Workstation

Playback audio (speaker)

speakerLine Level

microphoneLine Level

Recording audio (microphone)

Router

T1

Media may flow peer-to-peer or through the VoIP Network component

PESQ evaluated off-line via batch process

Testing VoIP Phone-to-VoIP Phone

JKAudioAdapter

JKAudioAdapter

IP Phone #1 (handset removed)

IP Phone #2 (handset removed)

WAN emulator

Automated Call Generator

Good setup when isolated device performance test is needed.Phone calls are manually placed with this setup.

Testing VoIP Phone to PSTN calls

Automated Call Generator

PSTN

VoIP Network

WAN emulatorRouter

POTS or T1

T1

JKAudioAdapter

VoIP Phone

Example: WAN Impairment Conditionsfor VQ Test

Conditions suitable for emulation of overseas Internet dialup conditions

Broadband and Dialup IP bandwidths for each condition below:

Packet Loss = 0%Latency / Jitter =10/30 mSec (uniform distributed latency model)

Packet Loss – Random = 2.5%, Latency / Jitter = 10/30 mSec

Packet Loss – Burst = 5.0%, 1-5 packet burst size Latency / Jitter = 50/80 mSec

Packet Loss – Burst = 10.0%, 1-8 packet burst size Latency / Jitter = 125/250 mSec

Watch out for…

• Do not try to compare “MOS” scores derived from different sources or evaluation engineso Even the numeric ranges from “worse” to “best”

can vary (i.e. “best” = 4.5, not 5.0)o Especially, don’t compare passive with active VQ

results

Real-World Next-Gen Network

Product Testing

www.ct-labs.com 916-577-2100

Chris [email protected]

916-577-2110 direct line