Troubleshooting Voice and Video Performance over IP using Measurement Tools… Prasad Calyam and Paul Schopis, ITEC-Ohio, OARnet and Ohio Supercomputer Center,

Troubleshooting Voice and Troubleshooting Voice and Video Performance over IP Video Performance over IP

using Measurement Tools…using Measurement Tools… Prasad Calyam and Paul Schopis,Prasad Calyam and Paul Schopis,ITEC-Ohio, OARnet and Ohio Supercomputer Center, ITEC-Ohio, OARnet and Ohio Supercomputer Center,

Tools Tutorial,Tools Tutorial, Joint Techs, Salt Lake City, Utah, February 2005Joint Techs, Salt Lake City, Utah, February 2005

Topics of Discussion

Basics of Voice and Video over IP performance measurementMeasurement Tools we have experimented with…“H.323 Beacon” Tool we have developed!

How is this tool different?Tool FeaturesTwo Case-studies

An Useful H.323 Beacon Tool Supplement!!!Ethereal RTP Analysis Module

Conclusion

Basics of Voice and Video over IP performance measurement

Voice and Video Traffic are inherently different from Data Traffic

They use Signaling protocols such as H.323, SIP, …Media (Voice and Video payload) is delivered over IP using RTP packets irrespective of which codec or end-point technology (PC-based Vs Appliance-based)

General ICMP and UDP based tools fail to totally capture the performance bottlenecks faced by actual voice and video traffic at the end-host and in the network

i.e.- ping, traceroute, Iperf, pathrate, … don’t suffice!We need to measure end-user experience of Voice and Video over IP applications also!

Measuring End-user experience…

Two approaches to evaluating end-user experience of audiovisual quality

Subjective Measurements Involve human participants to rate audiovisual quality Can you hear me now? Mean Opinion Score (MOS) Ranking technique (ITU-T P.800) Not just “Good”!

Objective Measurements Automated techniques to rate audiovisual quality

“E-Model” [ITU-T G.107]Perceptual Evaluation of Speech Quality (PESQ) [ITU-T P.862]

Mean Opinion Score (MOS) Concept

Scenario I: A Researcher and an Industry professional want to Videoconference

Internet2 Abilene Network

GigaPOP

OC2

OC192

What are the end-host and network problems?

Case1:Researcher is unable to make a call!

Internet2 Abilene Network

GigaPOP

OC2

OC192

There was a mis-configured firewall blocking necessary ports…

Internet2 Abilene Network

GigaPOP

OC2

OC192

Case2: Industry professional is unable to make a call!

Internet2 Abilene Network

GigaPOP

OC2

OC192

His LAN’s Internet connectivity was non-functional at that time…

Internet2 Abilene Network

GigaPOP

OC2

OC192

Case3: They connected, but of them experienced bad audio & video!

Internet2 Abilene Network

GigaPOP

OC2

OC192

There was congestion at one of the intermediate routers along the path…

Internet2 Abilene Network

GigaPOP

OC2

OC192

There was congestion at one of the intermediate routers along the path…

Internet2 Abilene Network

GigaPOP

OC2

OC192

There was congestion at one of the intermediate routers along the path…

Internet2 Abilene Network

GigaPOP

OC2

OC192

The performance problem can be anywhere in the E2E Path!!!

Internet2 Abilene Network

GigaPOP

OC2

OC192

What are the other common end-to-end performance problems?

Common problems involving endpoint devicesFailure of audio and video hardware, out dated or buggy end-point application softwareFaulty connections and configurations of audio and video interfacesLack of lip-synchronizationMis-configured jitter buffer sizesNon inter-operable end-point application softwareLack of forward error correction and echo cancellation mechanisms in end-point clientsLack of end-user training

Common End-to-End performance problems in VVoIP Systems (2)

Common problems involving network devices Insufficient network capacity for handling multiple high data rate videoconferencing calls

Excessive delay, loss, jitter, out of order packets and re-ordered packets in the network

Duplex mis-match problems

Traffic congestion at peak-usage periods of the network

Mis-configured priorities for real-time audio and video traffic streams in the network

Asymmetric routing with excessive delays on one path

Lack of network engineering resource personnel

Common End-to-End performance problems in VVoIP Systems (3)

Common problems involving application service devices

Misconfigured firewalls that block required ports

Non-H.323 friendly NATs

Misconfigured devices

MCUs

Gatekeeper

Gateway devices

Outdated or buggy application-service device application software

Measurement Tools we have experimented with…

MCU-specific Performance Monitors Per-call monitoring of many network and end-point parameters (delay, jitter, loss, reordering)

Telchemy VQMonBased on E-Model Standard; Integrated into many vendor VoIP troubleshooting products (e.g., Brix Networks, Agilent, etc…)

PESQ Based on PESQ Standard; Used in many products (e.g., Opticom, Acterna, etc…)

Hammer Call AnalyzerPacket Analyzer (Ethereal backend) with many features such as call-flow display, jitter buffer analysis, wave file analysis, etc…

NettachDeveloped originally by PictureTel; Similar to H.323 Beacon where you emulate Voice and Video traffic and measure performance

Many other good tools… (AppareNet, ViDeNet Scout (based on NetIQ Chariot), etc.)

MCU-specific Performance Monitors

Increased Video Sync Loss!

Telchemy VQMon

PESQ

SAMPLE RESULTS: REFERENCE DEGRADED PESQMOS PESQMOS original.wav degraded.wav SQValue=2.930 2.930

Hammer Call Analyzer

Taken from- http://www.empirix.com

Nettach

H.323 Beacon Overview

An application-specific measurement tool To monitor and qualify the performance of H.323 Videoconferencing sessions at the host and in the network (end-to-end)

Useful to an end-user/conference operator/network engineer

Uses OpenH323 and J323Engine libraries

Comparison with other existing tools…

Addresses problems due to H.323 protocol-specific idiosyncrasies

Can be generalized to RTP packets performance over the networkMany in-built tools that generate various kinds of measurement data for pre/during/post Videoconference troubleshooting!

An “easy to install and use” tool that is open source (FREE!)

Commercial tools are VERY expensive!NetIQ Chariot, Spirent Smartbits, Telchemy VQMon,…

Initial call setup failures and haphazard disconnections…

Test Status In-Session, Normal Close, Exception Close

Exception Close Alarms “Possible Firewall/NAT presence obstruction”

“No Internet connectivity”

“Incompatible codec being used”

“Insufficient bandwidth”

“Remote H.323 Beacon Server not online”

Initial call setup failures and haphazard disconnections…

Network Health Status…

Delay, Jitter and Loss data Real-time, offline raw data and test session summary

Network Health Plots…

Watermarks for “Good”, “Acceptable” and “Poor” grade of quality as experienced by end-userDelay: (0-150)ms, (150-300)ms, > 300msJitter: (0-20)ms, (20-50)ms, > 50msLoss: (0-0.5)%, (0.5-1.5)%, >1.5%

Levels obtained from our PAM 2004 paper

Poor

Acceptable

Good

Audio and Video Quality Assessments

Audio and video loopback feature

E-Model-based objective MOS ranking

Slider-based subjective MOS ranking

Customization of tests…

Test results data folder, TCP/UDP/RTP port settings, H.225 and H.245 parameters, preferred codec, watermarks for delay, jitter, loss, …

Use-case I

Effect of a mis-configured firewall on a video stream

Problem report Intermittent frame freezing

Lot of pixilation

No significant audio problems

Sudden disconnections

Use-case I (2)

MCU Software ~15 fps for 384Kbps call and ~0.6% packet loss ~30 fps for 128Kbps call and ~0% packet loss

H.323 Beacon Sluggish call-setup Delayed packet-events Initial jitter variations in poor range

Jitter variations indicated by H.323 Beacon client-to-server test

Increasing the rate-limit and firewall re-configuration solved the problem!

Use-case II

Problem report Poor audio and video reception at the site of user Problem only in the streams from Campus A to Campus B

MCU Software Packet loss (~1%) Packet re-ordering

H.323 Beacon Jitter values consistently in poor range Traceroute and Reverse-Traceroute showed an asymmetric route E-Model results

Campus A to Campus B: 3.78Campus B to Campus A: 4.37

Use-case II (2)

Fig. 2 Time-plot of Delay

Fig. 3 Time-plot of Loss Fig. 4 Time-plot of Mean Opinion Score Ranking

Fig. 1 Time-plot of Equipment Impairment Factor

Problem was solved by contacting the neighboring ISP who was suspected to be experiencing problems!

Scenario I: H.323 Beacon used in a Network Measurement Infrastructure…

Internet2 Abilene Network

GigaPOP

OC2

OC192

3Com

CISCOSYSTEMS

3Com

CISCOSYSTEMS

Core Router

Switch

NMS

CDMA Device

How do you setup the H.323 Beacon in a network?

Scenario II: In a Multi-point setting…

Scenario II: In a Multi-point setting…

Scenario II: In a Multi-point setting…

Ethereal RTP Analysis Module

An useful H.323 Beacon Tool supplement!!!Adds advantages of passive measurement to the H.323 Beacon

In general, to any RTP-based application!

Generates inter-packet jitter, packet-size distribution, packet loss and re-ordering information from input traffic tracesWe have written simple parsing scripts that can process data in a format that facilitates analysis using Minitab or ExcelUse the OPENXTRA version of Ethereal!

http://resource.intel.com/telecom/support/appnotes/9008/9008an.pdf

Ethereal RTP Analysis Module (2)

Steps for analyzing the TracesLoad the packet trace into Ethereal

Trace will contain both forward and reverse direction streams (Check “Source” and “Destination” IP addresses)

Decode streams as RTP (default is UDP) This will mark all related packets as belonging to a specific audio and video codec streams

Analyze individual audio or video streamsImport various information fields as .csv file (“Save as CSV” option)Also has wave file generation relating to an audio stream (“Save Payload” option)

Works only for G.711 Codec streams!Good for PESQ where you want to compare original and degraded wave files to obtain Objective MOS information

Ethereal RTP Analysis Module (3)

General UDP Stream decoded as an H.263

payload stream

Ethereal RTP Analysis Module (4)

Audio Stream Video Stream

Ethereal RTP Analysis Module (5)

Pink-marked packets relate to either lost or re-ordered packets!

Re-ordering; 45413 45415 45414(Observe Sequence #s; could also be 2

consecutive packet losses)

Loss

Ethereal RTP Analysis Module (6)

An Imported CSV File!

Ethereal RTP Analysis Module (6)

Create interesting visualizations to understand various RTP packet characteristics; can do the same for both Voice and Video packets!!!

Questions?

http://www.itecohio.org/beacon

H.323 Beacon Reference: