Measuring Interaction QoE in Internet Videoconferencing Prasad Calyam (Presenter) Ohio Supercomputer Center, The Ohio State University Mark Haffner, Prof.

Measuring Interaction QoE in Internet Videoconferencing

Prasad Calyam (Presenter)Ohio Supercomputer Center, The Ohio State University

Mark Haffner, Prof. Eylem Ekici Prof. Chang-Gun Lee The Ohio State University Seoul National University

MMNS, November 1st 2007

http://www.osu.edu/

2

Outline• Background

• Voice and Video over IP (VVoIP) Overview• Network QoS and End-user QoE in VVoIP• Streaming QoE versus Interaction QoE• Network Fault Events

• Multi-Activity Packet Trains (MAPTs) methodology• Participant Interaction Patterns• Traffic Model for MAPTs emulation

• Vperf tool implementation of MAPTs• Performance Evaluation• Concluding Remarks and Future Work

3





4

Voice and Video over IP (VVoIP) Overview

Large-scale deployments of VVoIP are on the rise Video streaming (one-way voice and video)

MySpace, Google Video, YouTube, IPTV, … Video conferencing (two-way voice and video)

Polycom, MSN Messenger, WebEx, Acrobat Connect, …

Challenges for large-scale VVoIP deployment Real-time or online monitoring of end-user Quality of Experience (QoE)

Traditional network Quality of Service (QoS) monitoring not adequate Network QoS metrics: bandwidth, delay, jitter, loss

Need objective techniques for automated network-wide monitoring Cannot rely on end-users to provide subjective rankings – expensive and

time consuming

5

Network QoS and End-user QoE

End-user QoE is mainly dependent on the combined impact of network factors Device factors such as voice/video codecs, peak video bit rate (a.k.a. dialing speed)

also matter

Our study maps the network QoS to end-user QoE for a given set of commonly used device factors H.263 video codec, G.711 voice codec, 256/384/768 Kbps dialing speeds

End-user QoENetwork QoS

6

Voice and Video Packet Streams

Total packet size (tps) – sum of payload (ps), IP/UDP/RTP header (40 bytes), and Ethernet header (14 bytes)

Dialing speed is ; = 64 Kbps fixed for G.711 voice codec Voice has fixed packet sizes (tpsvoice ≤ 534 bytes) Video packet sizes are dependent on alev in the content

7

Video alev Low alev

Slow body movements and constant background; E.g. Claire video sequence

High alev

Rapid body movements and/or quick scene changes; E.g. Foreman video sequence

‘Listening’ versus ‘Talking’ Talking video alev(i.e., High) consumes more bandwidth than Listening video alev (i.e., Low)

Claire Foreman

8

End-user QoE Types Streaming QoE

End-user QoE affected just by voice and video impairments Video frame freezing Voice drop-outs Lack of lip sync between voice and video

Interaction QoE End-user QoE also affected by additional interaction effort in a conversation

“Can you repeat what you just said?” “This line is noisy, lets hang-up and reconnect…”

QoE is measured using “Mean Opinion Score” (MOS) rankings

9

Network Fault Events

“Best-effort service” of Internet causes network fault events that impact application performance Cross traffic congestion, routing instabilities, physical link failures, DDoS attacks

Our definition of network fault events is based on the “Good”, “Acceptable” and “Poor” (GAP) performance levels for QoS metrics causing GAP QoE Type-I: Performance of any network factor changes from Good grade to

Acceptable grade over a 5 second duration Type-II: Performance of any network factor changes from Good grade to Poor

grade over a 10 second duration

10

Related Work Characteristics of network fault events well understood

Bursts, spikes, complex patterns – lasting few seconds to a few minutes (Markopoulou et. al., Ciavattone et. al.)

Measuring Streaming QoE impact due to network fault events has been well studied ITU-T E-Model is a success story for VoIP QoE estimation

Designed for CBR voice traffic and handles only voice related impairments ITU-T J.144 developed for VVoIP QoE measurement

“PSNR-based MOS” – Requires original and reconstructed video frames for frame-by-frame comparisons

Offline method - PSNR calculation is a time consuming and computationally intensive process

Online VVoIP QoE measurement proposals PSQA (G. Rubino, et. al.), rPSNR (S. Tao, et. al.)

Measuring Interaction QoE impact due to network fault events has NOT received due attention Need for schemes to measure interaction difficulties in voice and video conferences

presented by A. Rix, et. al.

11

Problem Summary

Given: Voice/video codecs used in a videoconference Dialing speed of the videoconference Network fault event types to monitor

Develop: An objective technique that can measure Interactive VVoIP QoE Real-time measurement without involving actual end-users, video

sequences and VVoIP appliances An active measurement tool that can: (a) emulate VVoIP traffic on a network

path, and (b) use the objective technique to produce Interaction QoE measurements

Vperf Tool

Multi-Activity Packet Trains Methodology

12





13

Proposed Solution Methodology

“Multi-Activity Packet Trains” (MAPTs) measure Interaction QoE in an automated manner They mimic participant interaction patterns and video activity levels

as affected by network fault events Given a session-agenda, excessive talking than normal due to

unwanted participant interaction patterns impacts Interaction QoE “Unwanted Agenda-bandwidth” measurement and compare with

baseline (consumption during normal conditions) Higher values indicate poor interaction QoE and caution about

potential increase in Internet traffic congestion levels Measurements serve as an input for ISPs to improve network

performance using suitable traffic engineering techniques

14

Proposed Solution Methodology (2)

‘repeat’‘disconnect’‘reconnect’‘reorient’

Type-I and Type-II fault detection

15

Participant Interaction Patterns Assumption: Question (Request) and Answer (Response) items in a

session agenda Side-A listening when Side-B talking, and vice versa

Normal – PIP1

16

Participant Interaction Patterns (2)

Participant Interaction Patterns (PIPs) using MAPTs for a “Type-I” network fault event

Repeat – PIP2

17


Participant Interaction Patterns (PIPs) using MAPTs for a “Type-II” network fault event

Disconnect/Reconnect/Reorient – PIP3

18


Our goal is to measure the Unwanted Agenda-bandwidth and Unwanted Agenda-time measurements after MAPTs emulation of the Q & A session agenda

19

Traffic Model for MAPTs Emulation

Traffic Model for probing packet trains obtained from trace-analysis Combine popularly used low and high alev video sequences and model them at

256/384/768 Kbps dialing speeds for H.263 video codec Low – Grandma, Kelly, Claire, Mother/Daughter, Salesman High – Foreman, Car Phone, Tempete, Mobile, Park Run

Modeling Video Encoding Rates (bsnd) time series Packet Size (tps) distribution Derived instantaneous inter-packet times (tps) by dividing instantaneous

packet sizes by video encoding rates

20

Video Encoding Rates (bsnd) Modeling Time-series modeling of the bsnd data using the classical

decomposition method We find a Second order moving average [MA(2)] process

model fit θ1 – MA(1) parameter θ2 – MA(2) parameter

Low alev High alev

21

Video Packet Size (tps) Distribution Modeling

Distribution-fit analysis on the tps data We find a Gamma distribution fit

α – shape parameter β – scale parameter

For High alevat 256 Kbps

22

Traffic Model Parameters for MAPTs Emulation

23





24

Vperf Tool Implementation of MAPTs

Per-second frequency of “Interim Test Report” generation Interaction QoE reported by Vperf tool - based on the progress of the session-

agenda

25

Example – Session Agenda and Network Factor Limits File

26





27

MAPTs Emulation at different Dialing Speeds

256 Kbps

768 Kbps384 Kbps

28

MAPTs Measurements Evaluation Increased the number of Type-I and Type-II network fault events in a

controlled LAN testbed for a fixed session-agenda NISTnet network emulator for network fault generation

Recorded Unwanted Agenda-Bandwidth and Unwanted Agenda-Time measured by Vperf tool

(a) Impact of Type-I Network Fault Events on Unwanted Agenda-Bandwidth

(b) Impact of Type-II Network Fault Events on Unwanted Agenda-Bandwidth

29

MAPTs Measurements Evaluation (2)

(c) Impact of Type-I and Type-II Network Fault Events on Unwanted Agenda-Time

30





31

Conclusion

We proposed a Multi-Activity Packet Trains methodology Mimic participant interaction patterns and video activity levels as

affected by network fault events

MAPTs provide real-time objective measurements of Interaction QoE in a large-scale videoconferencing system Without requiring end-users, actual video sequences, VVoIP appliances

Defined new Interaction QoE Metrics Unwanted Agenda-Bandwidth, Unwanted Agenda-Time

Implemented MAPTs in an active measurement tool called Vperf and evaluated Interaction QoE measurements on a network testbed

32

Future Work

Our work is a first-step towards measuring how network fault events impact Interaction QoE in videoconferencing sessions

We considered basic participant interaction patterns and network fault event types Future scope could include several other participant interaction

patterns and network fault event types E.g. MAPTs for network fault events that cause lack of lip-sync

Human subject testing to more accurately map and validate network fault event types and participant interaction patterns

33

Thank you for your attention!☺

Any Questions?

Measuring Interaction QoE in Internet Videoconferencing Prasad Calyam (Presenter) Ohio Supercomputer Center, The Ohio State University Mark Haffner, Prof.

Documents

peak video

google video

vvoipstreaming qoe

enduser qoeenduser qoe

talking talking video

oneway voice

voice codecvoice

ip vvoip overviewnetwork