Measuring Quality of Experience for Successful IPTV Deployments Dr. Stefan Winkler.

Measuring Quality of Experience for Successful IPTV Deployments

Dr. Stefan Winkler

Outline

• Challenges– Digital Video Quality Issues– Traditional Measurements (QoS) vs.

Quality of Experience (QoE)

• Possible Solutions– QoE Measurement Approaches– End-to-end QoE Management

• Conclusions

Digital Video Challenges

Demanding traffic profiles High bandwidth streams

High traffic volumes

Live, VOD

Network effects Video impacted heavily with

minor network impairments

Multi-vendor network complicates diagnosis / troubleshooting

High end-user expectations Defined with decades of history

Grow rapidly with HD

Low tolerance for poor quality

New architectures Sensitive video processing

devices create possibility for various impairment sources

Ad-insertion, middleware

Service quality degradations

Difficult diagnosis, troubleshooting

Rising management and

OPEX costs

Higher customer churn

What Drives End-Users

Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com

Service Providers View

Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com

Service Providers’ View

7Source: MRG 2007 IPTV Video Quality Survey, available at http://qoe.symmetricom.com

Sources of Video Issues

Consider all elements for true end-to-end solution

Compression Artifacts

Original MPEG-2 H.264

PSNR vs. QoE

Same amount of distortion (PSNR) – different perceived quality

Understand & model human vision system

QoS vs. QoE

• Quality of Service– Network-centric– Delay, packet loss, jitter – Transmission quality– Content agnostic

• Quality of Experience– Content impairments – Blockiness, Jerkiness, …– End-user quality– Application driven

QoS QoE

Same network impairmentsPacket Loss: 1%

Delay: 10msJitter: 50us

Bandwidth: 500kbps

Different perceived quality!

QoS vs. QoE

MDI vs. QoE

• Media Delivery Index (MDI)• MDI consists of two metrics:

– Delay Factor (DF)– Media Loss Rate (MLR)

• MDI limitations:– MDI assumes constant bit rate (CBR) traffic– MDI does not consider video payload or content– MDI values are not intuitive– MDI doesn’t correlate with video quality

MDI vs. QoE

0

1

2

3

4

5

V-F

AC

TO

R

0

500

1000

1500

2000

2500

3000

Med

ia L

oss

Jitter1 to 50ms

Packet Drop1 in 500 periodic

Packet Drop1 in 500 poisson

Packet Drop1 in 500 uniform

Packet Drop1 in 500 and Jitter

Duplicate Packets1 in 10

Duplicate Packets1 in 500

Reordering Packets1 in 500

Med

ia L

oss

MO

S

QoS/QoE Cycle

Desired QoE

Perceived QoE

Targeted QoS

DeliveredQoS

End-userService

provider

Alignment gap

Perception gap

Value gap Execution gap

Adapted from ITU-T Rec. G.1000 and COM12–C185–E

Outline

• Challenges– Digital Video Quality Issues– Traditional Measurements (QoS) vs.

Quality of Experience (QoE)

• Possible Solutions– QoE Measurement Approaches– End-to-end QoE Management

• Conclusions

Full-Reference Approach

• Comparison of individual video frames• Offline analysis (capture is required) – lab applications• High detail and accuracy• Alignment procedure

Sender Receiver

Full reference information

Full Ref. QualityMeasurement

No-Reference Approach

• Non-intrusive, in-service measurement• Real-time monitoring applications• No alignment required

Sender Receiver

No-Ref. QualityMeasurement

• Monitoring applications• Correlation of content and network impairments• Encrypted environments

Sender Receiver

Reduced Ref.Measurement

Feature Extraction

Feature Extraction

Reduced-Reference Approach

Content & Network Metrics

20

"Vision is the most highly developed of the human senses, so people are even more sensitive to flaws in video images than, say, the sound of a telephone conversation.”

Ken Wirt, Cisco Vice President Consumer Marketing, Jan 2008

(Correlation Engine)

• Contrast perception– Visibility of different patterns– Frequency dependencies

• Masking effects– Interaction of content and impairments– Texture, edges, luminance– Spatial and temporal masking

• Color perception

Spatial frequency [cpd]Temporal frequency [Hz]

Sen

sitiv

ity

Vision Modeling

Masker contrast

Visibility threshold

Targ

et

contr

ast

Maskingcurve

Thresholdwithoutmasker

End-to-end QoE

Deep Content Analysis(pixel by pixel)

Source content and encoder / transcoder validation

Human Vision System Model

VideoQualityReports

Content Impairments:• Blockiness, blur• Jerkiness• Freeze/black frame• Noise, Color

Network Impairments:• Loss• Delay• Jitter• Bandwidth

Content Stream Analysis:• PES inspection• PCR jitter etc.

Deep Content Analysis (bitstream)

Detect content impairmentsDeep inspection to associate content to timestamps (eg: TS1 = I-Frame) Network (header or stream) Analysis

Detect QoS issuesContent analysis where possible (unencrypted)Inspection of QoS to associate timestamps to impairments (eg: TS1 = Packet Loss)

Q-Advisor

Correlation Engine

TS1 = I-Frame

TS1 = Packet Loss

Packet Loss -> I-Frame

IPTV QoE Management

25

Issue Possible Causes

Blockiness Encoder Transcoder Network Loss

Blur Camera (focus) Encoder Transcoder STB (bad filtering)

Freeze Frame, Jerkiness Encoder (dropped frames)

Network loss Bad synchronization

Black Screen, Blue Screen

No Video Signal (source)

Ads not inserted Major network loss

Color Encoder Camera Transcoder

Video Noise (analog noise)

Camera STB

Noise (digital) Encoder Transcoder

Audio Microphone Encoder (bad mono stereo encoding

Encoder (lip sync) STB

1.0

1. Understand the Service Is there an issue?Does it matter?

1. Understand the Service Is there an issue?Does it matter?

2. Understand the ProblemWhat does the customer see?What is the exact cause?

2. Understand the ProblemWhat does the customer see?What is the exact cause?

3. Understand the SolutionWhat is the impairment source?

3. Understand the SolutionWhat is the impairment source?

1

2

3

4

5

Very Annoying

Annoying

Slightly Annoying

Perceptible

Imperceptible

Conclusions

• QoE is application-driven– Measure both network and content impairments

• QoE is user-oriented– Measure how end-user perceives service issues

• End-to-end quality measurement– Cover different impairment sources– Identify problem causes

Stefan [email protected]

Company:qoe.symmetricom.com

Further Reading:stefan.winkler.net/book.html

Contact Info