Video Quality Measurements

www.dsp-ip.comFast Forward Your Development

Video Quality Measurements


Overview

•QoS and QoE•Measurement Methods

▫Objective, Subjective, payload/codec based•Measurement and Monitoring Goals

▫Lab, pre-deployment, Monitoring, failure analysis

•Monitoring Levels▫Transport, Transaction, content

•Monitoring Domains ▫Head-end, network, Home


User Quality of Experience factors (QoE)

•What does subscribers want?▫Content

Content quality Content pricing Content availability

▫Viewing experience Video Quality Channel Zapping delay

▫Integration of services Usability One bill New services

▫Customer services

Technical Factors


MEASUREMENT TAXONOMYSubjectiveObjectivePayload based, codec aware, codec anaware


Measurement methods review

•Subjective▫Human assessment of quality▫Expensive, not for monitoring▫Not repeatable

•Objective▫Measurement devices▫Repeatable▫For both testing and monitoring, failure

analysis

ObjectiveSubjective

Measurements


Multimedia monitoring methods

Subjective

MOS(Voice)

BT500(Video)

Measurements

•Subjective measurement arte done by humans according to pre-defined protocols

•Voice – MOS▫Grade voice quality between 1-5▫4+ - is very good quality

•Video – BT500


Objective methods

Measurements

•Objective methods are divided into:•Payload based•Packet based

▫CODEC Aware ▫CODEC Independent

•Network Monitoring


Payload based Measurements•Payload base methods assess the video

quality based on the video itself•Reference based methods compares

Original video quality to distorted video.•Used mostly in lab equipment for codec

performance analysis and comparison


Full Reference Methods

PSNR

FullReference

Objective Payload based

MSE

SSIM

J.144

•Full reference methods compares each frame of the original video to frame of the distorted video and provide distortion measurement

•MSE & PSNR are pixel based similarity measurements

•Most video quality measurements are done on Luma (Y)


PSNR & MSE •PSNR and MSE is a pixel base video

quality comparison tools

MSE

LPSNR

2

10log10


MSE=0, MSSIM=1

MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723

original Image

PSNR/MSE Problems•Quality degradation is not

reflected by MSE/PSNR index


SSIM – “Solves” PSNR Problems•SSIM is Structural Similarity index.

Unlike PNR and MSE it does not compare images “pixel by pixel” but as small NxN “environments”

• are average variance and covariance

• the dynamic range of the pixel-values (typically )


ITU-T J.144 and ITU-R BT.1683 Full-reference perceptual models•Digital TV•Rec. 601 image resolution (PAL/NTSC)•Bit rates: 768 kbps ~ 5 Mbps•Compression errors

Full Reference testing standards


Back to Objective measurement

Measurements

•What happens when we don’t have the original (Reference video) or when we don’t have the processing power to do an extensive comparison?

•We could use network measurements and codec based degradation info to asses video quality


Packet – Codec Aware•Monitoring technique•Degrades video based on codec type by

incorporating network parameters data with codec behavior data

•Scales- could monitor thousands of channels

•Examples:▫ VQS (Telchemy)▫ VQI(Brix)▫ V-Factor (QoSMetrics)

0

5

10

15

20

25

30

35

0 5 10 15 20

Packet Loss (%)

PS

NR

(d

B)

Problem area

Robust codec

“Raw”codec

The need a codec aware metrics


Codec Aware Methods

Codec awarePacket based

VQI

V-Factor

VQSTelchemy

Objective methods


Example V-Factor

•Based on MPQM (Moving Picture Quality Metrics) – high quality video measurement standard

•V = f(QER, PLR, R)▫QER – relative video codec quality▫PLR – Packet loss ratio (based on actual

packet loss, jitter data and jitter buffer model)

▫R – Image complexity factor (2-3)•Adopted by Spirnet


Packet – Codec Independent

•Monitoring only•Codec independent•Based on network parameters data only•Scales - could monitor thousands of

channels•Examples:

▫MDI IneoQuest standardized by IETF


MEASUREMENT & MONITORINGIn the Lab & In The FieldsPre-Deployment/monitoring/Failure Analysis


Measurement & Monitoring phases

AnalysisProblem Solving

Tuning

PreDeployment

Testing

LabTesting

Design

24/7Monitoring

DeploymentDeploymentPhasePhase

Pre-Pre-DeploymentDeploymentPhasePhase


Measurement & Monitoring phases

•Design & Lab testing▫Simulation and Emulation of the network▫Lab and testing tools

•Pre Deployment Stage▫Work on actual network▫Load testing▫Lab, testing, diagnosing and monitoring tools

•Deployment (production) Phase▫Mostly monitoring (probes) equipment,

management systems, data filtering and diagnostics equipment


MONITORING LEVELS


Measurement Levels

•Transport Level•Service (transaction) Level•Media Quality Level

Video Quality

Transport Quality

Transaction Quality

Author

Add refrence to capex and opex expense


Transaction Level•Examples

▫Post-dial delay in PSTN/mobile networks▫Video start time for channel zapping &

Video conf•Requires understanding in both network

monitoring and signaling (IGMP, SIP) and in media coding (analysis of the media to discover dial tone or I frame)


Channel zapping delay

•Multicast saves bandwidth but creates signaling delays:▫Multicast Leave +▫Multicast Join + ▫First I Frame +▫Up to 2 seconds buffering time

Leave latency

Join latency

Signaling Latency

First I Frame

Media Latency

Total Channel zapping Latency

Buffering latency

First frame viewed


Transport Level

•Example: ▫Packet Loss▫Loss Patterns▫Jitter▫Delay

•Well understood•Defined by ITU and IETF


Content Level• Content quality is a payload based measurement.

▫ Requires decoding of the video stream▫ Understanding of the buffering and error concealment

algorithms of the decoder • CPU intensive – Does not scale• Accurate• Used mostly is Lab equipment and diagnostic

equipment• Examples:

▫ PSNR ▫ ITU-T J.144

• Usually requires the reference (original) stream• Tests:

▫ Source artifacts▫ Source quality


Standardization landscape

DSL

DSL Forum

TR-64, TR-69LAN and WAN monitoring standards

ITU

Study Group 12

Algorithms for end-to-end transmission performance

ITU

VQEG – Video Quality Expert Group

Video performance measurement based on Subjective tests Database

Used in Diagnostics / Lab

ATIS

IIF – IPTV Interoperability Forum

QoS Metrics Standardization

Used for Monitoring


Example: ATIS IIF Quality Metrics

VSTQ - Video Service Transmission Quality ▫ Transmission Quality - codec/ content independent▫ Based on the rate and distribution of effective packet loss and

discardVSPQ - Video Service Picture Quality

▫ Estimated viewing quality▫ Considers the impact of VSTQ, video codec type and rate,

resolutionVSAQ - Video Service Audio Quality

▫ audio listening quality▫ Considers the impact of VSTQ, audio codec type, sample rate, …..

VSMQ - Video Service Multimedia Quality ▫ overall user experience▫ Combined effect of VSPQ, VSAQ, audio-video synchronization..

VSCQ - Video Service Control (Plane) Quality ▫ Considers responsiveness and reliability of control plane (trick

play)


Monitoring levels• J.144 and PSNR examines the video content only (payload

measurements)• TR101290 examines only transport stream data and

coherence without examining the video content• V -Factor and VQS looks at packet loss, jitter and loss patterns

data and incorporate it with codec information and video header information

• MDI – Examines only packet loss and packet loss patterns without considering the codec or video information

TR101290MPEG2TS Headers

V-Factor, VQSMDI

J.144, PSNRVideo payload

Video Quality Measurements

Technology

original video quality

based methods

distorted video

codec independent based

original reference video

monitoring goals lab

network measurements

codec type