Spring 2020: Venu: Haag 315, Time: M/W 4-5:15pm ECE 5578 Multimedia Communication Lec 10a: Quality of Experience (QoE) Zhu Li Dept of CSEE, UMKC Office: FH560E, Email: [email protected], Ph: x 2346. http://l.web.umkc.edu/lizhu Z. Li: ECE 5578 Multimedia Comm, 2020 p.1 slides created with WPS Office Linux and EqualX LaTex equation editor
48
Embed
ECE 5578 Multimedia Communication Lec 10a: Quality of ......QoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS ... WebRTC, and QUIC Congestion Measure and Modeling
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Spring 2020: Venu: Haag 315, Time: M/W 4-5:15pm
ECE 5578 Multimedia Communication
Lec 10a: Quality of Experience (QoE)
Zhu LiDept of CSEE, UMKC
Office: FH560E, Email: [email protected], Ph: x 2346.http://l.web.umkc.edu/lizhu
Z. Li: ECE 5578 Multimedia Comm, 2020 p.1
slides created with WPS Office Linux and EqualX LaTex equation editor
Outline
QoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics
Summary
Z. Li: ECE 5578 Multimedia Comm, 2020 p.2
Video Communication System Coverage
Tentative Topics: QoE Metrics: Referenced, Light Reference, and Reference-less QoE
metrics MPEG Systems: File Format (MP4Box), Streaming Solution
(DASH.js), MMT Media Transport: RTP/RTSP, HTTP/WebSocket, WebRTC, and QUIC Congestion Measure and Modeling in Media Networking Video over Multiple Access Networks : Resource Pricing Solution,
DP+Lagrangian Framework FEC P2P Systems Content Identification and Info Centric Networking
Z. Li: ECE 5578 Multimedia Comm, 2020 p.3
QoS
QoS – Quality of Service A network centric metric Measuring the delay, loss, throughput, ..etc Does not directly translate into user experiences Typically characterized by the packet arrival and departure curves Buffer size: b(A,D, t), delay, d(A,D, t)
Z. Li: ECE 5578 Multimedia Comm, 2020 p.4
QoE
Quality of Experience A user centric metric, how a piece of audio/visual signal delivered by
the network looks/feels Usually a function of QoS, content, and viewing conditions
Fig credit: Touradj Ebrahimi, EPFL, ACM MM 2009
Z. Li: ECE 5578 Multimedia Comm, 2020 p.5
ITU STRUCTURE
3 Sectors:
• Standardization (ITU-T): promotes enabling technical, policy and regulatory frameworks to boost ICT development
• Radiocommunication (ITU-R): coordinates the shared global use of radio spectrum and geostationary satellite orbit
• Development (ITU-D): works to improve telecommunication infrastructure in the developing world
Z. Li: ECE 5578 Multimedia Comm, 2020 p.6
ITU-T Structure and organization
WTSA
TSAG
Study Group x
Working Party 1/x
Question 1/1
Working Party 2/x
Question 1/2
Working Party 3/x
Question 1/3
Study Group y
Working Party 1/y
Question 1/1
Working Parties …
Study Groups …
Z. Li: ECE 5578 Multimedia Comm, 2020 p.7
Study Group 9 OverviewLead Study Group on integrated broadband cable and
television networksResponsible for studies relating to: use of telecommunication systems for contribution, primary
distribution and secondary distribution of television, sound programmes and related data services including interactive services. use of cable and hybrid networks, primarily designed for television
and sound programme delivery to the home, as integrated broadband networks to also carry voice or other time-critical services, video on demand, interactive services, etc.
Z. Li: ECE 5578 Multimedia Comm, 2020 p.8
SG9 QoE metrics work
Rec. # Name Qu Title Timing
J.249 J.redref Q2/9 Perceptual video quality measurement techniques for digital cable television in the presence of a reduced reference
2009
J.340 J.ra-psnr Q2/9 Reference Algorithm for Computing Peak Signal to Noise Ratio (PSNR) of a Video Sequence with Constant Spatial Shifts and a Constant Delay
2009
J.341 J.vqhdtv-fr Q2/9 Objective perceptual multimedia video quality measurement of HDTV for digital cable television in the presence of a full reference
2010
J.bitvqm J.bitvqm Q12/9 Hybrid perceptual bitstream video quality assessment
2013
J.av-dist J.av-dist Q12/9 Methods for subjectively assessing audiovisual quality of internet video and distribution quality television, including separate assessment of video quality and audio quality
2013
J.3D-disp-req
J.3D-disp-req
Q2/9 Display requirements for 3D video quality assessment
2013
Z. Li: ECE 5578 Multimedia Comm, 2020 p.9
Study Group 12 Overview
‘Performance, QoS and QoE’Responsible for Recommendations on performance, quality of
service (QoS) and quality of experience (QoE) for the full spectrum of terminals, networks and services ranging from speech over fixed circuit-based networks to multimedia applications over networks that are mobile and packet based. Included in this scope are the operational aspects of performance, QoS and QoE.
A special focus is given to interoperability to ensure end-to-end users' satisfaction.
SG 12 is the Lead SG on QoS and Performance
Z. Li: ECE 5578 Multimedia Comm, 2020 p.10
SG12 Visual Quality Assessment
Rec. # Name Qu Title Timing
P.1201 P.NAMS Q14/12 Parametric non-intrusive assessment of audiovisual media streaming quality
2012-09
P.1202 P.NBAMS Q14/12 Parametric non-intrusive bitstream assessment of video media streaming quality
2012-09
G.1080 G.IPTV-QoE
Q13/12 Quality of experience requirements for IPTV services
2008
G.1050 Q13/12 Network model for evaluating multimedia transmission performance over the Internet Protocol
2011
G.OMVAS G.OMVAS Q13/12 Opinion model for video and audio streaming applications
2014
P.1401 P.STAT Q9/12 Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models
2012
Z. Li: ECE 5578 Multimedia Comm, 2020 p.11
Video Quality Experts GroupFounded 1997 ITU-T SG 12, SG 9, and ITU-R 11E (now 6C) experts Web ( www.vqeg.org );
First VQEG meeting (Turin, Italy 1997)
Primary mission: Advance the field of video quality
assessment by investigating new and advanced subjective and objective measurement techniques
VQEG does not develop or publish standards Conducts tests and reports results to
ITU and other standards organizations Tests are conducted using specifically
defined procedures (i.e., carefully developed test plans).
Z. Li: ECE 5578 Multimedia Comm, 2020 p.12
VQEG ProjectsCompleted
FRTV I & II (5 ITU Recommendations) Multimedia I (7 ITU Recommendations) RRNR (3 ITU Recommendations) HDTV I (2 ITU Recommendations)
Active: 3DTV (3 ITU Recommendations in progress) Joint Effort Group—JEG-Hybrid Hybrid Perceptual/Bitstream (3 ITU Recommendations in progress) Multimedia II—MM2 (1 ITU Recommendation in progress) Quality for Recognition Tasks—QART (Public Safety, Surveillance
Applications) (1 ITU Recommendation)Ramping up
High Dynamic Range Video—HDR HDTV Phase II—HDTV2 Monitoring of Audio Visual Quality by Key Indicators—MOAVI Real-Time Interactive Communications Evaluation—RICE
About the Project Sign UpQoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics
Summary
Z. Li: ECE 5578 Multimedia Comm, 2020 p.16
QoE Subjective Evaluation
MOS – Mean Opinion Score, an user study based quality evaluation
A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices:• Test/lab environment• Test material• Test methodology• Analysis of the data
credit: Touradj Ebrahimi, EPFL, ACM MM 2009
Z. Li: ECE 5578 Multimedia Comm, 2020 p.17
Test/lab environmentType of Monitors/Speakers and other test equipments
Lighting /Acoustic conditions
Laboratory architecture, background, …
Viewing distance /Hearing position
…
Z. Li: ECE 5578 Multimedia Comm, 2020 p.18
Test material
Meaningful content for the envisaged scenario/application Typical content Worst case content …
p01 p06 p10 bike cafe woman
Z. Li: ECE 5578 Multimedia Comm, 2020 p.19
Test methodology (I)
Single Stimulus (SS)
Non-categorical adjectival or numerical grading scale
Single Stimulus Continuous Quality Evaluation (SSCQE)
(Very annoying)
(Imperceptible)
Z. Li: ECE 5578 Multimedia Comm, 2020 p.24
Test methodology (VI)
Simultaneous Double Stimulus for Continuous Evaluation (SDSCE)
(Much better)
(Much worse)(Reference) (Test sequence)
Z. Li: ECE 5578 Multimedia Comm, 2020 p.25
Analysis of the MOS data
Improve MOS data quality• Scores distributions across subjects (testing people) is assumed
to be close to normal distribution• Outlier detection and removal• Mean Opinion Scores (MOS) and 95% confidence intervals (CIj)
Nm
MOSN
i ijjå == 1
NNtCI j
j
sa ×-= ),2/1(
mij = score by subject i for test condition j.
N = number of subjects after outliers removal.
t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 Degrees of Freedom (DoF) and a desired significance level α (α=0.05 in our case, 95% confidence).
σ j = s t a n d a r d d e v i a t i o n o f t h e s c o r e s distribution across subjects for test condition j.
Z. Li: ECE 5578 Multimedia Comm, 2020 p.26
NN
MOSMOStBA
BAobs 22 ss +
-=
What is behind a MOS?
JPEG Image Quality Assessment Study:
Z. Li: ECE 5578 Multimedia Comm, 2020 p.27
Relationship between estimated mean values
• Hypothesis test to find out whether the difference between two MOS values are statistically significant
Two-sided t-test:
• T-statistic:
• Decision rule to reject H0:
BA MOSMOSH =:0
BAa MOSMOSH ¹:
NN
MOSMOStBA
BAobs 22 ss
+
-=
),2/1(),2/( NttORNtt obsobs aa -><
Z. Li: ECE 5578 Multimedia Comm, 2020 p.28
MOS hypothesis test
JPEG
200
0 4:
2:0
JPEG
20
00
4:4:
4JP
EG
JPEG
XR
MS
JPEG
XR
PSJP
EG 2
000
4:2:
0JP
EG
2000
4:
4:4
JPEG
JPEG
XR
MS
JPEG
XR
PSJP
EG 2
000
4:2:
0JP
EG
2000
4:
4:4
JPEG
JPEG
XR
MS
JPEG
XR
PS
JPEG 2000 4:2:0
JPEG 2000 4:4:4JPEG
JPEG XR MS
JPEG XR PS
JPEG 2000 4:2:0
JPEG 2000 4:4:4JPEG
JPEG XR MS
JPEG XR PS
0.25 bpp
0.50 bpp
0.75 bpp
1.00 bpp
1.25 bpp
1.50 bpp
6
5
4
3
2
1
0Number of times
H0 is rejected
Z. Li: ECE 5578 Multimedia Comm, 2020 p.29
Outline
About the Project Sign UpQoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics
Summary
Z. Li: ECE 5578 Multimedia Comm, 2020 p.30
Objective QoE metrics
• Subjective tests are time consuming, expensive, and difficult to design
• Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired• Full reference metrics
• No reference metrics
• Reduced reference metrics
Input/Reference signal
Output/Processed signal
signalprocessing
FR METRIC
Input/Reference signal
Output/Processed signal
signalprocessing
Input/Reference signal
Output/Processed signal
signalprocessing
Features extraction
RR METRIC
Z. Li: ECE 5578 Multimedia Comm, 2020 p.31
PSNR - Peak Signal to Noise Ratio
PSNR def:
Widely used because of its simplicity and ease in formalizing optimization problems!
For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation
åå= =
-=M
1y
N
1x
2ba y)](x,Imy)(x,[Im
MN1MSE
where:
M, N = image dimensions Ima , Imb = pictures to compare B= bit depth
Z. Li: ECE 5578 Multimedia Comm, 2020 p.32
PSNR for color images/video
Multiple channel info, several options to compute metric Weighted PSNR
The SSIM system: Image is a product of illuminance and object reflectance Try to separate the object structural info from the illuminance Full reference solution, compare image block x with y, have 3
components:o Luminance, contrast, and structure comparison
Z. Li: ECE 5578 Multimedia Comm, 2020 p.37
[1]
[2]
[3]
�(�,�)
�(�,�)
�(�,�)
Luminance Comparison – l(x,y)
Basic Operations Operate on image regions (can be block, or circular) For each channel, compute the region mean and variance for block x
and its reference y:
Luminance comparison function: L=dynamic range, 2B, e.g, 256 for B=8 K1, small const << 1.
Z. Li: ECE 5578 Multimedia Comm, 2020 p.38
�� =1� �
�=1
��� �� =
1� �
�=1
���
�(�, �) =2���� + (���)�
��� +��
� + (���)
Contrast Comparison – c(x,y)
Compare the illuminance dynamic range and behavior of two blocks Based on the variance of the channel
Validation with MOS Scores SSIM is a better predictor than PSNR
Z. Li: ECE 5578 Multimedia Comm, 2020 p.47
Summary
QoE is an important component in the multimedia communication system Subjective QoE study: User study generate MOS scores Objective Metrics: compare communicated content as pieces of
signals Perceptive Metrics: try to model HVS and have a better approximation
of MOS
Next Class: Reduced Reference, Non-Reference QoE metrics