Metrics for Evaluating Video Streaming Quality in Lossy IEEE 802.11 Wireless Networks Dept. of Mobile Systems Engineering Junghoon Kim
Metrics for Evaluating Video Streaming Qualityin Lossy IEEE 802.11 Wireless Networks
Dept. of Mobile Systems EngineeringJunghoon Kim
OutlinePaper InfoIntroductionBackgroundMotivationIdeaExperimentsEvaluationContribution
Paper InfoIEEE INFOCOM 2010
The 29th conference on computer communica-tions sponsored by IEEE communications society
March 15-19, 2010, San Diego, CA, USAAuthors
An (Jack) Chan, Kai Zeng, Prasant Mohapatra Dept. of Computer Science University of California, Davis
Sung-Ju Lee, Sujata Banerjee Multimedia Communication & Networking Lab Hewlett-Packard Labs
IntroductionImportant issue
Multimedia streaming is becoming one of the most popular applications recently
Video streaming over WLANs in very commonVideo quality can be measured objectively and
automatically by a computer program It is important to government and industries For specification of system performance require-
ments Comparison of competing service offerings
IntroductionPeak Signal-to-Noise Ratio (PSNR)
simplest and the most widely used video quality evaluation methodology
Problem of traditional PSNRFail to capture the packet loss characteristics of
wireless networksNon-linearity of the human visual system
MPSNR (Modification of PSNR)Retaining the simplicity of PSNR calculationHandles video frame losses
IntroductionDeriving two specific objective video quality
metricsPSNR-based Objective MOS (POMOS)Rates-based Objective MOS (ROMOS)Demonstrate high correlation with MOS
Our metrics evaluate video streaming quality in wireless networks with a much higher accu-racy
BackgroundMean Opinion Score (MOS)
Measured through each viewers giving a score ranging from one to five
Arithmetic mean of all these individual scoresPros
MOS is subjective metricCons
Expensive process Needs a large number of viewers Controlled evaluation environments
BackgroundPeak Signal-to-Noise Ratio (PSNR)
Most widely used objective video quality metric
MSE : Mean Squared Error
BackgroundPeak Signal-to-Noise Ratio (PSNR)
Problem A missing frame results in the latter frames in
shifted positions when compared with the reference video
MotivationInaccuracy in the existing PSNR calculation
Average PSNR value of the reference video : 100dB
Video streaming A : 38dBVideo streaming B : 40dB
(a) Reference video (b) Video streaming A
(c) Video streaming B
IdeaMPSNR
Modification of PSNRAdd matching process in the correct PSNR cal-
culationTwo ways
An optimized algorithm for matching corresponding frames
A heuristic algorithm for matching corresponding frames
IdeaAn optimized algorithm
Assumption The sum of PSNR of all frames in a streamed video is
the maximum when all the frames are correctly matched with the corresponding frames in the refer-ence video
Each frame in a streamed video must have a matched frame in the reference video
We consider a global maximization of the sum of PSNR
IdeaAn optimized algorithm (Cont’d)
Define Maximum total PSNR value achieved when a
streamed video with j frames is matched to the ref-erence video with i frames
Define PSNR value of frame x and frame y
If no match can be found for a frame in the ref-erence video, we ignore the frame in the calcu-lation of the total PSNR value
IdeaAn optimized algorithm (Cont’d)
Three possible cases for the last match in two videos But, Case 3 would never happen
Recurrence equation
IdeaAn optimized algorithm (Cont’d)
Use dynamic programming! Time complexity :
g : the total number of frames lost during streaming n : the number of frames in the streamed video
Given a streamed video of 40 seconds (1000 frames) with 20 frames lost (about 2% frame loss rate), a personal computer with 2.8GHz CPU and 1GB RAM Traditional PSNR : less than 2 seconds Optimized algorithm : about 20 seconds
We need a faster algorithm!!!
IdeaA heuristic algorithm
Define The PSNR value calculated for frame j in the
streamed video when it is compared with frame i in the reference video
Define The set containing the continuous frames in the ref-
erence video when frame j in the streamed video is processed
Define The PSNR value of the frame j in the streamed video
IdeaA heuristic algorithm (Cont’d)
A parameter called PSNR threshold, thresh To mitigate this problem
Frame j in the streamed video is distorted severely and has a larger similarity to a non-corresponding frame k than to the actual corresponding frame h
Take the maximum only if it is greater than thresh Otherwise, we will regard the first frame in as the
matched frame
IdeaA heuristic algorithm (Cont’d)
Time complexity : t : the number of different thresh tried w : window size n : the total number of frames in the streamed video
t and w are small constants. Therefore, time complexity is
Previous experiment Traditional PSNR : less than 2 seconds Heuristic algorithm : about 4 seconds
IdeaMeasuring other parameters
Distorted frame rate Averaged PSNR of distorted frames Frame loss rate
ExperimentsCollecting videos of dif-
ferent qualityA total of 40 streamed
videos with different qualities 30 video clips in the
training set 10 video clips in the vali-
dation set
(a) Streaming with intra-flow interfer-ence
(b) Streaming with inter-flow interfer-ence
(c) Streaming with background data flow
ExperimentsCollecting subjective evaluation for video
qualityEngaged 21 volunteers
Diversity was taken into account Age : from 20 to 45 Occupation : from university undergraduate students
to laboratory techniciansFor each video clip, average the quality scores
given by the subjects and obtain MOS
ExperimentsCollecting subjective evaluation for video
quality
MOS and 95% confidence intervals of videos in the train-ing set
ExperimentsDeriving metrics from subjective evaluation
and MPSNRPSNR-based Objective MOS (POMOS)
Define The average PSNR calculated from MPSNR
Define By setting the window size to one
ExperimentsDeriving metrics from subjective evaluation
and MPSNR (Cont’d)PSNR-based Objective MOS (POMOS)
Use the linear model package of the statistics tool R
ExperimentsDeriving metrics from subjective evaluation
and MPSNR (Cont’d)Rates-based Objective MOS (ROMOS)
To mitigate this problem Assigned a PSNR of 100dB for the perfect frames
ExperimentsDeriving metrics from subjective evaluation
and MPSNR (Cont’d)Rates-based Objective MOS (ROMOS)
Use the linear model package of the statistics tool R
EvaluationEvaluation of objective metrics
Pearson correlation (= correlation coefficient) A heuristic algorithm
: 0.8666 : 0.9346
An optimized algorithm : 0.8838 : 0.9509
EvaluationEvaluation of objective metrics
ContributionIdentify the detrimental impact of packet
losses during video streaming on video quality metric, such as PSNR
Propose a simple objective video quality eval-uation methodology, MPSNR, that alleviates the inaccuracy caused by packet loss
Derive two specific video quality metrics that provide a tool for evaluating video streaming over lossy wireless networks