1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang
Mar 31, 2015
1
A Quest for an Internet VideoQuality-of-Experience Metric
Athula Balachandran, Vyas Sekar,Aditya Akella, Srinivasan Seshan,
Ion Stoica, Hui Zhang
2
Internet Video is taking off
Improve Users’ Quality of Experience
3
Video Quality Metrics: The State of the Art
Objective Score (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
4
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
5
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
6
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
7
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
8
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
9
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
10
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
Join Time Buffering RatioRate of buffering
Rate of switchingAverage bitrate
11
Problem 2: Opinion Scores Engagement
Opinion Scores- Not representative of “in the wild”
experience- Combinatorial explosion of parameters
Engagement as replacement for opinion score. (e.g., Play time, customer return rate)
12
Internet Video QoE
Objective ScoresPSNR
Subjective ScoresMOS
13
Internet Video QoE
Objective ScoresPSNR
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
14
Internet Video QoE
Objective ScoresPSNR
Join Time, Avg. bitrate, …?
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
15
Internet Video QoE
Objective ScoresPSNR
Join Time, Avg. bitrate, …?f(Join Time, Avg. bitrate, …)
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
16
Internet Video QoE
Objective ScoresPSNR
Join Time, Avg. bitrate, …?f(Join Time, Avg. bitrate, …)
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
17
Outline
• Need for a unified QoE
• What makes this hard?
• Our proposed approach
18
Challenge: Complex Engagement-to-metric Relationships
Enga
gem
ent
Quality Metric
[Dobrian et al. Sigcomm 2011] 19
Challenge: Complex Engagement-to-metric Relationships
Enga
gem
ent
Quality Metric
Non-monotonic
E
ngag
emen
t
Average bitrate
[Dobrian et al. Sigcomm 2011] 20
Challenge: Complex Engagement-to-metric Relationships
Enga
gem
ent
Quality Metric
Non-monotonic
E
ngag
emen
t
Average bitrate
En
gage
men
t
Rate of switching
Threshold
21
Challenge: Complex Metric Interdependencies
Join Time Bitrate
Rate of buffering
Rate of switching
Buffering Ratio
22
Challenge: Complex Metric Interdependencies
Join Time Bitrate
Rate of buffering
Rate of switching
Buffering Ratio
23
Challenge: Complex Metric Interdependencies
Join Time
Rate of buffering
Rate of switching
Buffering Ratio
Bitrate
24
Challenge: Complex Metric Interdependencies
Join Time Avg. bitrate
Rate of buffering
Rate of switching
Buffering Ratio
25
Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies
26
Casting as a Learning Problem
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies
27
Impact of the ML algorithm• Classify engagement into uniform classes• Accuracy = # of accurate predictions/ # of cases
ML algorithm must be expressive enough to handle the complex relationships and interdependencies
28
Challenge: Confounding Factors
Live and VOD sessions experience similar quality
29
Challenge: Confounding Factors
However, user viewing behavior is very different
30
Challenge: Confounding Factors
Devices User InterestConnectivity
Need systematic approach to identify and handle confounding factors
31
Domain-specific Refinement
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
32
Domain-specific Refinement
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
Confounding Factors
33
Improved prediction accuracy
Refined ML models can handle confounding factors
34
Concluding Remarks• Internet Video needs unified quantitative QoE
• What makes this hard?– Complex engagement-to-metric relationships– Complex metric-to-metric interdependencies– Confounding factors (e.g., genre, device)
• Promising start– Machine learning + domain-specific refinements
• Open Challenges– Coverage over confounding factors– System Design