The Role of Sensory Psychology to VoIP Rate Adaptation : A Study on Skype Calls

Network and Systems Laboratorynslab.ee.ntu.edu.tw

The Role of Sensory Psychology to VoIP Rate Adaptation: A Study on Skype Calls

Skype Group, NSLABINFOCOMM2012(Hopefully)

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Tx/Rx ContentBitrateJitterPacket Loss RateQuality of

Service(QoS)

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Mean Opinion Score (MOS)

Reaction TimeReactivity/

ResponsivenessQuality of Experience

(QoE)

Network and Systems Laboratorynslab.ee.ntu.edu.tw

QoEMOSReaction

TimeReactivity

QoSTx/Rx

ContentBitrateJitterPacket

Loss

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Related WorksOn the TCP-Friendliness of VoIP Traffic, Tian Bu et al., INFOCOM2006Disprove the conjecture that VoIP is not TCP-

Friendly after taking the user back-off mechanism into account.

User back-off: real time appswill drop out completely ifthe user perceived unacceptable quality due tonetwork congestion.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Related WorksQuantifying Skype User Satisfaction, K. T. Chen et al., SIGCOMM2006The User Satisfaction

Index(USI)Using traditional metrics

(RTT, jitter, bitrate) to infer user-centric metrics (reactivity, duration, MOS.)

Allow real-time and user-centric adaptation .

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Related WorksCould Skype be More Satisfying?, T. Y. Huang et al., IEEE Network 2010Skype’s adaptation does not take the

individual codec and packet loss patterns into consideration.

The inconsistency in voice quality results in over-utilization of bandwidth.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Related WorksAn Experimental Investigation of the CongestionControl Used by Skype, L. D. Cicco et al., WWIC 2007Skype’s slow adaptation to bandwidth drop

causes coexisting TCP flows to be suppressed.

Skype’s over-utilization of bandwidth causes massive fluctuation on bitrate, which may result in user frustration.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

MotivationClearly, there are many to be improved on

Skype’s rate adaptation algorithm.Skype’s over-utilization of bandwidth is

1) wasting network resource and2) threating other applications at the risk of3) producing massive fluctuation on quality.

Our major assumption: This selfish deed of Skype is actually NOT helping user satisfaction. Users dislike changes on audio quality, even if they actually increase the average rate.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Roadmap

Preliminary

Experiments

Large-Scale

Experiments

Evaluation

Conclusion

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Roadmap

Preliminary

Experiments

Large-Scale Experiments

Evaluation

Conclusion

Network and Systems Laboratorynslab.ee.ntu.edu.tw

GoalConfirm our assumption about user’s

impression towards audio quality fluctuation.Get a ballpark idea of the possible

relationships between parameter and MOS. (formulation)

Network and Systems Laboratorynslab.ee.ntu.edu.tw

MethodExploit audio encoder/decoder to create

audio track with fluctuating qualities (bitrates.)

We will focus on Silk in all following experiments due to its1) potential of domination of VoIP codec and2) flexibility on fine-tuning bitrate.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Test Tracks

High rate

Low rate

Time

∆T ∆T

Bitrate

Network and Systems Laboratorynslab.ee.ntu.edu.tw

PCM

PCM

PCMPCM

Test Tracks SetupHeade

rPCM

Encoder

Encoder

Decoder

Decoder

High rate

Low rateCombin

e

∆T

Header

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: GoalWe target three variables, High Rate, Low

Rate, ∆T, that affect the user’s perception.Interactions between the three variables.Exp1: Find the relation between fixed bitrate

and MOS.

Exp2: Find the formula that combines the three dimensions with MOS.𝑓 𝐹𝐿𝑈𝐶 (h𝑟 , 𝑙𝑟 ,∆𝑇 )=𝑀𝑂𝑆

𝑓 𝐹𝐼𝑋 (𝑏𝑟 )=𝑀𝑂𝑆

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Test Tracks SetupThe maximum

and minimum bitrate of Silk are 40.6 and 5.6 kbps.

We chose 10 rates uniformly from the interval.

Experiment 1: Fixed Rate vs. MOSq1 40.6 kbpsq2 36.6 kbpsq3 32.8 kbpsq4 28.9 kbpsq5 25.0 kbpsq6 21.1 kbpsq7 17.2 kbpsq8 13.3 kbpsq9 9.5 kbpsq10 5.6 kbps

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Test Tracks SetupThe source track

is 30 seconds long. We set ∆T as its factors.

We picked 4 rates (q1, q4, q7, q10) to be the candidates of high and low rates.

HR LR

∆T10 sec5 sec3 sec2 sec1 sec

{40.6, 28.9, 17.2, 5.6} kbps

Experiment 2

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Test Tracks SetupFollows the ITU

recommendations.Four voices: 2 male and 2 female.Sentences with no coherent plot.30 seconds, 44.1 kbpsReference tracks (original 44.1 kbps) are

inserted in the test cases in order to provide a standard of rating.

The tracks of Exp1&2 are mixed up and the order of rating for each subject is randomly picked.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Results (Exp1)

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Analysis (Exp1)The plot can be fitted by a shifted logarithm

function.The shift is due to the lower boundary of

human audio perception.Observed rapid MOS drop with lower bitrate.

𝑓 𝐹𝐼𝑋 (𝑏𝑟 )=γ ×ln (𝑏𝑟 −𝛼 )+ 𝛽

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Why Logarithms?Weber–Fechner law

 The smallest noticeable difference in stimulus (the least difference that the test person can still perceive as a difference,) was proportional to the starting value.

The law is shown plausible in a wide range of human perceptions including hearing, vision, taste, sense of touch and heat, and even temporal and spatial cognitions.

𝑑𝑝=𝑘 𝑑𝑆𝑆 ❑

⇒𝑝=𝑘× ln (𝑆 )+𝐶

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Results (Exp2)

Adapting to an “optimal rate” and ignoring how users feel about changes might be over-optimistic.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Results (Exp2)

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Analysis (Exp2)R2 of logarithm regression of each track are

generally higher than 0.9.An outlier is discovered: 28.9+17.2. This is

attributed to:1) the similarity of the two bitrates and 2) they both reside in middle- or low-level qualities.

The phenomena is also supported by the ANOVA test on the similarity of 28.9 and 17.2 kbps data sets (p = 0.2155).

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Formulation: Analysis (Exp2)In short, the MOS to frequency of rate

change relationship, although shows logarithmic behavior in general, depends on the magnitude of rate changes.

𝑓 𝐹𝐿𝑈𝐶 (h𝑟 , 𝑙𝑟 ,∆𝑇 )=𝑆𝐶𝐴𝐿𝐸 (h𝑟 , 𝑙𝑟 )× ln (∆𝑇 )+𝑆𝐻𝐼𝐹𝑇 (h𝑟 , 𝑙𝑟 )

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Some Guessing About the SubroutinesSCALE()Directly associated with the

difference between hr and lr. The results in Fig. 7 provide evidence to this inference: same average bitrate, different magnitudes.

Positive correlation between the scale of regression function and rate change magnitude.

Another intention of SCALE() is to deal with small magnitude tracks that does not fit well.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Some Guessing About the SubroutinesSHIFT()Cope with human’s expectation. As ∆T grows, the effect of fluctuation

decreases and the variable-rate case will become indiscernible to a fixed-rate version.

We call this imaginary, fixed rate equivalent the dominant quality of the fluctuation. (dominant quality ≠ average quality)

The dominant quality is the exact quality a user expects to observe when the negative impact of fluctuation diminishes.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Roadmap

Preliminary

Experiments

Large-Scale Experiments

Evaluation

Conclusion

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Large-Scale Experiments: GoalWe need massive data to construct the detail

of our formulas:- verify the structures of our formulas.- factors in the fixed-rate formula:

- subroutines in the variable-rate formula: SCALE(hr,lr) & SHIFT(hr,lr)

𝑓 𝐹𝐼𝑋 (𝑏𝑟 )=γ ×ln (𝑏𝑟−𝛼 )+ 𝛽

Network and Systems Laboratorynslab.ee.ntu.edu.tw

MethodSame source track.Nine levels of quality are exponentially

chosen.Five levels of rate changing frequency

{1,2,3,5,10}.127 participants.Score calibration with hidden reference

track.ITU Recommendations

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Results: Formula Structure

Figural support:Non-parallel plots

Statistic support:ANOVA of interactivity (p=8e-14)

𝑓 𝐹𝐿𝑈𝐶 (h𝑟 , 𝑙𝑟 , ∆𝑇 )=𝑆𝐶𝐴𝐿𝐸 (h𝑟 , 𝑙𝑟 )× ln (∆𝑇 )+𝑆𝐻𝐼𝐹𝑇 (h𝑟 , 𝑙𝑟 )

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Results: Fixed-rate Formulaα=4.091β=1.515γ=1.000

Another interesting discovery: lower bound of Silk.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Results: SCALENot surprisingly, SCALE subroutine is

positively correlated with magnitude.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Results: SHIFTThis is more tricky… due its relationship with

user expectation.Base on our definition of dominant quality:

Where D(hr,lr) is the MOS of the dominant quality of rate changing pair: (hr,lr)

Network and Systems Laboratorynslab.ee.ntu.edu.tw

SHIFT (Conti.)First we plot the estimated MOS of fixed hr,

fixed lr, and D.There is an apparent difference when

hr<14.1.Not surprising, we

have already seen this reaction of MOS when a track ispaired by two similar, inferior rates.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

SHIFT (Conti.)We plot them again in

percentages:hr = 100%lr = 0%

We can then see a clear pattern when we group the tracks by their MOS magnitudes.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

SHIFT (Conti.)Finally…

𝑓 𝐹𝐿𝑈𝐶 (h𝑟 , 𝑙𝑟 ,∆𝑇 )=𝑆𝐶𝐴𝐿𝐸 (h𝑟 , 𝑙𝑟 )× ln (∆𝑇 )+𝑆𝐻𝐼𝐹𝑇 (h𝑟 , 𝑙𝑟 )

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Roadmap

Preliminary

Experiments

Large-Scale Experiments

Evaluation

Conclusion

Network and Systems Laboratorynslab.ee.ntu.edu.tw

EvaluationIt is not surprising that the formula outcomes

of preliminary and large-scale experiments fit their ground truth.

We need a third dataset for verifying purpose.

The Verifying ExperimentsDifferent source track: conversation of two

males.Different length: 60 secondsDifferent rates: {44.1, 11.8, 6.4} kpbsDifferent frequencies: {1,5,10} seconds

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Results

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Roadmap

Preliminary

Experiments

Large-Scale

Experiments

Evaluation

Conclusion

Network and Systems Laboratorynslab.ee.ntu.edu.tw

ConclusionVerified the user experience versus

magnitude of rate change relationship exhibits the log-like behavior, echoing the Weber’s theory.

Discovered that experience versus frequency of rate change relationship also exhibits the log-like behavior.

Derived the closed form model of user experience to rate changes with 97%+ goodness of fit.

Network and Systems Laboratorynslab.ee.ntu.edu.tw

Thanks For Your Attention