IMPLEMENTATION AND ANALYSIS OF USER ADAPTIVE MOBILE VIDEO STREAMING USING MPEG- DASH Abhijith Jagannath Department of Electrical Engineering University.

1

IMPLEMENTATION AND ANALYSIS OF USER ADAPTIVE MOBILE VIDEO STREAMING USING MPEG-

DASH

Abhijith JagannathDepartment of Electrical Engineering

University of Texas at ArlingtonAdvisor: Dr. K. R. Rao

Committee: Dr. W Alan Davis and Dr. Jonathan Bredow

2

Overview• Need for bandwidth savings• Factors affecting visual perception of mobile video• Some concepts on limits of human vision• Derivation of Maximum visible frequency• Calculation of sufficient resolution• Adaptive video streaming• MPEG-DASH Overview• User environment estimate from device sensors• Implementation of user adaptive video streaming using MPEG-

DASH• Experimental results• Conclusions

3

Increasing need for bandwidth savings

• Mobile Internet use is expanding dramatically• Video traffic is growing exponentially • Challenges:

– Mobile users expect high quality video experience– Network operators need to offer quality experience affordably

4

Ambient Light

Factors affecting visual perception of mobile videos

• Display Characteristics– Brightness– Contrast– Physical size– Pixel depth

• Viewing angle• Distance• Reflected ambient light

5

Visual Acuity

– Snellen’s Chart– Person with normal eye sight can see

details of 20/20 letter from 20 feet.

6 meters / 20 feet viewing distance

6/6 (20/20) – row letters are designed such that when viewed from 6 meters their smallest spatial details (strokes, gaps) constitute 1 minute of arc (1/60 of a degree of viewing angle)

6

Spatial Frequency

• cycles per degree

𝑢 = 1𝛽, 𝛽 = 𝜋360arctan൬𝑛2𝑑𝜌൰

n: # of pixels, p : pixel density

8

Contrast

• Ratio between brightest (typically white) to darkest (typically black) colors of a display.

• Michelson contrast

• Contrast sensitivity

maxL

minL

9

Effects of Ambient light

10

𝐿𝑚𝑖𝑛𝐷 ≤ 𝐿𝑚𝑖𝑛≤ 𝐿𝑚𝑎𝑥≤ 𝐿𝑚𝑎𝑥

𝐷

• Reflected Luminance :

• Effective Luminance of display

• Ambient contrast ratio

• Effective display contrast

Ambient light (lux)

Disp

lay co

ntras

t rati

o (

CRA

)

Effects of Ambient light

𝑠𝑚𝑖𝑛=1

𝐶𝐷

=CRA+1

CRA−1

Minimum contrast sensitivity

11

Contrast Sensitivity Function (CSF)• CSF is measured by projecting small (2-12o) Gabor patches on screen

• When CSF is measured, the contrast of a patch is progressively reduced till it becomes barely visible (50% of viewers can still see it while the rest of viewers can’t). Such threshold points form CSF.

Spatial Frequency (cpd)

Con

tras

t Sen

sitiv

ity

Visible

Invisible

Contrast sensitivity

12

CSF Models

• Many models proposed in the literature• Latest models account for variety of factors:

– Object (screen) luminance– Field size– Oblique effect– Background luminance– etc.

13

Barten-04 CSF Model [20]

• Main formula :

• Low pass branch:

14

Finding maximum visible frequency• Analytic inverse of :

• Lambert W function

• At

• is the maximum visible frequency

15

Sufficient Resolution • Minimum display density:

• Sufficient Resolution:

𝜌𝑚𝑖𝑛 ≈ 360𝜋 u𝑚𝑎𝑥𝑑

𝑊 𝑠𝑢𝑓𝑓 =𝑀 𝜌𝑚𝑖𝑛 ,𝐻 𝑠𝑢𝑓𝑓 =𝑁 𝜌𝑚𝑖𝑛

𝑊 𝑠𝑢𝑓𝑓 =𝑊 𝑑𝑖𝑠𝑝

𝜌𝑚𝑖𝑛

𝜌,𝐻 𝑠𝑢𝑓𝑓=𝐻𝑑𝑖𝑠𝑝

𝜌𝑚𝑖𝑛

𝜌

𝑀

𝑁

( pixels)

( pixels)

𝑢 = 1𝛽, 𝛽 = 𝜋360arctan൬𝑛2𝑑𝜌൰

16

Adaptive Video Streaming

• Server• Can be standard web

server• Media segment can be

prepared in-line or off-line

• Client • Sends series of HTTP GET

segment requests and receives segments

• Performs rate adaptation before sending a new GET segment request

18

Dynamic Adaptive Steaming over HTTP (DASH)

• Dash is NOT– System, protocol, presentation, codec, interactivity

• What is DASH – Enabler which provides formats to enable efficient and high-quality

delivery of streaming services over the Internet– Component of end-to-end service– Enabler to reuse existing technologies (containers, DRM (Digital Rights

Management), codecs)– Enabler for deployment on top of HTTP-CDNs– Enabler for very high user experience (low start-up, no re-buffering)– Provides simple inter-operability points (profiles)

MPEG-DASH: A standard for adaptive streaming

19

MPEG-DASH Overview

20

Media Presentation Description (MPD)

21

MPD example

22

Client centric

• Client has best view of network conditions• No session state in network• Faster innovation and experimentation• But, relies on client for operational metrics

– Only client knows what really happens

23

DASH JavaScript Player

24

DASH-JS (JavaScript Implementation)

25

Implementing User Adaptation

• Built-in Sensors used for viewing setup estimation– Device front camera : Distance measurement– Accelerometer: State detection and distance estimation.– Proximity sensor: Ambient light

• Display information provided by manufactures (Software APIs)– Backlight brightness– Device pixel density– Actual display resolution

26

Distance calculation using front camera

𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒=𝑊 ∗h𝑓𝑤

2∗𝑤∗ tan (𝜃 /2)

27

Device state estimation using accelerometer

28

Effective Contrast measurement

• Reflection coefficient given by manufacturer. • Luminance of screen with white background and maximum brightness ()• Luminance of screen with white background and minimum brightness () • Luminance of screen with black background and maximum brightness ()• Luminance of screen with black background and minimum brightness () • Brightness saturation level () at which luminance with white background reaches maximum.

• Best fit gamma () calculated from power law model.

29

Effective contrast: calculation• Reflected luminance

• Correction factor

• Bright and dark intensities

• Max and min luminance

• Effective contrast

𝐿𝑚𝑎𝑥𝐷 =𝐿 h𝑤 𝑖𝑡𝑒+𝐿𝑟𝑒𝑓 ,𝐿𝑚𝑖𝑛

𝐷 =𝐿𝑏𝑙𝑎𝑐𝑘+𝐿𝑟𝑒𝑓

𝐶𝐷=𝐿𝑚𝑎𝑥

𝐷 −𝐿𝑚𝑖𝑛𝐷

𝐿𝑚𝑎𝑥𝐷 +𝐿𝑚𝑖𝑛

𝐷

𝑄 (𝐵)=( 𝐵𝐵𝑠𝑎𝑡

)𝛾

𝐿𝑟𝑒𝑓=( 𝐼𝜋 )𝐾

30

Implementation of User adaptive DASH

31

Modified JavaScript Player

32

Configuration• Device used : Microsoft surface Pro

• Device state vs distance usedDevice State Corresponding distance

In Hand 12”

On Lap 22”

On Stand 30”

On Table facing up 60”

33

Representations of media used

Representations available Resolutions (pixels)

Corresponding bitrates (bits /sec)

1 1280 x 720 30000002 1024 x 576 20000003 704 x 396 10000004 480 x 270 6000005 320 x 180 349952

Representations available Resolutions (pixels)

Corresponding bitrates (bits /sec)

1 1920 X 1080 60000002 1568 X 880 44410003 1280 X 720 32870004 1056 X 592 24330005 848 X 480 18010006 688 X 384 13330007 576 X 320 9860008 448 X 256 7300009 368 X 208 540000

10 320 X 176 400000

Representations available Resolutions (pixels)

Corresponding bitrates (bits /sec)

1 1920 X 1080 7706632 1280 X 720 5147933 640 X 360 1948344 320 X 180 50842

Representations available Resolutions (pixels)

Corresponding bitrates (bits /sec)

1 1920 X 1080 59334862 1280x720 33604413 960 X 540 22223524 640 X 360 9853215 320 X 180 391544

A B

C D

35

Results• Bandwidth savings for ‘A’ when all the

resolutions are supported by network

In hand On Lap On Stand On Table facing up1280 x 720 1024 x 576 704 x 396 480 x 270

0

10

20

30

40

50

60

70

80

90

100

Sequence A

Device state and selected resolution

Band

wid

th S

avin

gs i

n %

Available resolution: 1280 x 720

36

Results• Bandwidth savings for ‘B’ when all the resolutions are

supported by network

In Hand On Lap On Stand On Table facing up1568 X 880 1280 X 720 1056 X 592 848 X 480

0

10

20

30

40

50

60

70

80

90

100

Sequence B

Device state and selected resolution

Band

wid

th sa

ving

s in

%

Available Resolution: 1920 X 1080

37

Results• Bandwidth savings for ‘C’ when all the resolutions are

supported by network

In Hand On Lap On Stand On Table facing up1920 X 1080 1280 X 720 1280 X 720 640 X 360

0

10

20

30

40

50

60

70

80

90

100

Sequence C

Device state and selected resolution

Band

wid

th sa

ving

s in

%

Available Resolution: 1920X1080

38

Results• Bandwidth savings for ‘D’ when all the resolutions are

supported by network

In Hand On Lap On Stand On Table facing up1920 X 1080 1920 X 1080 960 X 540 640 X 360

0

10

20

30

40

50

60

70

80

90

100

Sequence D

Device state and selected resolution

Band

wid

th sa

ving

s in

%

Available Resolution: 1920X1080

39

Summary• Limitations of human visual system are reviewed and

sufficient resolution of a display to discern information is derived

• MPEG-DASH offered a good platform to implement the concept and analyze the user adaptation technology.

• Using several sensors present in the device, the viewing environment is estimated. And hence the resolution sufficient to display information is calculated.

• Reference DASH player was modified to incorporate the sufficient resolution to select from the bit streams available in MPD.

• Considerable bandwidth savings (10-40%) are observed with this implementation.

40

Conclusions

• By incorporating the characteristics of viewing environment and by understanding the limits of HVS, the streaming of videos to mobile phones can be adaptive to viewing preferences of the user.

• This can be done through MPEG-DASH standard which enables design of intelligent streaming systems adapting not only to bandwidth but also to factors affecting user ability to see visual information.

• It is shown that such adaptation can result in reduced bandwidth usage, increased battery life, and improved quality of user experience

41

Future work

• HVS is a vast area of research, there can be more limits to human other than ones that are described in this thesis. Exploring more in this area may add additional benefits.

• Further research in the direction of detecting user attention while playing the multimedia content may help in improving not only the delivery of media but also the quality of the content played.

42

References[1] D. Wu, Y. T. Hou, W. Zhu, Y.-Q. Zhang and J. M. Peha, "Streaming video over the Internet: approaches and directions," IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, no. 3, pp. 282-300, Mar 2001. [2] G. J. Conklin, G. S. Greenbaum, K. O. Lillevold, A. F. Lippman and Y. A. Reznik, "Video coding for streaming media delivery on the Internet," IEEE Transactions on Circuits and Systems for Video Technology, vol. 11, no. 3, pp. 269-281, Mar 2001. [3] I. Sodagar, "The MPEG-DASH Standard for Multimedia Streaming Over the internet," IEEE MultiMedia, vol. 18, no. 4, pp. 62-67, Apr 2011. [4] ISO/IEC 23009-1, "Dynamic adaptive streaming over HTTP (DASH) - Part 1: Media presentation description and segment formats," Information Technology, 5 Jan 2012. [5] A. Talukdar, M. Cudak and A. Ghosh, "Streaming Video Capacities of LTE Air-Interface," IEEE International Conference on Communications (ICC), pp. 1-5, May 2010. [6] J. Bergquist, "Resolution and contrast requirements on mobile displays for different applications in varying luminous environments," 2nd Int Symp Nanovision Sceince, pp. 143-145, 2005. [7] J. Xue and C. W. Chen, "A study on perception of mobile video with surrounding contextual influences," Fourth International Workshop on Quality of Multimedia Experience, vol. 5, no. 7, pp. 248-253, Jul 2012. [8] Y. A. Reznik, "User-adaptive mobile video streaming using MPEG-DASH," Proc. SPIE, Applications of Digital Image Processing XXXVI, vol. 8856, Sep 2013. [9] Wikipedia, "Snellen's Chart," [Online]. Available: http://en.wikipedia.org/wiki/Snellen_chart.[10] H. Snellen, "Probebuchstaben zur Bestimmung der Sehschärfe (Sample letters to determine the visual acuity)", Berlin: Hermann Peters, 1873.

43

References[11] K. Michael and L. Charles, "Visual Acuity," Web Vision, 5 June 2007. [Online]. Available: http://webvision.med.utah.edu/book/part-viii-gabac-receptors/visual-acuity/. [Accessed 29 June 2014].[12] D. A. Owens and K. Wolf-Kelly, "Near work, visual fatigue, and variations of oculomotor tonus," Investigative ophthalmology & visual science, vol. 28, no. 4, pp. 743-749, 1987. [13] J.Anshel, "Visual ergonomics in the workplace", CRC Press, 2002. [14] Y. Bababekova, M. Rosenfield, J. E. Hue and R. R. Huang, "Font Size and Viewing Distance of Handheld Smart Phones," Optometry and Vision Science, vol. 88, no. 7, pp. 795-797, Jul 2011. [15] G. E. Legge, G. S. Rubin and A. Luebker, "Psychophysics of reading. V. The role of contrast in normal vision," Vision Research, vol. 27, no. 7, pp. 1165-1177, 1987. [16] Recommendation ITU-R BT.500-13, "Methodology for the subjective assessment of the quality of television pictures," International Telecommnication Union, 2012. [17] R. Vanam and Y. Reznik, "Improving the Efficiency of Video Coding by using Perceptual Preprocessing Filter," IEEE Data Compression Conference, 2013. [18] J. A. Movshon and L. Kiorpes, "Analysis of the development of spatial contrast sensitivity in monkey and human infants.," Journal of the Optical Society of America A, vol. 5, no. 12, pp. 2166-2172, 1988. [19] P. G. J. Barten, "Evaluation of subjective image quality with the square-root integral method," Journal of the Optical Society of America A, vol. 7, no. 10, pp. 2024-2031, Oct 1990. [20] P. G. J. Barten, "Formula for the contrast sensitivity of the human eye," Proc. SPIE , Image Quality and System Performance, vol. 5294, pp. 231-238, 2003.

44

References• [21] S. J. Daly, "Visible differences predictor: an algorithm for the assessment of image fidelity,"

Proc of SPIE in Human Vision, Visual Processing, and Digital Display III, vol. 1666, pp. 2-15, 1992. • [22] Y. A. Reznik and R. Vanam, "Improving coding and delivery of video by exploiting the oblique

effect," IEEE Global Conference on Signal and Information Processing, Dec 2013. • [23] L. J. Kerofsky, R. Vanam and Y. A. Reznik, "Improved Adaptive Video Delivery System Using a

Preceptual Pre-processing Filter," IEEE Global Conference on Signal and Information Processing, 2014. • [24] R. M. Corless, G. H. Gonnet, D. E. G. Hare, D. J. Jeffrey and D. E. Knuth, "On the LambertW

function," Advances in Computational mathematics, vol. 5, no. 1, pp. 329-359, 1996. • [25] A. Saamer, A. C. Begen and C. Dovrolis, "An Experimental Evaluation of Rate-Adaptation

Algorithms in Adaptive Streaming over HTTP," Proceedings of the second annual ACM conference on Multimedia systems, pp. 157-168, 2011.

• [26] "Overview of MPEG-DASH," DASH Industry Foundation, [Online]. Available: http://dashif.org/mpeg-dash/.

• [27] DASH-IF, "DASH Industry Forum," [Online]. Available: http://dashif.org/.• [28] DASH-IF, "Reference Client 1.0.0," DASH Industry Forum, [Online]. Available:

http://dashif.org/reference/players/javascript/1.0.0/index.html.• [29] B. Rainer, S. Lederer, C. Muller and C. Timmerer, "A seamless Web integration of adaptive

HTTP streaming," IEEE Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp. 1519-1523, Aug 2012.

• [30] M. Katsikitis, “The Human Face: Measurement and Meaning”, Springler, 2003.

45

References• [31] J. Young, M. Trudeau, D. Odell, K. Marinelli and J. Dennerlein, "Touch-screen tablet user configurations

and case-supported tilt affect head and neck flexion angles," Work: A Journal of Prevention, Assessment and Rehabilitation, vol. 41, no. 1, pp. 81-91, 2012.

• [32] Microsoft, "Surface pro," [Online]. Available: http://www.microsoft.com/surface/en-us/products/surface-pro-2.

• [33] T. Montgomery, "Anatomy, Physiology & Pathology of the Human Eye," [Online]. Available: http://www.tedmontgomery.com/the_eye/acuity.html.

• [34] J. Chipchase, C. Yanqing and Y. Jung, "Personal Television: A Qualitative Study of Mobile TV Users", Springer, 2007.

• [35] K. R. Rao and H. Wu, "Digital Video Image Quality and Perceptual Coding", CRC Press, 2005. • [36] S. Dernbach, B. Das, N. C. Krishnan, B. L. Thomas and D. J. Cook, "Simple and Complex Activity

Recognition through Smart Phones," 8th International Conference on Intelligent Environments (IE), vol. 26, no. 29, pp. 214-221, 2012.

• [37] D. Lamming, "Contrast Sensitivity," in Vision and visual dysfunction, London, Macmillan Press, 1991. • [38] J. Mannos and D. J. Sakrison, "The effects of a visual fidelity criterion of the encoding of images," IEEE

Transactions on Information Theory, vol. 20, no. 4, pp. 525-536, Jul 1974. • [39] V. Cerf and R. E. Kahn, "A Protocol for Packet Network Intercommunication," IEEE Transactions on

Communications, vol. 22, no. 5, pp. 637-648, May 1974. • [40] Wikipedia, "Adaptive bitrate streaming," [Online]. Available:

http://en.wikipedia.org/wiki/Adaptive_bitrate_streaming.• [41] K. R. Boff and L. E. Janet, "Engineering data compendium. Human perception and performance. User's

guide," 1988.

46

Thank you