Multimedia streaming over HTTP has gained momentum with the approval of the MPEG-DASH standard and many research papers evaluated various aspects thereof but mainly within controlled environments. However, the actual behaviour of a DASH client within real-world environments has not yet been evaluated. The aim of this paper is to compare the QoE performance of existing DASH-based Web clients within real-world environments using crowdsourcing. Therefore, we select Google’s YouTube player and two open source implementations of the MPEG-DASH standard, namely DASH-JS from Alpen-Adria-Universitaet Klagenfurt and dash.js which is the official reference client of the DASH Industry Forum. Based on a predefined content con- figuration, which is comparable among the clients, we run a crowdsourcing campaign to determine the QoE of each implementation in order to determine the current state-of- the-art for MPEG-DASH systems within real-world environments. The gathered data and its analysis will be presented in the paper. It provides insights with respect to the QoE performance of current Web-based adaptive HTTP stream- ing systems.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Quality of Experience of Web-‐based Adap8ve HTTP Streaming Clients in Real-‐World Environments using Crowdsourcing
Benjamin Rainer and Chris8an Timmerer
Alpen-‐Adria-‐Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Informa8on Technology (ITEC) w Mul8media Communica8on (MMC) w Sensory Experience Lab (SELab)
h"p://blog.+mmerer.com w h"p://dash.itec.aau.at/ w h"p://selab.itec.aau.at mailto:[email protected]‐klu.ac.at
December 2, 2014
Slides: hVp://www.slideshare.net/chris8an.8mmerer
Outline
• Introduc+on • How to evaluate DASH and QoE • Methodology • Results • Conclusions
December 2, 2014 VideoNext 2014, Sydney 2
Mul+media is Predominant on the Internet
• Real-‐+me entertainment – Streaming video and audio – More than 50% of Internet traffic at peak periods
• Popular services – NeVlix (34.9%), YouTube (14.0%), Amazon Video (2.6%), Hulu (1.4%)
– All delivered over-‐the-‐top (OTT)
– MPEG Dynamic Adap+ve Streaming over HTTP
December 2, 2014 VideoNext 2014, Sydney 3
Global Internet Phenomena Report: 2H 2014
Over-‐The-‐Top – Adap+ve Media Streaming
• In a nutshell …
December 2, 2014 VideoNext 2014, Sydney 4
C. Timmerer and A. C. Begen, “Over-‐the-‐Top Content Delivery: State of the Art and Challenges Ahead”, In Proceedings of the ACM interna+onal conference on Mul+media (MM '14), Orlando, FL, USA, Nov. 2014. h"p://www.slideshare.net/chris+an.+mmerer/over-‐the-‐top-‐content-‐delivery-‐state-‐of-‐the-‐art-‐and-‐challenges-‐ahead
Adapta8on logic is within the client, not norma8vely specified
by the standard, subject to research and development
MPEG Dynamic Adap+ve Streaming over HTTP
What is specified – and what is not?
December 2, 2014 VideoNext 2014, Sydney 5
Media Presenta+on on HTTP Server
DASH-‐enabled Client Media Presenta8on Descrip8on
. . .
Segment
…
.
.
. Segment
…
. . .
Segment
…
.
.
. Segment
…
…
Segments located by HTTP-‐URLs
DASH Control Engine
HTTP/1.1 HTTP Client
MPD Parser
Media Engine
On-‐8me HTTP requests to segments
MPEG Dynamic Adap+ve Streaming over HTTP
What is specified – and what is not?
December 2, 2014 VideoNext 2014, Sydney 6
Media Presenta+on on HTTP Server
DASH-‐enabled Client Media Presenta8on Descrip8on
– Dataset, tools (see backup slides for details) – Common evalua+on setup – Bandwidth traces (real/synthe+c) vs. models
• Metrics – Average media bitrate/throughput at the client – Number of representa+on/quality switches – Number of stalls (in seconds) – buffer level
December 2, 2014 VideoNext 2014, Sydney 9
C. Mueller, S. Lederer, C. Timmerer, “An Evalua+on of Dynamic Adap+ve Streaming over HTTP in Vehicular Environments”, In Proceedings of the Fourth Annual ACM SIGMM Workshop on Mobile Video (MoVid12), Chapel Hill, North Carolina, February 2012.
Quality of Experience • Quality of Experience
– “… is the degree of delight or annoyance of the user of an applica+on or service…”
– Factors influencing / features of QoE may lead to applica+on-‐specific defini+ons
• Subjec+ve quality assessments – Laboratory environment [ITU-‐T B.500 / P.910] – Crowdsourcing with special plaVorms or social networks
• QoE of DASH-‐based services – Startup delay (low) – Buffer underrun / stalls (zero) – Quality switches (low) and media throughput (high)
December 2, 2014 VideoNext 2014, Sydney 10
P. Le Callet, S. Möller and A. Perkis, eds., “Qualinet White Paper on Defini+ons of Quality of Experience (2012)”, European Network on Quality of Experience in Mul>media Systems and Services (COST Ac>on IC 1003), Lausanne, Switzerland, Version 1.2, March 2013."
Methodology • Quality of Experience …
– Mean Opinion Score [0..100] – [other objec+ve metrics:
start-‐up +me, throughput, number of stalls] • … Web-‐based Adap+ve HTTP Streaming Clients …
S. Lederer, C. Müller, C. Timmerer, “Dynamic Adap+ve Streaming over HTTP Dataset”, In Proceedings of the ACM Conference on Mul+media Systems 2012, Chapel Hill, North Carolina, February 2012. // S. Lederer, C. Mueller, C. Timmerer, C. Concolato, J. Le Feuvre, K. Fliegel, “Distributed DASH Dataset”, In Proceedings of the ACM Conference on Mul+media Systems 2013, Oslo, Norway, 2013.
C. Müller and C. Timmerer, “A VLC Media Player Plugin enabling Dynamic Adap+ve Streaming over HTTP”, In Proceedings of the ACM Mul+media 2011, Sco"sdale, Arizona, November 2011. // B. Rainer, S. Lederer, C. Müller, C. Timmerer, “A Seamless Web Integra+on of Adap+ve HTTP Streaming”, In Proceedings of the 20th European Signal Processing Conference 2012, Bucharest, Romania, August 2012.