1 Figure 1 SANDVINE.COM Reveal network quality with ScoreCard Scorecards such as Speedtest.net, Google’s Video Quality Report and Netflix’s Internet Service Provider (ISP) Speed Index as well as newly launched fast.com broadband speed testing site, are popular with consumers and the media due to their controversial nature. However, they offer very little beyond measuring bandwidth - an incomplete measurement of quality of experience (QoE). Broadband network engineers need data they can use to isolate and address network problems and shortcomings or to offer next-gen services for enterprise clients. To gain a better perspective of QoE, service providers need to understand how much data is traversing their networks and where the traffic is heaviest. Application, device type, location, time of day and service plan can all impact Quality of Experience. When it comes to over-the-top (OTT) applications, different outlets (Skype, YouTube, Netflix etc.) run different streaming types with differing resolution levels and may originate from Content Delivery Networks (CDNs) with different levels of robustness or congestion. When you add WiFi and the wide variety of “Bring Your Own Device” (BYOD) devices, operating systems and screen sizes, it presents an even more complex measurement scenario. The figure below is an Netflix ISP Index, which only addresses the average streaming speed for the networks listed (while figures are real we have changed the names of the ISPs). Netflix ISP Speed Index Speed = / Quality WHITEPAPER
16
Embed
Reveal network quality with ScoreCard - Sandvine · Reveal network quality with ScoreCard Scorecards such as Speedtest.net, Google’s Video Quality Report and Netflix’s Internet
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CASE STUDYWHITEPAPER
1
Figure 1
SANDVINE.COM
Reveal network quality with ScoreCardScorecards such as Speedtest.net, Google’s Video Quality Report and Netflix’s Internet
Service Provider (ISP) Speed Index as well as newly launched fast.com broadband
speed testing site, are popular with consumers and the media due to their controversial
nature. However, they offer very little beyond measuring bandwidth - an incomplete
measurement of quality of experience (QoE). Broadband network engineers need data
they can use to isolate and address network problems and shortcomings or to offer
next-gen services for enterprise clients. To gain a better perspective of QoE, service
providers need to understand how much data is traversing their networks and where
the traffic is heaviest.
Application, device type, location, time of day and service plan can all impact Quality of
Experience. When it comes to over-the-top (OTT) applications, different outlets (Skype,
YouTube, Netflix etc.) run different streaming types with differing resolution levels and may
originate from Content Delivery Networks (CDNs) with different levels of robustness or
congestion. When you add WiFi and the wide variety of “Bring Your Own Device” (BYOD)
devices, operating systems and screen sizes, it presents an even more complex measurement
scenario. The figure below is an Netflix ISP Index, which only addresses the average streaming
speed for the networks listed (while figures are real we have changed the names of the ISPs).
Netflix ISP Speed
Index
Speed =/ Quality
WHITEPAPER
CASE STUDYWHITEPAPER
2 SANDVINE.COM
Service Providers need a more granular understanding of these metrics with regard to
their own networks. Approximately 10% of QoE concerns are caused by content, service
availability, device or OSS issues; which engineers can do little about. This paper will explore
how network engineers can use their own QoE scorecard to isolate the location and cause of
problems for better network planning and management and to improve current and next-
generation business class services. It will also explore how Service Providers can fight back
against scorecards publshed by Google and Netflix.
BANDWIDTH METRICS AS A QOE PROXYService Providers are battling for consumer mindshare on broadband services. On one side
are consumers and telecommunications regulatory bodies who are demanding better quality,
greater value and service transparency from broadband operators; on the other side are
service providers’ competitors, ranging from cable operators, DSL and fiber and x (FTTx)
operators (including municipal fiber networks) all the way to fixed-mobile substitution offerings
from mobile operators. To succeed in this environment, service provider network engineers
need more a better view of the actual service being delivered to their customers.
Most service provider networks measure their quality by the speed of a user’s connection,
often by using peak and average bandwidth delivered from their broadband aggregation
or mobile anchoring point. However, this is not a good measurement of quality as it neither
accurately measures the subscriber experience at application level nor gives the operator a
view on whether a subscriber has had a bad experience at any point in the billing period.
Not all applications are bandwidth sensitive, meaning subscribers can have a great experience
even at low bandwidth rates. However, they may require low latency and/or low or no packet
loss. For example, VoIP applications do not need much bandwidth, but low latency and a
fairly robust connection ensure smooth conversations. Additionally, Web browsing does not
usually require high bandwidth, but a poor connection and high latency can cause consumers
to abandon their browsing sessions if pages take longer than 5-7 seconds to load. Gaming,
particularly first person shooters and massively multiplayer online role-playing games
(MMORPGs), require low latency in the network to ensure that the gaming experience is “fair”
and seamless. Even social network sharing applications such as Instagram, Facebook or
YouTube can be a bad experience if the network is poor and uploads take longer than normal
as a result of packet loss.
Average bandwidth is smoothed over an interval period and traffic at non-peak hours may be
tens of Mbps, while less than 1Mbps during peak usage if congestion is not managed and a
high profile online event is occurring. A good example of this scenario is the Super Bowl or
World Cup where many viewers stream the same content simultaneously. In this scenario, the
average bandwidth on the network may appear to be satisfactory, but users on a congested
link during peak hours will not be happy. Peak hours are where the greatest opportunity for
a bad experience to cause churn occurs, especially for heavy video streamers. It also means
that a subscriber may get one (1) second of 20Mbps followed by 60 seconds of 100kbps
utilization. While statistically this may look ok, the user experience will be extremely poor.
Lastly, average bandwidth reporting is also heavily influenced by what devices are accessing
the network, as mobile or Internet of Things (IoT) devices may consume smaller amounts of
bandwidth. This is one of the biggest weaknesses in Netflix’s ISP Scorecard. An HD stream
from a gaming console or smart TV will consume a lot of bandwidth, while the same video
on an iPhone will use much less traffic – even though both cases may result in a good quality
of experience. Inversely, if an HDTV is getting the bandwidth needed by an iPhone for video
streaming, the user will most likely be unsatisfied with the quality of the video.
WHITEPAPER
DOWNLOADING
GAMING
SOCIAL MEDIA
STREAMING VIDEO
UPLOADING
VOICE APPLICATIONS
WEB SURFING
Different applications have different network requirements in order to provide good quality of experience
CASE STUDYWHITEPAPER
3
DEFINING SUBSCRIBE EXPERIENCE METRICSThe Federal Communications Commission (FCC) and the European Union (EU) have selected
several metrics in their quest to ensure that consumers receive good quality experience from
their broadband operators. An excerpt from the FCC filing [Emphasis added in bold]:
“With respect to network performance, we adopt the following enhancements:
l The existing transparency rule requires disclosure of actual network performance. In
adopting that requirement, the Commission mentioned speed and latency as two
key measures. Today we include packet loss as a necessary part of the network
performance disclosure.
l We expect that disclosures to consumers of actual network performance data
should be reasonably related to the performance the consumer would likely
experience in the geographic area in which the consumer is purchasing service.
l We also expect that network performance will be measured in terms of average
performance over a reasonable period of time and during times of peak usage.
l We clarify that, for mobile broadband providers, the obligation in the existing
transparency rule to disclose network performance information for “each broadband
service” refers to separate disclosures for services with each technology (e.g., 3G
and 4G). Furthermore, with the exception of small providers, mobile broadband
providers today can be expected to have access to reliable actual data on
performance of their networks representative of the geographic area in which the
consumer is purchasing service - through their own or third-party testing - that
would be the source of the disclosure. 410 Commission staff also continue to refine
the mobile MBA program, which could at the appropriate time be declared a safe
harbor for mobile broadband providers.”
The excerpt above, from pages 73-74 of FCC-15-24A1* of the official FCC document on
network neutrality, is one of the most critical parts of the announcement for subscribers.
Taken outside the financial aspect of the bright line rules to protect the “free” aspect of the
Internet, this section highlights a growing problem for consumers with broadband services:
Speed is not always a good indicator of service.
We will explore each one of the metrics mentioned above and why that metric is important for
ABOUT SANDVINESandvine helps organizations run world-class networks with Active Network Intelligence, leveraging machine learning analytics and closed-loop automation to identify and adapt to network behavior in real-time. With Sandvine, organizations have the power of a highly automated platform from a single vendor that delivers a deep understanding of their network data to drive faster, better decisions. For more information, visit sandvine.com or follow Sandvine on Twitter at @Sandvine.