8/6/2019 Analysys Mason Domestic Peering Paper
1/47
Public report
Overview of recent changes in the IP
interconnection ecosystem
May 2011
17038-93
.
8/6/2019 Analysys Mason Domestic Peering Paper
2/47
Contents
1
Executive summary 1
1.1 Internet trends 11.2 Evolution of Internet interconnection 22 Introduction 43 Internet trends 73.1 Internet traffic has grown exponentially 73.2 Internet traffic has become globalized 93.3 The value of Internet content and applications has increased dramatically 113.4 Bandwidth-intensive types of Internet traffic have become predominant 133.5 Cloud computing services have become widely utilized 163.6 Internet connections are a key feature for many new devices and services 173.7 Conclusion 184 Evolution of Internet interconnection 204.1 Description of the early Internet 204.2 Internet exchange points (IXPs) 214.3 Internet service providers (ISPs) 234.4 Content providers 254.5 Impact on backbone providers 285 Conclusion 34
Annex A: Introduction to the IP interconnection ecosystem
Annex B: About Analysys Mason
8/6/2019 Analysys Mason Domestic Peering Paper
3/47
.
Copyright 2011. The information contained herein is the property of Analysys Mason
Limited and is provided on condition that it will not be reproduced, copied, lent or
disclosed, directly or indirectly, nor used for any purpose other than that for which it was
specifically furnished.
Analysys Mason Limited
818 Connecticut Avenue NW
Suite 300
Washington DC 20006
USA
Tel: (202) 331 3080
Fax (202) 331 3083
www.analysysmason.com
Registered in England: Analysys Mason Limited
Bush House, North West Wing, Aldwych
London WC2B 4PJ, UK
Reg. No. 5177472
8/6/2019 Analysys Mason Domestic Peering Paper
4/47
Overview of recent changes in the IP interconnection ecosystem | 1
1 Executive summary
Last year saw the 15th anniversary of the commercialization of the Internet backbone as we know it
today, when the US National Science Foundation Network (NSFNET) Backbone Service wasdecommissioned in favor of the current commercial Internet backbone. On that day, April 30, 1995,
the Netscape browser had just been introduced, AOL was the most popular ISP in the US and was still
charging hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been
released. The Internet was mainly used for file transfer, email was simple text, and the Web was not
yet multimedia.
Over the intervening 15 years, there have been significant changes in the Internet ecosystem,
including an enormous increase in Internet traffic and bandwidth along with a corresponding dramatic
increase in the value generated by Internet content. During this period, interconnection arrangements
between all of the players in the ecosystem have continuously evolved to meet the challenges of the
changing Internet ecosystem, driven by commercial considerations and not regulation.1
1.1 Internet trends
The changes in the Internet ecosystem are mainly due to the following interrelated trends:
Internet traffic has
grown
exponentially
There has been an explosion of Internet access and in the traffic generated
by each individual user, for the reasons described below. As a consequence,
market players have adapted by modifying their network architectures andrevamping their relationships with suppliers and peers.
Internet traffic has
become globalized
From its historical origins in the US, Internet usage has grown significantly
around the world, and correspondingly, Internet content creation has also
globalized. As a result, the proportion of Internet traffic originated and
terminated outside the US has significantly increased over time.
Additionally, the amount of Internet traffic exchanged in US-based Internet
exchange points (IXPs) is now much less predominant, as many countries
and regions have expanded their Internet traffic exchange capabilities
through one or more local IXPs.
The value of
Internet content has
dramatically
increased
15 years ago, money flows were mostly directed to ISPs and backbone
providers for Internet access and traffic, while content was mostly free or
had limited value. This has changed significantly, as content has gained in
value thanks to an improved quality of access and the corresponding
availability of premium content. As a result, content providers have been
able to leverage the value of their content in order to reduce their costs for
1The only instances of regulatory intervention in IP interconnection have been during merger proceedings, when agencies have imposed
conditions ranging from divestiture to temporary limits on changes in peering agreements.
8/6/2019 Analysys Mason Domestic Peering Paper
5/47
Overview of recent changes in the IP interconnection ecosystem | 2
.
delivering their traffic (for instance, by building their own infrastructure to
distribute content around the country closer to end-users).
Bandwidth-
intensive and
quality needy types
of Internet traffic
such as video have
become
predominant
Bandwidth-intensive and quality needy types of traffic, such as video, have
experienced a massive growth, which has had a major impact on bandwidth
requirements and the direction of traffic flows. There has been a significant
increase in the amount of traffic being downloaded from centralized sites
such as YouTube, while at the same time the popularity of peer-to-peer
services has resulted in significant traffic between end users. These
increases in overall traffic, along with changes in traffic patterns, have led
ISPs and content providers to adapt their network policies and architectures.
Cloud computing
services have
become widely
utilized
Cloud computing services, ranging from online email to business
applications, have been embraced as barriers to their adoption have been
progressively lifted, thanks to the improvement of Internet access, the
development of security safeguards, and the resulting increase in awareness
and confidence in cloud computing. This has increased overall usage of the
Internet, while also making consumers more sensitive to latency and other
quality measures that impact access to their cloud applications.
Internet connection
became a
mandatory feature
for many new
consumer
electronic devices
Many new devices have emerged that incorporate Internet access: mobile
devices with features adapted to data traffic, netbooks and tablets designed
to enable Internet access to cloud-computing applications, and new video
streaming devices relying on broadband. As a result, many traditional
communications services have migrated to the Internet, for example TVover IP and Voice over IP (VoIP). This has led market players to increase
mobile access, install additional network capacity, and establish new forms
of interconnection.
The changes described above, individually and in sum, have led to significant changes in the
underlying Internet ecosystem as it was established 15 years ago.
1.2 Evolution of Internet interconnection
15 years ago, the Internet ecosystem had a relatively simple structure, with clear and hierarchical
relationships between end users and content providers buying Internet access from ISPs, ISPs buying
Internet transit from backbone providers, and backbone providers peering with each other. 2 At the
time, given the historical development of the Internet, much of the global traffic originated,
terminated, or transited the US.
But Internet players have steadily adapted to the changing market conditions described above in a
number of ways, in order to carry greater traffic volume with lower latency, and to reduce their transit
2For an introduction to Internet market players and interconnection definitions, please see Annex A.
8/6/2019 Analysys Mason Domestic Peering Paper
6/47
Overview of recent changes in the IP interconnection ecosystem | 3
.
costs, thereby changing the traditional hierarchy of the ecosystem in the process, while also making it
truly global.
Internet Exchange
Points (IXPs)
facilitated
increased
connectivity
The development of IXPs around the world has benefited backbone
providers by facilitating distributed interconnection between peers and
transit customers, which improves the quality of service and reduces traffic
carrying costs. However, the increase in utilization of IXPs also helped in
marginalizing the role of the backbone providers, as former customers
began to interconnect directly with one another at the IXPs.
ISPs and content
providers
increasingly
connect directly
Many route around options have emerged for ISPs and content providers
to exchange Internet traffic, while avoiding Internet backbone transit costs.
These include secondary peering arrangements between ISPs to exchange
traffic directly, much of it from peer-to-peer applications. Increased traffic
has also led content providers to negotiate paid peering arrangements with
ISPs to deliver content to their end users. Backbones have adapted to these
changes by selling only partial transit to ISPs and content providers, which
have arranged much of their traffic exchange directly. At the same time,
peering relationships have evolved based on volume, traffic flows and
routing.
Content providers
have a variety of
options for
delivering traffic
Increases in content traffic have led to traffic imbalances, such that ISPs
delivered much more traffic from content providers to their end users. This
has led to paid peering arrangements to cover the cost of the traffic. Content
providers have leveraged their expanding financial resources to delivertraffic directly to ISPs by building private networks or using third-party
content delivery networks (CDNs) to deliver content directly to caches at
the edge of ISPs networks, further reducing transit costs and the resulting
revenues for backbones.
From the backbone providers point of view, these changes led to a reduction in demand for transit
services, and an increase in competition from former customers who now have a number of choices
for delivering and exchanging traffic. Further, backbones must compete vigorously on the price of
transit in order to generate the traffic volume to continue to peer with one another. This has resulted in
an increase in the level of competition for Internet transit services, as evidenced for example in the
fall in transit prices over the past five years, with no sign of respite.
In conclusion, in the 15 years since the commercialization of the Internet backbone, the Internet
ecosystem has proven itself to be able to develop and sustain interconnection in the absence of sector-
specific regulation. It has also shown itself to be able to adapt well to rapid and profound market
changes without regulatory intervention, which in telecommunications is typically much slower to
implement changes in interconnection arrangements and has issues with implicit subsidies and
arbitrage. As a consequence of these changes, todays Internet ecosystem is no longer hierarchical, but
rather a dynamic web of interconnections between a variety of Internet players.
8/6/2019 Analysys Mason Domestic Peering Paper
7/47
Overview of recent changes in the IP interconnection ecosystem | 4
.
2 Introduction
This paper marks the 15th anniversary of the commercialization of the Internet backbone as we know
it today. There have been significant changes in the Internet ecosystem over the last 15 years,resulting in enormous increases in Internet traffic, supported by a massive expansion of the Internet
bandwidth, and accompanied by a major increase in the value generated by Internet services and
applications. Through these changes, one feature remains prominent : interconnection arrangements
between all of the players in the ecosystem have continued to evolve to meet the challenges of the
changing ecosystem, driven almost entirely by commercial considerations and not regulation.3
On April 30, 1995, the US National Science Foundation Network (NSFNET) Backbone Service was
decommissioned in favor of the current commercial Internet backbone. The NSFNET operated a
national backbone network allowing regional networks to connect at supercomputing sites.4 At this
time, the objective was to create an open network linking academic researchers, and allowing them
access to distant supercomputers over the network at no cost. When the NSFNET network went
online in 1986 and allowed commercial traffic (and not just NSFNET traffic) to be carried, it
consisted of six sites interconnected with leased 56kbit/s Digital Data System (DDS) links this was
upgraded to 45Mbit/s links in 1991.
At the time that the NSFNET was decommissioned in 1995, Netscape the first commercial Web
browserhad just been introduced, AOL was the most popular ISP in the US and was still charging
hourly fees, and the DOCSIS standard for cable modem Internet access had not yet been released. The
Internet was used mainly for file transfer, email was simple text, and the Web was not yet multimedia.Since then, traffic has exploded, as shown in Figure 2.1below.
3The only instances of regulatory intervention in IP interconnection have occurred during merger proceedings, when agencies have
imposed conditions ranging from divestiture to temporary limits on changes in peering agreements.
4 From 1987 to 1995, the NSFNET Backbone was designed, managed, and operated on behalf of the NSF by Merit Network, Inc., a non-
profit corporation governed by public universities partnering with IBM, MCI, and the State of Michigan.
8/6/2019 Analysys Mason Domestic Peering Paper
8/47
Overview of recent changes in the IP interconnection ecosystem | 5
.
Figure 2.1: Monthly
backbone traffic in the
US [Source: Minnesota
Internet Traffic Studies;
Analysys Mason
Estimates]
This increase in traffic comes as a result of a virtuous circle of more powerful computers, the
availability and adoption of broadband access, the increase in bandwidth available for carrying
Internet traffic, and the introduction of multimedia content (particularly video), along with new ways
to share that content, such as peer-to-peer. Figure 2.2 below shows how new categories of Internet
video have arisen in the last five years and are projected to become the dominant category of content
over the next few years.
Figure 2.2: Consumer
Internet traffic forecasts
[Source: Cisco Visual
Networking Index]
Note: in the figure above, the File sharing category includes traffic from peer-to-peer applications such as BitTorrent and eDonkey, as well as
web-based file sharing.
The NSFNET was the backbone over which regional networks exchanged traffic. When it was closed
in favor of commercial backbones, there was no longer a single backbone that could be used for
0
500,000
1,000,000
1,500,000
2,000,000
2,500,000
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
TB/month
0
2,000
4,000
6,000
8,000
10,000
12,000
2006 2007 2008 2009 2010 2011 2012 2013 2014
PBpermonth
Web and Email File Sharing
Internet Gaming Internet Voice
Internet Video com . Internet Video to PC
Internet Video to TV
8/6/2019 Analysys Mason Domestic Peering Paper
9/47
Overview of recent changes in the IP interconnection ecosystem | 6
.
traffic exchange. Instead, four Network Access Points (NAPs) across the country were designated as
the points at which traffic could be exchanged by the new commercial backbones. Such
interconnection was not regulated, making it possible for commercial arrangements known aspeering
and transit to be negotiated between the backbones and ISPs. The introduction of commercial
backbones not only modified the structure of the Internet infrastructure ecosystem, but also
accommodated to changes to the type of Internet traffic carried, as this opening increased greatly the
carrying and exchange of commercial traffic in addition to the university and research traffic
previously carried.
This document aims at describing changes in the Internet ecosystem since the commercialization of
the Internet backbone. We analyze the changes in terms of the type of Internet traffic carried, the
infrastructure supporting the Internet traffic, as well as the evolution of the different players and their
inter-relationships. We also assess how commercial models adapted to these changes, and analyze the
impact of these changes on competition between the backbones.
The remainder of this document is laid out as follows:
Section 3 provides an outlook of the main trends over the past 15 years that have impacted the
Internet backbone
Section 4 analyses the evolution of the Internet backbone, with a focus on interconnection
arrangements, and describes the current market situation from point of view of each of the main
types of Internet players
Section 5 summarizes the main conclusions reached.
8/6/2019 Analysys Mason Domestic Peering Paper
10/47
Overview of recent changes in the IP interconnection ecosystem | 7
.
3 Internet trends
The Internet ecosystem has evolved significantly over the past 15 years, with corresponding impacts
on interconnection arrangements. This evolution is partly due to a number of trends that are allinterrelated, but that can be described as follows:
Internet traffic has grown exponentially
Internet traffic has globalized
the value of Internet content and Applications has increased dramatically
bandwidth-intensive types of Internet traffic such as video have become predominant
cloud-computing services have become widely utilized
Internet connections are a mandatory feature for many new consumer electronic devices and
services.
Figure 3.1: Trends that
have impacted the
Internet ecosystem over
the past 15 years
[Source: Analysys
Mason]
We provide in this section an analysis of these trends, which will help to better understand the context
of the evolution of Internet interconnection described in the following section.
3.1 Internet traffic has grown exponentially
The past 15 years have seen an explosion of Internet usage around the world, particularly from
Internet broadband subscriptions that allow end users to use their connections to access the new high
bandwidth services such as video streaming. At the end of September 2009, there were more than 75
million fixed broadband subscriptions in the US, representing a penetration of around 60% of
households, and significant growth from the end of 2001, when there were only around 10 million
broadband subscribers, as shown in the Figure 3.2 below.
Globalization ofInternet traffic
Value of InternetContent
Peer-to-peer andvideo streaming
Cloud computing
Online devices
Volume ofInternet traffic
8/6/2019 Analysys Mason Domestic Peering Paper
11/47
Overview of recent changes in the IP interconnection ecosystem | 8
.
Figure 3.2: Growth in
number of broadband
subscribers in the US
[Source: Analysys Mason
Research]
In the past 10 years, this dramatic increase in the adoption of fixed broadband has been accompanied
by an explosion of mobile broadband usage. According to CTIA, wireless data revenues in the US
rose from US$211 million in 2000 to US$41.5 billion in 2009. 5 Similar growth has characterized
markets in Europe and much of Asia as well, resulting in demand for Internet services that is
distributed much more evenly across the world than 15 years ago, as shown in Section 3.2.
Not only has the number of Internet access users experienced a significant growth, but also the
bandwidth usage by each individual user has exploded. This can be explained in part by the increase
in the availability of even higher capacity Internet links, and the growing availability of bandwidth-
intensive applications such as video downloads (see section 3.4), the development of cloud-computing
services (see section 3.5), and the take-up of a large variety of devices connected to the Internet (see
section 3.6).
These combined effects have resulted in an exponential growth of Internet traffic in general, and in
the traffic carried by Internet backbone providers in particular, as shown above in Figure 2.1. The fast
growth in traffic has naturally generated significant changes in the Internet ecosystem, as marketplayers have adapted to this demand increase by modifying their network architectures (e.g.
expanding their networks by deploying fiber and upgrading their capacity to accommodate higher
traffic volumes and traffic that is more latency sensitive, and creating new routes to carry the traffic
more efficiently), as well as revamping their relationships with suppliers and peers, as described
below.
5Source: http://www.ctia.org/media/industry_info/index.cfm/AID/10323
0
10
20
30
40
50
60
70
80
90
2001 2002 20032004 2005 20062007 2008 2009
Q3
Millions
0%
10%
20%
30%
40%
50%60%
70%
80%
90%
100%
Penetrationas%households
DSL Cable modem
Residential FTTB Broadband FWABroadband penetration
http://www.ctia.org/media/industry_info/index.cfm/AID/10323http://www.ctia.org/media/industry_info/index.cfm/AID/10323http://www.ctia.org/media/industry_info/index.cfm/AID/10323http://www.ctia.org/media/industry_info/index.cfm/AID/103238/6/2019 Analysys Mason Domestic Peering Paper
12/47
Overview of recent changes in the IP interconnection ecosystem | 9
.
3.2 Internet traffic has become globalized
For historical reasons, in the early 90s Internet usage was biased towards the US, as was Internet
traffic exchange at the time that the NSFNET was decommissioned in 1995. This situation evolved
significantly over time, as demand for Internet access exploded first in all the other developed
countries, and then in developing countries. Indeed, today a significant portion of Internet users are to
be found in large markets like China and India: for example, China today boasts more than 400
million Internet users, and other emerging Asian nations are also fuelling the growth of traffic in this
region.
Figure 3.3: Evolution of
the distribution of Internet
users by region[Source:
ITU]
The increase in the number of Internet users outside the US led to a corresponding increase in Internet
traffic in other regions of world, with Internet content being created, hosted and accessed from outside
the US. As a result, the Internet network topology began to evolve, so as to enable a more efficient
carrying of traffic. The historical use of US-based IXPs to exchange foreign traffic was not only
inefficient, but also led to a call for intervention that went by the rubric of International Charging
Arrangements for Internet Services (ICAIS). In this debate at the Asia-Pacific Economic Cooperation
(APEC) forum and the International Telecommunication Union (ITU), countries underserved by
Internet infrastructure sought to address the issue of bearing the costs of connecting to US-based IXPs
for traffic exchange.
To deal with these issues, national and regional networks started to interconnect to each other and
exchange their traffic at regional IXPs such as AMS-IX in Amsterdam and LINX in London. These
IXPs exchange a significant amount of traffic among a growing number of members. For instance,
AMS-IX has 389 connected networks exchanging 1091.9Gbit/s of traffic at peak times, while LINX
0%
10%
20%
30%
40%50%
60%
70%
80%
90%
100%
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
North America Europe and Central Asia
East Asia and Pacific Latin America and Carabbean
Middle East and North Africa South Asia
Sub-Saharan Africa
8/6/2019 Analysys Mason Domestic Peering Paper
13/47
Overview of recent changes in the IP interconnection ecosystem | 10
.
has 379 members with a peak traffic exchange rate of 614Gbit/s.6 This traffic is not just national, but
also international, and the result is that the host countries have turned into significant international
hubs. As shown in Figure 3.4 below, the Netherlands, followed by the UK, has a distinct edge in the
amount of international Internet bandwidth per population compared with a wide variety of countries,
based in no small part on the significant role of Dutch and British IXPs in attracting international
backbones and content providers. Similar activity is taking place in Asia and to a lesser degree in
other parts of the world, as IXPs act to help to localize traffic and attract a surrounding ecosystem of
backbones and content providers.
Figure 3.4: Growth in international Internet bandwidth per head of population [Source: Telegeography,
ITU, Euromonitor, company data]
Consequently, the role of the US as a hub for international exchange of Internet traffic has diminished
significantly. Figure 3.5 below shows that even though Latin America still relies heavily on the US
for Internet bandwidth, regional networks in Europe, Asia and Africa are less and less dependent on
US-based facilities to carry and exchange traffic. Africas Internet traffic is now focused morepredominantly on Europe, while Europes traffic is mainly intra-regional, which is also becoming the
norm in Asia.
6In October 2010 AMS-IX had 389 networks connected, measured by autonomous system numbers, with a peak traffic exchange of
1091.9Gbpit/s (source: www.ams-ix.net). LINX provides general statistics showing that in October 2010 it had 379 members exchanging
614Gbit/s peak traffic (source: www.linx.net).
00
20
40
60
80
100
120
140
160
2002
2003
2004
2005
2006
2007
2008
kbit/s
Australia
USA
Canada
Germany
France
Singapore
UK
Netherlands
http://www.ams-ix.net/http://www.ams-ix.net/8/6/2019 Analysys Mason Domestic Peering Paper
14/47
Overview of recent changes in the IP interconnection ecosystem | 11
.
Figure 3.5: Share of
Internet bandwidth
connected to US and
Canada, by region
[Source: Telegeography]
Figures represent Internet bandwidth connected across international borders as of mid-year. Domestic routes are excluded.
Today, 89 countries possess their own IXPs to exchange national and regional Internet traffic, and
thus lower the costs associated with international Internet access. 7 It is interesting to note that this
evolution has not required specific regulatory intervention, but is the result of commercial solutions
adopted by national and regional networks.
3.3 The value of Internet content and applications has increased dramatically
As new Internet services have developed, the positioning of the different stakeholders in the Internet
ecosystem has evolved (see Annex A for an introduction to the IP interconnection ecosystem). 15
years ago, ISPs and backbone providers were at the top of the hierarchy, and money flows were
mostly directed to them for Internet access and traffic, while content was mostly free or of limited
value. This has changed significantly in recent years as the value of content has risen, thanks to an
improved quality of service and the development of premium content.
Online music provides a good illustration of the evolution of content values in the past decade.
Initially, music labels were reluctant to sell their music online, for a variety of reasons. As a result, the
first widespread distributor of online music was the peer-to-peer network Napster. This was free, but
content was largely pirated and copyright issues led to its shutdown in 2001. Following this, labels
became comfortable with the level of protection being offered by digital rights management (DRM),
and began to make their music available online, leading to the emergence of paid downloads. A good
7Source: Packet Clearing House Report on Internet Exchange Point Locations (https://prefix.pch.net/applications/ixpdir/summary/)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2002 2003 2004 2005 2006 2007 2008 2009 2010
Africa Asia Europe Latin America
https://prefix.pch.net/applications/ixpdir/summary/https://prefix.pch.net/applications/ixpdir/summary/8/6/2019 Analysys Mason Domestic Peering Paper
15/47
Overview of recent changes in the IP interconnection ecosystem | 12
.
illustration is the success of the media platform iTunes Store, launched by Apple in April 2003: as of
February 2010, 10 billion songs have been sold on this platform.8
Other significant value-added services have emerged from mobile Internet access, with the explosion
of the mobile paid applications market. The success of Apples App Store can illustrate this trend: as
of January 2010, 3 billion applications (both free and paid-for) had been downloaded through the
store.9 More generally, the Yankee Group estimates that the total revenues for the mobile paid
applications industry in the US alone reached US$1.6 billion in 2010, against US$537 million the
preceding year. This rise in content value will continue in the coming years, as this market growth
will be supported by devices such as the new iPad designed for the consumption of online multimedia
content. According to the same source, application revenues are forecast to reach US$11 billion by
2014.10
The massive growth in Internet advertising revenue illustrates well the growth in content value. Figure
3.6 below provides an estimate of the evolution of the fixed and mobile Internet advertising markets
in North America. The Internet advertising market is estimated to have been worth around US$25
billion in 2009, compared with approximately US$10 billion five years before, and is projected to
grow to more than US$35 billion by 2013. It should be noted that a large part of these revenues are
captured by content providers such as Google.
Figure 3.6: Internet
advertising market in
North America [Source:
PricewaterhouseCoopers
LLP, Wilkofsky Gruen
Associates]
8Source: http://www.apple.com/pr/library/2010/02/25itunes.html
9Source : http://en.wikipedia.org/wiki/ITunes_Store#Video_2
10Source : http://www.lesechos.fr/info/comm/020439905748-applications-mobiles-l-institut-yankee-group-a-revu-a-la-.htm
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
40,000
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
USD
million
Wired Internet advertising Mobile Internet advertising
Forecasts
http://www.apple.com/pr/library/2010/02/25itunes.htmlhttp://en.wikipedia.org/wiki/ITunes_Store#Video_2http://en.wikipedia.org/wiki/ITunes_Store#Video_2http://www.apple.com/pr/library/2010/02/25itunes.html8/6/2019 Analysys Mason Domestic Peering Paper
16/47
Overview of recent changes in the IP interconnection ecosystem | 13
.
An interesting example of the evolution of the value of content is ESPN360.com, a broadband
network for live sports programming in the US and several other countries. ESPN has essentially
adopted a pay-TV model, in which ISPs must sign up and pay ESPN for offering the service, rather
than charging end users directly. The customer pays for access as a part of its ISP fees, whether it uses
the service or not, and may choose an ISP based on the availability of ESPN360.
This example suggests the extent to which the relative positions of content providers and ISPs have
evolved in the Internet ecosystem. In particular, content providers have been able to leverage the
attractiveness of their content in order to reduce their supply costs for accessing the Internet.
Furthermore, as we will see in Section 4.4, content providers can use their fast-growing revenues to
invest in their own network infrastructures, in order to further reduce the transit costs by developing
self-supply solutions, and thus avoid paying other market players to deliver their traffic.
3.4 Bandwidth-intensive types of Internet traffic have become predominant
The nature of Internet traffic has changed dramatically in recent years, and in particular there has been
a massive growth in the proportion of video traffic as a percentage of total Internet traffic. Video
traffic is distributed either via peer-to-peer services or streamed from a centralized server. Projections
show that video will remain a major driver for traffic increase (see Figure 2.2) and sustain the growth
in Internet traffic. We focus in this section on the growth of streaming video and peer-to-peer traffic,
which not only has a major impact on the bandwidth requirement for Internet traffic, but also on the
traffic patterns, as video streaming modifies the traffic pattern from traditional two-way
communication to one-way content delivery).
3.4.1Sustained growth in video traffic
The consumption of video media consists mostly of Internet video to PC (i.e. free or pay-TV viewed
on a PC), and in a more limited scale in Internet video to TV (the same as video to PC, but streamed
to the TV set). Thus today, 82% of Internet users in the US watch videos online, and an average user
views 182 videos per month.11 YouTube, a video-sharing website on which users can upload and view
videos, is a leading symbol of this Internet explosion. YouTube was founded in February 2005 and
bought by Google in 2006. In only five years, this company achieved an almost unimaginable growth:
by October 2009, it was serving more than a billion videos per day worldwide.12 In November 2009,
12.2 billion videos were viewed on YouTube on a monthly basis in the US alone. 13
Not only has YouTube managed to draw a very large audience, it has also succeeded in attracting a
large amount of free user-generated content: YouTube announced on March 2010 that users were
uploading some 24 hours of video to the platform every minute.14 This represents double what was
11Source : http://website101.com/statistics/Internet-stats-video-stats-social-media-usage-traffic-email/#ixzz0jJUhngj3
12Source : http://bits.blogs.nytimes.com/2009/10/09/youtube-were-bigger-than-you-thought/?hp)
13Source: http://bits.blogs.nytimes.com/2009/10/09/youtube-were-bigger-than-you-thought/?hp)
14Source: http://youtube-global.blogspot.com/2010/03/oops-pow-surprise24-hours-of-video-all.html
8/6/2019 Analysys Mason Domestic Peering Paper
17/47
Overview of recent changes in the IP interconnection ecosystem | 14
.
uploaded only two years before (of course, some of this material is commercial, and YouTube has had
to address the copyright issues that arose first with Napster).
Figure 3.7: Hours of
video uploaded on
YouTube platform per
minute [Source:
YouTube]
But YouTube, while significant, is not the only player in the Internet video market: there are other
major video content providers that are experiencing similar growth levels. Another example would be
Hulu, a website offering commercial-supported streaming video of TV shows and movies from
several major TV channels and studios. In November 2009, Hulu was streaming 924 million videos
per month in the US alone, according to the research company Comscore.15 In the UK, the BBCs
online iPlayer is estimated to have received well over a billion requests for programming in 2010,
with 114 million requests in July 2010 alone.16 Other services such as Netflix stream video directly to
the TV. According to Sandvine, streamed audio and video represent almost 43% of data consumption
in North America during peak periods, with Netflix video alone accounting for over 20% of download
traffic during peak times.17 In addition to these video-streaming sites, a significant amount of video is
consumed using peer-to-peer networks.
15Source:
http://www.comscore.com/Press_Events/Press_Releases/2010/1/November_Sees_Number_of_US_Videos_Viewed_Online_Surpass_30
_Billion_for_First_Time_on_Record
16This includes both video and radio programming. See http://www.bbc.co.uk/blogs/bbcInternet/img/Publicity_Pack_July_2010.pdf.
17Source: Fall 2010 Global Internet Phenomena Report, Sandvine, p.9.
0
5
10
15
20
25
30
Q22007
Q32007
Q42007
Q12008
Q22008
Q32008
Q42008
Q12009
Q22009
Q32009
Q42009
Q12010
Hours
8/6/2019 Analysys Mason Domestic Peering Paper
18/47
Overview of recent changes in the IP interconnection ecosystem | 15
.
3.4.2Sustained growth in peer-to-peer traffic
Peer-to-peer networks allow a distributed community of users to share digital content or resources,
including video.18 But contrary to a classic clientserver architecture in which all network content is
held on a server in a central location and accessed via a client on the end-users devices, peer-to-peer
resources are located in and provided by devices at the edge of t he network by peers. Peer-to-peer
quickly became a popular way for Internet users to share video and music files, as it was particularly
adapted to transfer large files over the Internet. Indeed peer-to-peer networks use the resources
provided by users (bandwidth, storage space, and computing power) to increase the total capacity of
the system at little or no cost to the originator of the peer-to-peer network. In contrast, in a typical
clientserver architecture, clients make demands on the system, but do not supply any resources (as
such, as more clients join the system, the owner must provide more resources to meet the demand, or
congestion will impact performance).
Peer-to-peer services themselves are not a recent Internet application: they have their origins in many
of the early Internet services, such as Internet Relay Chat (IRC) which was developed at the end of the
1980s. The widespread popularization of peer-to-peer goes back to Napster in 1999. Napsters success
was striking, as only two years after the launch of operations it was reaching 26.4 million users. As
much of the material distributed over Napster violated copyrights, the service was closed down after a
lawsuit instigated by the Recording Industry Association of America (RIAA).19 Nonetheless, peer-to-
peer networks have become more and more popular, and have overcome the perceived drawbacks of
Napsters network (centralized servers distributing the files and requests to the peers) by relying on a
decentralized or a hybrid architecture: without central servers, the technical risks are minimized,
operational costs are reduced (database management in particular is distributed to peers) and the
ability of authorities to shut down the service following legal decisions is much more limited. Kazaa
and Gnutella 0.6 are notable examples of such new peer-to-peer networks.20
As broadband access grew, and music began to shift to commercial sites such as iTunes, peer-to-peer
networks were increasingly used for the exchange of higher-bandwidth video traffic. The result of the
popularization of peer-to-peer was a significant change, in which users became content providers as
well as consumers, and the flow of content shifted from a centralized location to become more
distributed. Unlike the streaming of a video from a centralized website such as YouTube, the
downloading of the same video using a peer-to-peer network has a very different impact in terms of
network utilization: in this case the traffic goes from one or more end users to another end user (andboth ways, as peer-to-peer is based on reciprocal sharing).21
18Even though the rise in peer-to-peer traffic is relatively recent, peer-to-peer services themselves should not be considered to be a recent
Internet application. Peer-to-peer networks have their origins in many of the early Internet services: for instance, the Internet Relay Chat
(IRC), which was developed at the end of the 1980s.
19See http://papers.ssrn.com/sol3/papers.cfm?abstract_id=504062
20See http://en.wikipedia.org/wiki/Timeline_of_file_sharing
21The difference may be most clearly understood by comparing YouTube with a peer-to-peer network. In either case, the content may be
user-generated, but with YouTube the originator uploads it once to YouTube, from which others download it. In contrast, with peer-to-
peer, each time a user downloads the content it is resent from the originator (or other peers).
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=504062http://en.wikipedia.org/wiki/Timeline_of_file_sharinghttp://en.wikipedia.org/wiki/Timeline_of_file_sharinghttp://papers.ssrn.com/sol3/papers.cfm?abstract_id=5040628/6/2019 Analysys Mason Domestic Peering Paper
19/47
Overview of recent changes in the IP interconnection ecosystem | 16
.
As a big proportion of traffic goes directly from ISP end users to ISP end users, and large file-sharing
traffic requires very high bandwidth, ISPs have responded by adapting their network policies and
architectures. In particular, they have had to find alternative routes to deliver this video traffic.
Combined with the general growth of Internet traffic, it started to make economic sense for ISP to
adopt solutions like secondary peering to address this traffic issue (see Section 4).
3.4.3Forecasts
Growth in video traffic is not expected to slow down in the medium term, and according to Cisco, the
sum of all forms of video (IPTV, video on demand, Internet video, and peer-to-peer) will account for
over 85% of global consumer traffic by 2013. Internet video excluding peer-to-peer (free or pay TV
and video on demand delivered on a PC or a TV) will account for over 55% of all consumer Internet
traffic by this time.22 It appears therefore that video traffic will be the major driver for the Internet
traffic growth in the coming years, in proportions that will require significant investments in Internet
infrastructures and technology development to carry the traffic and sustain the quality of service (viaan increase in available bandwidth).
It is interesting to note that in relative terms peer-to-peer is actually declining as a percentage of
overall IP traffic, in favor of video streaming from centralized servers. This evolution can be
explained by the growing availability of alternative ways to watch online premium content
(e.g. YouTube, iPlayer and Hulu.com) using centralized servers, and the relative drawbacks of peer-
to-peer downloads compared to video streaming. From video-streaming websites, content is available
with high quality, is often free, and can typically be downloaded quickly and easily. By contrast, peer-
to-peer content, while also free, is not immediately available (content is always downloaded in
random chunks, which makes it difficult to estimate the download completion time), there is no
quality guarantee, and users are subject to an additional risk resulting from possible copyright
violations.
3.5 Cloud computing services have become widely utilized
Online services have also emerged significantly over the past 15 years: examples include data storage,
and remote applications ranging from email or online photo albums, to suites of business applications
including spreadsheets and word processing. Because the content and applications are centralized in
the network, rather than distributed at its edges, this is often referred to as cloud computing.23Although the concept of cloud computing is not new, it has become more popular recently as the
barriers to take-up have been progressively lifted, thanks to the improvement of Internet access (e.g.
development of broadband, availability of cheap dedicated high capacity links), the improvement of
22Source:http://www.cisco.com/en/US/solutions/collateral/ns341/ns525/ns537/ns705/ns827/white_paper_c11-
481360_ns827_Networking_Solutions_White_Paper.html
23As defined by the National Institute of Standards and Technology, cloud computing is a model for enabling convenient, on-demand
network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that
can be rapidly provisioned and released with minimal management effort or service provider interaction. (See
http://csrc.nist.gov/groups/SNS/cloud-computing/)
8/6/2019 Analysys Mason Domestic Peering Paper
20/47
Overview of recent changes in the IP interconnection ecosystem | 17
.
security (in particular, enhanced encryption techniques), and the increase in awareness and confidence
in cloud-computing techniques.
YouTube is a good example of a popular cloud-computing application, where users store videos on
YouTubes servers, and access them remotely in a streaming mode. The email services offered by
Microsoft and others can also be seen as a very successful popularization of cloud-computing
services. For instance, Googles Gmail became available to the general public on February 7, 2007
(after an initial invitation-only beta release on April 1, 2004), and by December 2009 had already
gained 176 million users.24
This began a transformation of services that were formerly performed at the edge of the network and
perhaps involved some incidental Internet traffic, into ones that were performed in the network itself,
with integral Internet access and only intermittent processing at the edge. In addition to the examples
discussed above, applications such as word processing and spreadsheets began to be offered as
Internet services, threatening traditional software models. From the perspective of this paper, this
increased overall usage of the Internet, and also made consumers more sensitive to latency and other
quality measures that impacted their access to their cloud applications.
3.6 Internet connections are a key feature for many new devices and services
The past 15 years have seen the development (and the wide acceptance by the general public) of a
large variety of devices that are connected to the Internet to enable access to the new services
described above, as well as additional services that were unleashed in response to the new usage
patterns that resulted from these devices.
New connected devices
The release in 2007 of the first popular smartphone Apples iPhone marked the take-off of mobile
Internet access. More and more mobile devices then started to offer specific features adapted to
Internet browsing and data traffic (e.g. mobile applications). By June 2009, there were approximately
6.4 million active iPhone users in the US, with 88% of iPhone users accessing the Internet and 75% of
them downloading mobile applications.25 According to Validas, iPhone users in the US each
consumed 273MB of data per month on average, and 12% of iPhone users consumed at least 500MB
of data per month (as of February 2010).26
Most vendors today are offering similar large touch-screendevices with embedded Internet functionalities.
Netbooks are getting a mass-market status, and according to the research firm ABI Research, 35
million netbooks were shipped worldwide in 2009.27 Netbooks are specifically designed to connect
24Source: http://www.businessweek.com/technology/content/feb2010/tc2010029_989050.htm
25Source : http://blog.nielsen.com/nielsenwire/online_mobile/iphone-users-watch-more-video-and-are-older-than-you-think/
26Source : http://www.ilounge.com/index.php/news/comments/report-iphones-average-over-250mb-of-data-use-per-month/
27Source : http://www.computerworld.com/s/article/9140343/Linux_s_share_of_netbooks_surging_not_sagging_says_analyst
http://www.businessweek.com/technology/content/feb2010/tc2010029_989050.htmhttp://blog.nielsen.com/nielsenwire/online_mobile/iphone-users-watch-more-video-and-are-older-than-you-think/http://www.ilounge.com/index.php/news/comments/report-iphones-average-over-250mb-of-data-use-per-month/http://www.computerworld.com/s/article/9140343/Linux_s_share_of_netbooks_surging_not_sagging_says_analysthttp://www.computerworld.com/s/article/9140343/Linux_s_share_of_netbooks_surging_not_sagging_says_analysthttp://www.ilounge.com/index.php/news/comments/report-iphones-average-over-250mb-of-data-use-per-month/http://blog.nielsen.com/nielsenwire/online_mobile/iphone-users-watch-more-video-and-are-older-than-you-think/http://www.businessweek.com/technology/content/feb2010/tc2010029_989050.htm8/6/2019 Analysys Mason Domestic Peering Paper
21/47
Overview of recent changes in the IP interconnection ecosystem | 18
.
remotely to the Internet, either via a WiFi connection or a 3G connection, and are thus optimized for
cloud computing with relatively little memory for installed software.
Tablets can also support new usage intimately related to Internet connection, and the launch of
Apples iPad and other similar products supports the view that expectations are high for this market,
with more than 14 million iPads being sold in 2010.28
New video-streaming devices have also emerged that exclusively rely on a broadband connection:
examples include Vudu, the Boxee Box (so-called Social Media Center), etc. These boxes directly
enable video streaming to the TV, and video file-sharing in competition with pay-TV, and are
beginning to be embedded into other devices such as game consoles or TV sets.
New online multimedia services
Second, many new services related to communications and multimedia have been released that
substitute traditional telecom services with online equivalents. This evolution can be explained by the
fact that an embedded Internet connection can enable advanced features and reduce the costs of
carrying traffic.
In addition to these services offering video on demand, broadcast TV services can also be provided
over an Internet connection (IPTV), in particular as part of the triple-play offerings of telecom
operators.
Voice over IP (VoIP), which consists in delivering voice communication over IP networks, has a
tremendous potential, as it allows customers to make free or very cheap voice calls. So far, qualityissues have restrained a wide acceptance of VoIP, but increases in fixed network capacity are
changing the situation at a rapid pace. Moreover, the development of cheap mobile broadband offers
(unlimited data plans with mobile subscriptions) opens the way to VoIP over mobile: for instance,
the Line2 application for iPhone enables users to call using the data connection instead of the
traditional minutes included in the bundle and paid for by the user.
In summary, the development of electronic devices and services relying on Internet access has
generated significant changes in the Internet ecosystem, as market players have adapted to this
demand increase by modifying and investing in their network architectures: rolling out mobile
broadband networks, installing additional network capacity, and developing new forms of
interconnection as described below.
3.7 Conclusion
As explained previously, and illustrated in Figure 3.8, Internet traffic and revenues have been fed by
the simultaneous growing demand for Internet-based services and the development of Internet-based
solutions to answer and sustain this demand.
28Source: Apple financial results
8/6/2019 Analysys Mason Domestic Peering Paper
22/47
Overview of recent changes in the IP interconnection ecosystem | 19
.
Figure 3.8: Supply and
demand feed the growth
of Internet traffic and
revenues [Source:
Analysys Mason]
It is not only Internet traffic that has exploded over the past 15 years: so has the Internet market as a
whole. It is interesting to note that, while there has been a significant increase in the revenues
generated by the e-commerce marketand to a smaller scale by the online advertising and the device
manufacturing markets there has been a more measured increase in revenues from Internet access
itself. As shown in Figure 3.9 below, the revenues per month per Internet user generated by e-
commerce and online advertising significantly outweigh those from Internet access.
Figure 3.9: Evolution of the Internet ecosystem in terms of revenues per Internet user per month [Source:
PWC, US Department of Commerce, Bureau of economic analysis, Euromonitor]
Mobilebroadband
Netbooks
Willingnessto pay forcontent
Peer-to-peer
Cloudcomputing
Videostreaming
Inte
rnettraffic
andreve
nues
Demand
Supply
0
20
40
60
80
100
120
2000 2001 2002 2003 2004 2005 2006 2007 2008
$perInternetuserpermonth
Online advertising
E-Commerce
Devices
Internet access
8/6/2019 Analysys Mason Domestic Peering Paper
23/47
Overview of recent changes in the IP interconnection ecosystem | 20
.
4 Evolution of Internet interconnection
As discussed in the previous section, the trends in Internet usage, particularly high bandwidth video
traffic, have led to significant increases in traffic. This, in turn, has led to a significant increase inbandwidth capacity demands, which provides a corresponding increase in costs, and concerns over
latency, since streamed video content needs to be delivered with minimal delay. At the same time,
content is increasingly monetized, providing content providers with a means to help ensure high-
quality delivery to their users. Here we describe how the main Internet stakeholders have adapted to
these new bandwidth demands, how new players have emerged, and how interconnection
arrangements have evolved accordingly, changing the traditional hierarchy of the Internet ecosystem
in the process.
4.1 Description of the early Internet
The Internet ecosystem 15 years ago had a rather simple structure, with relatively clear and
hierarchical relationships between the different players: end users and content providers bought
Internet access from their local ISP; ISPs bought Internet transit from a limited number of backbone
providers to get access to the whole Internet; and backbone providers peered with each other at the
US-based NAPs to ensure the full extent of their Internet connectivity. The result was a hierarchy
focused largely in the US, with the backbones at the top, interconnecting with one another via peering,
and selling the resulting Internet access to downstream ISPs, who in turn sold access to their end
users. Figure 4.1 below illustrates the architecture of the early Internet 15 years ago. Annex A
provides more background on the relationships between the different players.
Figure 4.1: Simplified architecture of the early Internet [Source: Analysys Mason]
Content provider &aggregator
Backbone1
End-user
Backbone2
Backbone3
ISP 1
ISP 4
ISP 3
ISP 2
Internet Access
Transit
Peering
All traffic passes through thebackbone providers, that carry
the traffic between the ISPs
NAP
8/6/2019 Analysys Mason Domestic Peering Paper
24/47
Overview of recent changes in the IP interconnection ecosystem | 21
.
In these years, the online market was still very young and dial-up was the predominant means of
Internet access, imposing technical constraints on the services that could be provided. For these
reasons, content providers had a limited size. In this context, we should note that it is only 15 years
ago that the first of todays major Internet companies started to emerge: Amazon (founded in 1994),
Yahoo (1995), eBay (1995), Google (1998), etc.
The Internet ecosystem 15 years ago had a rather simple structure characterized by relatively
clear and hierarchical relationships, with end users and content providers buying Internet access
from ISPs, ISPs buying Internet transit from backbone providers, and backbone providers
peering with each other.
4.2 Internet exchange points (IXPs)
The nature of Internet exchange has been changing for the past 15 years. When the NSF decided to
commercialize the Internet backbone, it designated four dispersed Network Access Points (NAPs), in
San Francisco (operated by PacBell), Chicago (BellCore and Ameritech), New York (SprintLink) and
Washington DC (MFS). The NAPs were used for public peering, whereby traffic was exchanged
using a common switch, which soon congested due to the large volume of traffic. As the historical
home of the Internet, a significant amount of international traffic passed through the US at the time,
either to connect with US providers, or for transit, sometimes back to the same country of
origination.29 At the same time, the operators of the NAPs had the potential to favor their own
services: for instance, when the MAE-East NAP was owned by MCI WorldCom, it required the use of
MCI circuits to access MAE-East services.30
As a result, a migration began towards peering at IXPs around the world. IXPs, such as Equinix,
began to open large data centers, where operators could host their servers and/or connect to one
another using direct cross-connections. In some countries, non-profit IXPs run by a consortium of
members were established today several of the largest IXPs use this model, including LINX in
London, and AMS-IX in Amsterdam. The move to IXPs enabled what is known as private peering,
whereby service providers do not use a common switch and thus can control congestion over each
bilateral link with other service providers. The service providers could also sell or purchase transit
over these direct cross-connections at the IXPs. As a result, these IXPs became focal points for the
entire Internet ecosystem, where the stakeholders could buy and sell their services to one another at
relatively low cost.
There are clearly strong network effects that support the growth of an IXP. The more service
providers there are in an IXP and the more traffic they exchange, the more attractive it is for other
service providers to locate at the same IXP in order to arrange connectivity with the largest number of
29As all international ISPs at the time had to connect to the US for international transit, they sometimes used these links to exchange
domestic traffic, in order to avoid the cost of directly connecting to each other ISP in the country at a time when many domestic telecoms
markets were not yet competitive. This flow of traffic out of the country and then back in is often referred to as tromboning.
30See: Internet Service Providers and Peering, William B. Norton
8/6/2019 Analysys Mason Domestic Peering Paper
25/47
Overview of recent changes in the IP interconnection ecosystem | 22
.
service providers. However, the growth in traffic described above has provided a strong
countervailing pressure that has caused IXPs to proliferate across populated areas of countries and
regions. Having a dispersion of IXPs has benefits for backbone providers in a peering relationship, in
terms of reducing latency, ensuring quality of service (via redundancy) and controlling costs:
First, where the two parties in a transmission are in the same region, having a nearby IXP enables
the traffic to remain local, rather than tromboning to a more distant location. This enhances the
quality of service thanks to reduced latency. A certain level of redundancy is also provided by
there being multiple IXPs in different locations in the same region, giving some protection to
connectivity in case of natural disaster, for instance.
Second, where the two parties in a transmission are in different regions, having dispersed IXPs
enables backbone providers to effectively share traffic loads, whereby one provider will carry the
originating traffic and the other will carry the return traffic.
This second benefit results from what is sometimes referred to as deflection routing or more
commonly hot-potato routing, whereby each backbone will exchange the traffic with a peer at the
earliest exchange point (for further details see section A.2.3 of the Annex). When the traffic ratio (the
ratio of traffic flowing between peers in one direction compared to the traffic flowing in the other
direction) is reasonably even, the networks will evenly share the costs associated with carrying traffic
exchanged by their users, and thus the agreement is considered fair.
As a result, in their publicly available peering policies, backbone providers may require
interconnection in up to six different points in the US, for instance (typically with dedicated
connections in IXP locations), in order to reduce carriage requirements and balance traffic loads.Backbone providers will also specify a maximum traffic ratio in order to limit the traffic imbalance
between them and fairly split the costs of carrying traffic.31 These types of requirements are designed
to reflect comparable costs and benefits, so that the peering arrangement is mutually beneficial to both
parties.
While IXPs have clearly been beneficial to backbone providers in lowering costs and making more
efficient use of capacity, they have also enabled other trends whereby providers rely less on
backbones, as discussed below.
The development of IXPs has benefited backbone providers by improving the quality of service
and reducing traffic carrying costs. However, the benefits rely on roughly even traffic ratios
between the peers, as any significant traffic imbalance jeopardizes the peering principle that
each party should bear a fair share of the costs of carrying the traffic.
31See Annex A for an example of a current peering policy.
8/6/2019 Analysys Mason Domestic Peering Paper
26/47
Overview of recent changes in the IP interconnection ecosystem | 23
.
4.3 Internet service providers (ISPs)
ISPs have reacted to the changing mix of traffic described above in several ways, each of which
reduces their reliance on traditional backbones for transit services:
The rise in peer-to- peer usage has dramatically increased the traffic that ISPs customers are
exchanging between each other (acting in the role of both consumers and suppliers of content).
This has led to what is referred to as secondary peering.
At the same time, the rapid growth of centralized video content and traffic in sites such as
YouTube has led ISPs to connect directly to content providers. This typically results in a form of
interconnection known aspaid peering.
We describe each of these changes in turn.
First, the changing mix of traffic led the largest ISPs to reevaluate their need to purchase IP transit
from upstream backbone providers. Traditionally, IP transit was used as a means to access traditional
content providers, which may be connected to the upstream transit provider or another backbone. On
the other hand, IP transit is not necessary to access the customers of another ISP, particularly if that
ISP is located in the same IXP. As a result, many ISPs began to peer with one another, as shown in
Figure 4.2.
Figure 4.2: The principle
of secondary peering
[Source: Analysys
Mason]
This had two impacts for the ISPs. First of all, it lowered the cost of transit, because the traffic
exchanged via secondary peering no longer needed to be sent through the existing transit
relationship(s). As transit is based on a measure of traffic volume, this lowered costs. One estimate is
that up to 40% of a typical ISPs traffic in the US was peer-to-peer that could be exchanged using
secondary peering, with a corresponding reduction of cost.32 In addition, secondary peering reduced
latency for peer-to-peer traffic, as the traffic passed directly between the ISPs rather than via one or
32The Evolution of the US Internet Peering Ecosystem, William B. Norton, Equinix White Paper 11/19/2003. Norton was the Co-Founder
and the Chief Technical Liaison for Equinix. (See http://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdf)
Backbone
2
Backbone
1
ISP 1 ISP 2
Peering
TransitTransit
Secondary
peering
IXP
http://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdfhttp://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdfhttp://www.equinix.com/pdf/whitepapers/PeeringEcosystem.pdf8/6/2019 Analysys Mason Domestic Peering Paper
27/47
Overview of recent changes in the IP interconnection ecosystem | 24
.
more upstream backbones. ISPs that have engaged in such secondary peering are sometimes said to
purchase onlypartial transitfrom backbonesnamely only those destinations that they cannot access
via peering.
Second, the rapid growth of centralized video content and traffic in sites such as YouTube has led to a
different form of interconnection known aspaid peering. The genesis of paid peering between an ISP
and a content provider is similar to that of secondary peering between two ISPs, namely it is an
attempt to directly connect in order to reduce transit costs as well as the latency of transmissions.
However, there is a fundamental difference resulting from the ratio of traffic involved. When two
ISPs peer, it is likely that they will exchange roughly equal amounts of traffic, as their end users are
likely to have an equal propensity to generate and consume each others traffic, be it peer-to-peer or
interactive video games or just email. On the other hand, when an ISP and a content provider peer, the
traffic ratios will not be balanced, as for instance the request for a video consumes very little
bandwidth while the video itself takes far more bandwidth: we estimate that the streaming of a three-
minute video on YouTube generates around 35 times more downlink traffic than uplink traffic.33
As aresult, the ISP will deliver far less traffic to the content provider than it carries in return.
When the imbalance of traffic between service providers rises above a certain traffic ratio, the
convention is that the generator of the traffic pays the provider that carries the traffic for the
imbalance, in order to cover the cost of capacity needed. Nonetheless, paid peering is advantageous
for both parties. Previously, both the ISP and content provider used a transit connection to a backbone
to exchange traffic, and each party paid for the traffic that they sent and received. Now, by directly
interconnecting, they each lower their transit payments accordingly. Indeed, in this case the content
provider would have paid transit fees for the traffic that it delivered to the backbone, while the ISP
would have paid similar transit fees for receiving that traffic from the backbone. With paid peering,
the content provider pays the ISP directly for delivering the traffic, probably at a rate lower than it
would have paid for transit.
As a result of these actions on the part of ISPs to lower their transit fees, interconnection has begun to
look like Figure 4.3 below. In order to reduce transit fees, ISPs have begun peering with one another,
typically at IXP locations for efficiency. At the same time, content providers have begun to
interconnect directly with ISPs, using paid peering to compensate for the imbalance of traffic across
the connections. This not only further lowers the ISPs transit fees, but also results in fees from the
paid peering. These new interconnections come at the expense of backbones, which, as a result, endup selling less transit to the ISPs and content providers.
33The video streaming request sent by a user to YouTube generates around 430 kbytes of uplink traffic, while the resulting video stream
generates around 15 Mbytes.
8/6/2019 Analysys Mason Domestic Peering Paper
28/47
Overview of recent changes in the IP interconnection ecosystem | 25
.
Figure 4.3: Interconnection possibilities that emerged in the past 15 years [Source: Analysys Mason]
Many route around options have emerged for ISPs and content providers to carry Internet
traffic while avoiding Internet backbone transit costs. These options include secondary peering
between ISPs and partial transit agreements. Traffic imbalances have also led to the
development of paid peering, which is advantageous for ISPs and content providers.
4.4 Content providers
Paid peering with ISPs is only one of the means that content providers are using to lower their costs
efforts that go hand in hand with ways to reduce latency and thereby improve the experience for their
users. Another technique used for this purpose is called web caching. A web cache used by an ISP
stores a copy of a web page or video requested by an end user. The ISP can then use the cache to
satisfy subsequent requests for the same content, instead of reloading it from the original source. In
other words, if 500 users view the same popular video, it can be downloaded once from the original
content provider over the transit connection, and the next 499 times it will be viewed from the web
cache within the ISPs network, significantly lowering the amount of traffic over the transit
connection, and therefore reducing transit payments.
Web caching may be used by ISPs, as described above, but this technique is more typically used
either by third-party content delivery networks (CDNs) or by larger content providers, which are
effectively building their own delivery networks in order to deliver content closer to (or from within)
the ISPs networks. Commercial CDNs effectively compete with backbones in order to deliver their
clients content directly to ISPs. CDNs ensure that the content of their clients is distributed to end
users with good performance, independently of the location of the end user. To fulfill this role, a CDN
typically owns a network of caching servers, connected by fiber, which feed copies of content to ISPs
Content provider &
aggregator
Backbone1
End-user
Backbone
2
Backbone
3
ISP 4
ISP 3
ISP 2
IXP
Internet Access
Transit
Peering
Secondary peering
Partial transit Paid peering
ISP
Singleentity
8/6/2019 Analysys Mason Domestic Peering Paper
29/47
Overview of recent changes in the IP interconnection ecosystem | 26
.
at a limited number of points of presence (PoPs) where networks interconnect.34 The largest CDNs
include Akamai (founded in 1998) and Limelight (2001). According to one estimate, CDNs deliver
close to 10% of all Internet traffic.35 See Figure 4.4 below for a stylized diagram of how a CDN
interacts with the other players in the Internet ecosystem. The use of CDNs to carry content traffic
effectively reduces the traffic to be carried by the backbone provider, and avoids any traffic imbalance
issues. However, at the same time it reduces the transit revenues of backbone providers.
Figure 4.4: Structure of a content delivery network [Source: Analysys Mason]
Content providers themselves may use a similar configuration to build a content delivery network for
their own content. For instance, Google has embarked on a Google Global Cache project with similar
goals to a CDN, in this case to locate Google servers within ISP networks.36 The difference from a
CDN is that Google servers are dedicated to Google content. Other content providers such as
Microsoft are also building such networks to deliver content, such as Microsofts video games. The
difference in the traffic carried over the backbone can be significant: Arbor Networks estimates that
Google traffic represents at least 6% of global Internet traffic, and more than 60% of that traffic is
delivered to ISPs over direct connections rather than via third-party providers.37 Figure 4.5 shows how
the percentage of Google traffic using private peering has increased in recent years.
34CDNs usually try to connect directly to ISPs for cost and performance reasons, but may need in some instances to buy transit from
backbone providers (when they do not own the fiber network), and also buy transit from ISPs (when they cannot get free peering from
large ISPs).
35ATLAS Internet Observatory, 2009 Annual Report, Arbor Networks, Inc., University of Michigan, Merit Network, Inc., Pre-Publication Draft,
page 15.
36See http://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdf
37 Source: How Big is Google?, Craig Labovitz, http://asert.arbortnetworks.com/2010/03/how-big-is-google/
Contentprovider
Backbone
ContentDelivery
Network
End-user
Backbonenetwork B
Backbonenetwork A
RegionalISP
Mobileoperator
Aggregation & AccessAggregation & Access
RegionalISP
http://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdfhttp://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdfhttp://www.afnog.org/afnog2008/conference/talks/Google-AFNOG-presentation-public.pdf8/6/2019 Analysys Mason Domestic Peering Paper
30/47
Overview of recent changes in the IP interconnection ecosystem | 27
.
Figure 4.5: Percentage of
Google traffic using
direct peering [Source:
Arbor Networks38]
Even smaller content providers can also use technology to lower their costs, avoiding not just transit
charges, but also the use of CDNs. For instance, Vudu delivers high-quality video on demand to end
users TVs via a dedicated set-top box that is external or integrated into a TV or Blu-ray player. Each
device has a large hard drive, which contains the first few seconds of each title in order to enable the
title to begin as soon as it is chosen. The rest of the chosen content is pieced together via an Internet
connection to the device, using peer-to-peer networking from other Vudu devices that have stored thatcontent. As a result, a user is able to watch high-definition movies with no delay with as little as a
4Mbit/s broadband connection. In this way, Vudu uses its customers own broadband connections to
deliver content to other users in lieu of a more expensive CDN or backbone service.
As a result of the volume of traffic that they deliver, and the associated cost of delivering that traffic,
content providers have a strong incentive to build out their networks in order to reduce their transit
requirements and deliver the traffic directly to the ISP network, if not from within the ISPs network.
As a result of the increasing monetization of content, providers also have the means to engage in such
network build-outs. But even if CDNs are increasingly used or built to deliver Internet content to
ISPs, in particular to deliver the growing share of video traffic for which caching is appropriate,
backbones and/or ISPs still provide significant infrastructure to deliver the content. As explained
earlier, CDNs typically distribute their caching servers into specific PoPs to connect with backbones
and ISPs networks. The selection of these points by the CDNs is optimized in terms of coverage, i.e.
CDNs will primarily set up PoPs in regions concentrating significant amounts of Internet traffic.
However, backbones and ISPs then ensure the delivery of content to the end users from these PoPs to
the end users, regardless of final location.
38 http://gigaom.com/2010/03/17/stat-shot-googles-growing-infrastructure-advantage/
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
A
pr-08
Jun-08
Aug-08
O
ct-08
Dec-08
Feb-09
A
pr-09
Jun-09
Aug-09
O
ct-09
Dec-09
Feb-10
Percentage
http://gigaom.com/2010/03/17/stat-shot-googles-growing-infrastructure-advantage/http://gigaom.com/2010/03/17/stat-shot-googles-growing-infrastructure-advantage/8/6/2019 Analysys Mason Domestic Peering Paper
31/47
Overview of recent changes in the IP interconnection ecosystem | 28
.
Overall, the traffic carriage effort supported by backbones and ISPs is significant in comparison with
the CDNs. CDNs typically connect at tens of PoPs: for instance, Limelight notes that in 2010 its
network consisted of 75 PoPs where it interconnected globally, in order to deliver traffic to more than
900 ISPs and backbones. This is broadly similar to the number of PoPs that each single major
backbones typically possesses.39 However, the backbones and/or ISPs carry the Internet traffic further
into the network, in order to reach the 24 000 central offices40 counted in the US (as of February
2010), from where it is carried over the last mile to end users for DSL services. 41 On average, each
PoP may connect to up to 300 or more central offices, requiring significant network resources to carry
the content from the CDN to the last-mile connection.
In conclusion, as explained in the previous section, there are now many situations where content
providers can bypass the backbone providers transit fees. Content providers also have the opportunity
to rely on their own private network or a third-party CDN to carry their traffic to the ISP. Either way,
the consequence has been the continuing trend of providers relying less on backbones, which may
now be avoided for a significant amount of traffic between content providers and end users, even ifthey keep a key role in delivering content to and from end-user networks.
Content providers can employ many different options to carry their Internet traffic to the ISP
(e.g. using CDNs or building their own private networks), and can leverage their expanding
market power and financial resources to rely less on backbone providers.
4.5 Impact on backbone providers
From the backbone providers point of view, these market evolutions have had impacts at two levels :
First, changes in traffic patterns have led to changes in the arrangements between backbone
providers and their traditional customers, the ISPs and content providers. These changes have
impacted the traditional hierarchy of the Internet ecosystem and put downward pressure on transit
prices.
Second, although many traditional backbone customers are relying less on backbones to
accommodate changing traffic patterns, the same trends are impacting the traffic that they still
deliver to the backbones as customers, which puts pressure on the relationship between backbone
39The number of PoPs on backbones is usually not publicly revealed. However, studies providing solid estimates suggest that large
backbones in the US typically own around 50 to 100 PoPs
(Seehttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdf )
40Central offices are network facilities where end users lines get locally connected to the core network (local loop) for xDSL services. To
deliver the traffic to end users, ISPs carry aggregated Internet traffic to the central offices, where it is then distributed to the end users
over the last portion of the network (last mile).
41Source: FCC, Trends in Telephone Service, FCC, September 2010
(http://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdf ). Similar arrangements are true for cable
networks.
http://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdfhttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdfhttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.fcc.gov/Daily_Releases/Daily_Business/2010/db0930/DOC-301823A1.pdfhttp://www.cs.washington.edu/research/networking/rocketfuel/papers/sigcomm2002.pdf8/6/2019 Analysys Mason Domestic Peering Paper
32/47
Overview of recent changes in the IP interconnection ecosystem | 29
.
providers. In particular, the increase in content traffic being delivered to end users is impacting
the conditions under which backbones agree to peer with one another.
With respect to the first issue, the changes impacting ISPs and content providers as described in the
previous sections lead to a reduction in demand for transit. This is due to the changes with regards to
backbones customers at both ends of the value chain: namely ISPs and content providers, which are
able to self-supply a growing portion of their needs, and which are also entering into multiple
agreements with peers and other suppliers. This is true not just at a national level, but also
internationally as more and more traffic is regionalized through large IXPs that reduce a reliance on
international transit for traffic exchange. As a result, this puts pressure on transit prices (shown below)
as backbones effectively face greater competition from the self-supply of their customers.
Further, the changing nature of Internet traffic also impacts the relationships between the backbone
providers themselves. In particular, with the increase of bandwidth-intensive content, particularly
video, comes an increase in an imbalance of traffic sent from the content providers to the customers of
the ISPs. As a result, without a change in the mix of customers, backbones with content providers as
customers will send more traffic to backbones with ISP customers. As noted in Section 4.2, peering
policies typically specify a maximum traffic ratio governing the traffic flow between peering partners;
this has been a feature of peering policies at least since backbones began making their peering policies
publicly available a decade ago.42 Increases in traffic ratios resulting from delivering content can
begin to overrun the traffic ratio requirements in the corresponding peering policies.
With the default hot-potato routing that is common for peering arrangements, the sender of the traffic
quickly hands over the traffic to the receiving network for the latter to carry across the network to the
recipient, as described below in Annex A.2.3. As a result, a high traffic ratio can impose costs on thereceiving network, as it carries far more traffic than it hands off to the other network in return.
Specifically, the backbone providing the content will quickly hand it to the backbone with ISP
customers that are receiving the content. As a result, the latter backbone has greater network costs in
transporting the incoming traffic from the backbone providing the content, which could be seen as a
cost subsidy from the former to the latter.
As these costs result from content being exchanged near to the source under hot-potato routing, one
solution is what is known as cold-potato, or best-exitrouting, in which the sending network keeps the
high-bandwidth traffic on its own network longer before exchanging it closer to the consumers.
However, given the ratio between PoPs and the number of distribution points in an ISPs network,
cold-potato routing may still not alleviate the imbalance of costs, implying that the ISP may subsidize
the delivery of traffic from the content provider, on behalf of its customers who may already be
paying for the relevant content (directly or via associated advertising).
42For instance, in January 2001 WorldCom introduced a publicly-available peering policy specifying the requirements that a network must
meet in order to peer with World Com. This policy included a clause with the following requirement: The ratio of the aggregate amount of
traffic exchanged between the Requester and the WorldCom Internet Network with which it seeks to interconnect shall be roughly
balanced and shall not exceed 1.5:1.
See http://drpeering.net/a/Internet_Service_Providers_and_Peering___Peering_Policy_files/UUNET%20Global%20Site%20WorldCom%
20Policy%20for%20Settlement-Free%20Interconnection%20with%20Internet%20Networks.htm
8/6/2019 Analysys Mason Domestic Peering Paper
33/47
Overview of recent changes in the IP interconnection ecosystem | 30
.
Other solutions to the imbalance of costs issue include paid peering, discussed above, or even moving
to a transit relationship whereby the backbone whose traffic causes the imbalan