Content Peering William B. Norton Co-Founder and Chief Technical Liaison Content Peering Forum September 18, 2002 © Equinix 2002.

Content Peering

William B. Norton

Co-Founder and Chief Technical Liaison

Content Peering ForumSeptember 18, 2002

© Equinix 2002

Internet Researcher

• White Papers: Focused studies on Peering

1. Identify Relevant Internet Operations Topic

2. Speak with prominent Peering Coordinators

3. Write/Evolve Draft White Paper

4. Walk Peering Coordinators through paper

5. Goto Step 3

Most widely read WP: “Internet Service Providers and Peering”

Peering Research• Two-Year Study w/~200 Peering Coordinators

– What is Peering?– How do you determine who to peer with?– When does Peering make sense?– When does Peering NOT make sense?– What is the Process of Peering?– How is Peering implemented?

Goal: Document Internet Operations Practices I will share with you today The Motives, The Methods,

The Mindset of the Peering Coordinator

Summary Findings of Peering Research….

No Internet Operations DocumentationTo make things more difficult, subtle language differences!

Peering is a game of relationships

Level Setting Talk…

• Content Peering is…– Content-Heavy Companies leveraging Simple

Internet Peering, Peering as ISPs do.– A Routing Optimization (direct vs. through an

Intermediary)

• Not…– Content Peering Alliance

(see http://www.content-peering.org/peering.html)

– Or any end-to-end Content Alliance

To this end I will share…

Agenda

1. Definitions• Transit & Peering Terminology and Financial Cost Models

2. Introduction to the Peering Breakeven Analysis• Specific Examples (San Jose Peering) leading to Generalizations

3. Peering Process• 3 stages of Peering

4. Top 5 Reasons Not to Peer5. Peering Resources Available to you

Peering Makes Sense for Large Scale Content Players

WP: “Internet Service Providers and Peering”

Definitions• Gives us tools (lexicon) to facilitate discussion• Market Confusion over misuse of terms:

• Terminology critical to have meaningful discussion on Peering

TransitPeeringTransportTier 1 ISPEffective Peering Bandwidth

Defs: ISPs

Understand that…

1. Def: The Internet is a network of networks

2. Def: Internet Service Providers sell access to the Internet

3. Internet Service Providers must themselves connect to the Internet.

• How do they connect to the rest of the Internet?

Definition of Transit

1) ISP A buys Transit Service

Upstream Transit ProviderISP A

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit

Providers

Def: Transit is the business relationship whereby one ISP

announces (usually sells) reachability to the *entire* Internet to a customer.

•$$$ ? Typically usage-based•Pricing ’01: $100-$1200/Mbps•Volume based on 95th Percentile measure•Transit is Simple, Convenient:• Upstream handles the delivery of packets to the Internet by some meansUsage: “I’m purchasing transit from Level 3.”

3) Traffic Flows

INTERNET

NETWORKS

2) ReachabilityAnnouncement

Cost of Transit Traffic Exchange

2001 Pricing Sampling Range: $100-$1200/MbpsMy Financial Models used $388/MbpsSource: 2002 Survey Range: $50-$1200/Mbps 95th Percentile

Blended Avg: ~$200/Mbps wholesale Transit

Isn’t Transit Good Enough?

Transit Cost Model

$0

$50

$100

$150

$200

$250

$300

$350

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43

# Mbps Exchanged

$/M

bp

s

Transit Cost per Mbps

Mbps $/Mbps1-3Mbps $3004-10Mbps $22010-30Mbps $20530-50Mbps $18050-75Mbps $14675-100Mbps $129100-300Mbps $112300-1000Mbps $105

Transit Costs

Traffic Growth

Content Heavy ISP

Access Heavy ISP

UPSTREAMS

V

IGMBT

Streaming Media

56k 384k 1.5m

$$$

$$$

Yahoo! Broadcast: 100,000+ concurrent unicast streams15 million streaming hrs/mo, 1300 Live events/day

Access

AOL+ DSL: 1,000/dayRoadrunner Cable Modems: 1M subscribersKaZaa!!!! Two main motivations for Peering…

Streaming:BroadcastTelephonyVideoMultimediaInteractiveGamingSPOT Events

Seek transportInterconnection $

x

Why Peer? 2 Motivations for Peering1.Financial: Reduce load on expensive Transit

serviceTraffic src/destMeasure vs IntuitUsage-based Billing

2.Engineering: Lower latency

1st Stage of Peering:Top 10 destination ISP list

Transit$$$

Transit$$$

ISP A

ISP B

Transit ISP

Def: Peering…

Definition of PeeringDef: Peering is the business relationship whereby ISPs reciprocally announce reachability to each others’ transit customers.

Peering

USNet EastNetWestNet

Peering

TransitRoutingTables

•Peering is *not* a transitive relationship•Peering *does not* provide access to the entire Internet

Cost of Peering: Def. Transport+Port

Usage: “I buy transit from UUNet and Peer with EastNet, WestNet.”

The Costs of Peering

)__(

__

___

____

:

_

CostsEquipmentRouterR

FeesRackMonthlyr

FeesPortIXMonthlyp

IXIntoCostsTransportMonthlyt

where

RrptCostsPeering

Observation: Transport Prices have dropped like a rock. Observation: New Router prices have dropped like a rock.Observation: Used Router Market is also very healthy (cheap).

Graphically…

Transit ISP

ISP A

ISP B

Transit $$$

Transit $$$

1) Transport into Exchange

2) Rack Space atExchange PointFor Router

3) Router at IX

4) Ethernet Switch Port (/Private Cross Connect)

R

RX

Peering $Ethernet-Based Peering Model

When does Peering make sense? (SJ Mkt)Flat Monthly Fees vs. Metered Monthly Fee

San Jose Market Prices for Peering

Transit ISP

ISP A

ISP B

Transit $$$

Transit $$$

1) Transport into ExchangeOC3@$2500/mo

2) Rack Space atExchange PointFor Router½ rack $2500/mo

3) Switch Port on Public Peering FabricGigE

R

RX

Peering $

Total Cost of Peering$7000/month

Which is more cost effective?Peering or Transit? Allocate…

4) Cisco 7500 $2000/mo

Cost of Traffic Exchange Peering vs. solely Transit

Transit ISP

ISP A

ISP B

R

RX

Peering $

Cost of Peering: $7000/month

1 Mbps

Unit Cost of Traffic ExchangeIn Peering Relationship:

$7000=$7000 per Mbps1 Mbps

1 Mbps

TRANSIT~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit

Transit ISP

ISP A

ISP B

R

RX

Peering $

Cost of Peering: $7000/month

2 Mbps

Unit Cost of Traffic ExchangeIn Peering Relationship:

$7000=$3500 per Mbps2 Mbps

2 Mbps

TRANSIT~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit

Transit ISP

ISP A

ISP B

R

RX

Peering $

Cost of Peering: $7000/month

7 Mbps

Unit Cost of Traffic ExchangeIn Peering Relationship:

$7000=$1000 per Mbps7 Mbps

7 Mbps

TRANSIT~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit

Transit ISP

ISP A

ISP B

R

RX

Peering $

Cost of Peering: $7000/month

14 Mbps

Unit Cost of Traffic ExchangeIn Peering Relationship:

$4000=$500 per Mbps14 Mbps

14 Mbps

TRANSIT~ $200/Mbps

Cost of Traffic Exchange Peering vs. solely Transit

Transit ISP

ISP A

ISP B

R

RX

Peering $

Cost of Peering: $7000/month

70 Mbps

Unit Cost of Traffic ExchangeIn Peering Relationship:

$7000=$100 per Mbps70 Mbps

70 Mbps

TRANSIT~ $200/Mbps

OC-3 Peering vs. Transit in San JoseOC-3 Peering vs. Transit

$0

$100

$200

$300

$400

$500

$600

$700

$800

10 20 30 40 50 60 70 80 90 100 110 120 130 140 150

# Mbps Exchanged

$/M

bp

s

Peering

Transit

Mbps $/Mbps1-3Mbps $3004-10Mbps $22010-30Mbps $20530-50Mbps $18050-75Mbps $14675-100Mbps $129100-300Mbps $112300-1000Mbps $105

Transit Costs

Mbps Peering10 $70020 $35030 $23340 $17550 $14060 $11770 $10080 $8890 $78

100 $70110 $64120 $58130 $54140 $50150 $47

Generalized…

OC-12 Peering vs. Transit in San JoseMbps $/Mbps1-3Mbps $3004-10Mbps $22010-30Mbps $20530-50Mbps $18050-75Mbps $14675-100Mbps $129100-300Mbps $112300-1000Mbps $105

Transit Costs

Generalized…

OC-12 Peering vs. Transit

$0

$100

$200

$300

$400

$500

$600

$700

$800

$900

$1,000

10 30 50 70 90 110

130

150

170

190

210

230

250

270

290

310

330

350

370

390

410

430

450

470

490

510

530

550

570

590

610

#Mbps Exchanged

$/M

bp

s

Peering

Transit

$15/Mbps

Mbps Peering50 $190

100 $95150 $63200 $48250 $38300 $32350 $27400 $24450 $21500 $19550 $17

Peering Breakeven Effective Peering BandwidthDS-3 12 42.3OC-3 35 145.7OC-12 70 584.68

Effective Peering Range

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (axis)

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (EPB)

Definition: The Effective Peering Bandwidth is the maximum bandwidth available for peering, defined as the minimum of the available transport bandwidth and the usable bandwidth on the shared peering fabric.

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (minCost)

Definition: The Minimum Cost of Traffic Exchange is the unit cost of traffic exchange when the Effective Peering Bandwidth is fully utilized.

Cos

t of

Tra

ffic

Exc

hang

e in

$/M

bps

Amount of Traffic Exchanged (in Mbps)

Transit Price per Mbps

Peering Price per Mbps

Peering Breakeven Point(Peering=Transit)

EffectivePeering Bandwidth (in Mbps)

Min Cost ofTraffic Exchange(in $/Mbps)

Effective Peering Range

Transit Cheaper-Peering Cheaper

Peering Analysis Graph (EPR)Definition: The Effective Peering Range (EPR) is the range in which peering at an Internet Exchange makes sense (financially), measured as the range between the Peering Breakeven Point and the Effective Peering Bandwidth.

When does Peering Make Sense?

Peering Breakeven Effective Peering BandwidthDS-3 12 42.3OC-3 35 145.7OC-12 70 584.68

Effective Peering Range

The 3 Stages of Peering

The 3 Stages of Peering

Interviews with 200 ISP Peering Coordinators revealed…

3 General Phases of Peering:

1) Identification of Potential Peer – the who

2) Initial Contact and Qualification – the why

3) Implementation Discussions – the how

I. Phase 1: Identification of Peer: Traffic Engineering Data Collection and Analysis

Motivations:• Reduce load on expensive Transit service• Traffic src/dest• Measure or Intuit

• Usage-based Billing• 2nd Goal: Lower latency• ResultTop 10 list (see next page)• Part of larger business deal

Seekinterconnection

Transit$$

Transit$$

ISP A

ISP B

Transit ISP

Sample Top 10 Destination ListA S N u m b e r M b p s D e s t i n a t i o n I S P C o n t a c t

6 1 7 2 2 4 . 3 5 H O M E - N E T - 1 [ H O M E - N O C - A R I N ]

7 0 1 8 . 9 0 A L T E R N E T - A S [ I E 8 - A R I N ]

1 6 6 8 8 . 1 4 A O L - P R I M E H O S T [ A O L - N O C - A R I N ]

4 7 6 6 7 . 0 8 A P N I C - A S - B L O C K [ S A 9 0 - A R I N ]

3 3 2 0 5 . 1 2 R I P E - A S N B L O C K 4 [ R I P E - N C C - A R I N ]

5 7 7 4 . 2 4 B A C O M [ E Q - A R I N ]

6 3 2 7 3 . 9 0 S H A W F I B E R [ I A S - A R I N ]

1 3 . 8 9 B B N P L A N E T [ C S 1 5 - A R I N ]

7 0 1 8 3 . 6 6 A T T - I N T E R N E T 4 [ J B 3 3 1 0 - A R I N ]

9 3 1 8 3 . 1 3 A P N I C - A S - 3 - B L O C K [ S A 9 0 - A R I N ]

5 7 6 9 2 . 6 7 V I D E O T R O N [ N A V 1 - A R I N ]

6 8 3 0 2 . 3 0 H C S N E T - A S N B L K [ M D 2 0 5 - A R I N ]

9 2 7 7 2 . 2 2 A P N I C - A S - 3 - B L O C K [ S A 9 0 - A R I N ]

1 0 9 9 4 2 . 0 8 T A M P A 2 - T W C - 5 [ J D 6 - A R I N ]

1 2 3 9 2 . 0 5 S p r i n t L i n k [ S P R I N T - N O C - A R I N ]

I n t e r n e t S e r v i c e P r o v i d e r A

Phase 1: Identification of Potential

Peer

Part of BroadBusiness

Relationship?

Dom inant TrafficFlow?

TraversingExpensive Transit

C ircuit?

Yes

W ill Peering have apositive affect on m y

network?

Yes

Large new custom erim pact?

Yes

Yes

Proceedto Phase

2:Contact

Peer

Yes

II. Phase 2: Contact & Qualification, Initial Peering

DiscussionHow to make contact with potential peer ISP?

1. E-mail person or peering@<ispdomain>.net

2. Exchange point participant list

3. Tech-c/admin-c from DNS/ASN registries

4. Engineering Forums NANOG, IETF, RIPE, etc.

5. Trade shows: speakers and booth staff

6. Target ISP sales force

7. Target ISP NOC

II. Phase 2: Contact & Qualification, Initial Peering

DiscussionOnce contact is made…1. Sometimes Mutual NDA2. Exchange BiLateral Peering Agreement (BLPA)3. Traffic Data justification shared

One basis: Peering iff PeeringCost < TransitSavings?

4. Requirements Exchange(e.g. Must be at n Public Peering Points, xMbps, private

peering migration strategy, etc.)

Either Party may walk away….. If still interested, implementation discussion…

Phase 2: Contact and Qualification

InitialContact

LargerBusiness

Transaction

peering@ or

personalcontact

ExchangePoint

Contact list

tech-c oradm in-cin DNS/

ASNRegistry

OperationsForum

TradeShows

SalesForce

Finding the Right Contact

SignNDA,see

polic ies

Sharetrafficdata,BLPA

Do both parties findm otivation to continuepeering discussion?

Close discussion

Proceed toPhase 3:

Im plem entation

Discussion

YesNo

III. Phase 3: Implementation Discussions

How to interconnect?

Direct Circuit-based Interconnection

VS.

Exchange-Based Interconnection

Cost Comparison at n=5

costDCfn()=(n-1)*C/2 C=OC-3 @ $2,500n=5costDC=(4)*$2500/2costDC=$5,000/mo

costExchfn()=BDC+(n-1)*x/2+Racks BDC=OC-12 @ $5,000n=5, 1 Rack@$1000costExch=$5,000+(4)(250/2)+$1000costExch=$6,500/mo

More expensive to use Exchange-BasedInterconnection Strategy at n=5. N>5?

SSS

GGG

UUU

A A A

S

G

U

A

CCCC

S

G

U

A

C

OC-12

OC-12

OC-12

OC-12

OC-12

Exchange-based vs. Direct Circuit Interconnection

Cost Comparison of Interconnection Strategies

$-

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64

# of participants

Mo

nth

ly C

ost

of

Inte

rco

nn

ecti

on

Direct Circuits Model

MUX Big Pipes Model

Dark Fiber Model

9 Exchange Selection Criteria

1. Telecommunications Access Issues2. Deployment Issues (getting in & up)3. ISP Current Presences (there yet?)4. Operations Issues (restrictions?)5. Business Issues (neutrality/alignment)6. Cost Issues ($$)7. Credibility Issue (backing,attraction) 8. Exchange Population (side effect)9. Existing vs. Emerging Exchange?

Value of the Internet ExchangeVExchange

NParticipants

CostOf Coming In(Circuits+Routers+StaffTime)

VCapacity

The Exchange Startup Hump

First Carrier(s)First ISP(s)First CP(s) Critical Mass Point (Vexchange=CostExchange)

Lar

ge F

acili

ty S

calin

g

C o nte ntPro vid e r

C o nte ntPro vid e r

C o nte ntPro vid e r

ISP ISPISP

C a rrie r C a rrie r

C irc uitDe m a nd sTric kle Do wn

C =num b e r o f c o nte nt p ro vid e rs

I=n u m b e r o f I S P s

N =n u m b e r o fc a r r ie r s

Top 5 Reasons NOT to Peer

Top 5 Reasons not to Peer

1) Already get Traffic for “free” (through existing peering relationships)

Transit$$$

Yahoo!

Transit ISPEXODUS

Peering $

AOL

Top 5 Reasons not to Peer

2) Not True Peers• Traffic inequity

• Scale inequity• Not even investments in infrastructure• Form: “I don’t want to haul your traffic

around the globe”

Large Global Network Provider

SmallRegional

Player

Huge investment in Int’s circuits,100’s of routers and colo sites,Staff installs, peering negotiations, Millions of customers, etc.

Top 5 Reasons Not to Peer

3) Lack of Technical Competence

Troubleshooting network problems takes longer when the other ISP NOC and engineers lack the technical expertise during an outage…

Top 5 Reasons Not to Peer

4) Transit Sales Preferred

• We rather sell you transit…“Let me introduce you to our sales guys”

Top 5 Reasons Not to Peer5) BGP is Tough

“BGP? No ExpertiseNo measurements No Justification to hire expertsBGP?”

TransitService

ISP A

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit

Providers

INTERNET

NETWORKS

TransitService

ISP A

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit

Providers

INTERNET

NETWORKS

TransitService

ISP A

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit

Providers

INTERNET

NETWORKS

TransitService

ISP A

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit Provider

Upstream Transit

Providers

INTERNET

NETWORKS

Primary

BackupTransit

Seek transportInterconnection

$

x

Transit$$$

ISP B

Transit$$$

ISP A

Transit ISP

Transit$$$

ISP B

Transit$$$

ISP A

Transit ISP

Transit$$$

ISP BTransit$$$

ISP B

Transit$$$

ISP A

Transit$$$

ISP A

Transit ISP

PrimaryBackupTransit

Conceptual Hurdle Conceptual Hurdle ComplexSimple

6: personality

Top 5 Reasons Not To Peer

5+ Personality Clashes:

They don’t understand each other

and they didn’t like the interaction

Resources Available to Peering Coordinators

Resources Available for Peering Coordinators

• Gigabit Peering Forums

• Other White Papers document Peering Practices

• Peering Contact Database

Gigabit Peering Forums

Other White Papers

“Interconnection Strategies for ISPs”“Internet Service Providers and Peering”“A Business Case for Peering”“The Art of Peering: The Peering Playbook”“Do ATM-based Internet Exchange Points make

sense anymore?”“The Peering Simulation Game”

Freely available from the author: [email protected]

Peering Contact Database

For Peering Coordinators OnlyToss in your Business Card &Receive a copy of everyone’s Business CardsEvery 6 weeks (or so)

Managed as a community service.E-mail to [email protected](Or give me your business card)

Conclusions

• Language of the Peering Coordinator– Transit, Peering, Transport

• ISP Peering makes sense if you can offload x Mbps of traffic to peers…

• The Peering Process includes 3 Phases:– Identification of Peer– Contact and Qualification– Implementation of Peering

• These represent the Baseline Understandings of the Peering Coordinator

Acknowledgements?

Acknowledgements

For this white paper I’d like to thank a few folks in particular for their review, insights, and comments on this paper:

Dorian Kim (NTT/Verio), Ingrid Erkman (ICG), Dave McGaugh (ELI), Eric T. Bell (Time Warner Telecom), Chris Parker (StarNet), Brokaw Price (Yahoo!),Lane Patterson (Equinix), Jay Adelson (Equinix), Morgan Snyder (Equinix),John Hardie (Equinix), David Diaz (BellSouth), Joe Wood (Accretive Networks), Robert Seastrom (inter.net), Kevin Epperson (Level3), Petri Helenius (FICIX), Scott Sheppard (BellSouth), Ralph Doncaster (iStop.com), Leo Bicknell (ufp.org), Paul Vixie (vix.com), Ian Somerton and Dave Wodelet (Shaw/BigPipe), Tony Hain (Cisco), Jeff S. Wheeler (five-elements.com), Cliff Hafen, Dory Liefer, Shannon Lake (Omnivergent), Nenad Trifunovic (WorldCom), Andre Gironda (eBay), Jeb Linton (EarthLink), Daniel Golding (SockEye), Peter Moyer (Juniper), and others that preferred no recognition for their contributions to this paper.

Top 5 Reasons Not To Peer…

Questions?

Do these same Motivations to Peer apply to Content Companies?

Are the Financial Motivations more compelling thanthe Performance Improvement Motivations?