Modern Carrier Strategies for Traffic Engineering

Modern Carrier Strategies for Modern Carrier Strategies for Traffic EngineeringTraffic Engineering

Dr. Vishal Sharma Principal ConsultantMetanoia, Inc.Voice: +1 408 394 6321Email: [email protected] Web: http://www.metanoia-inc.com

Metanoia, Inc.Critical Systems Thinking™

© Copyright 2002All Rights Reserved

Modern Carrier Strategies for TE 2


Copyright 2002, All Rights Reserved

Basic Service Provider Goals

The two fundamental tasks before any service provider:

Deploy a physical topology that meets customers’ needs

Map customer traffic flows on to the physical topology

Earlier (1990s) the mapping task was uncontrolled!

By-product of shortest-path IGP routing

Often handled by over-provisioning




The Early Years (< 1994-95): Routed Network Topology

IP Router

Network Cloud

IP Router

IP RouterIP Router

RouterInterconnections




The Early Years (< 1994-95): A Stacked View

IP Routers

DCS/DXC

TDM overcopper (T1/T3)

FDDI rings(100 Mb/s)

Router

MUX

SDH/SONET

-- Service creation-- Pkt. switching-- Stat muxing-- Connectivity

-- Speed match I/Fs

-- TDM transport-- Fault isolation-- Restoration




TE in Carrier Networks: Traditional Routed Core (pre 1994-95)

Prior to advent of ATM ...

... IP metrics were the only means available to control traffic distribution through IP networks

Approach was ad-hoc

Observe traffic flow through network

Adjust weight of links with load lower/higher than desired

Overprovision network as, needed




Two Paths to TE in IP Networks With increase in traffic, emergence of ATM, and higher-speed

SONET, two approaches emerged

Use a Layer 2 (ATM) network

Build ATM backbone

Deploy complete PVC mesh, bypass use of IP metrics

TE at ATM layer

With time, evolve ATM to MPLS-based backbone

Use only Layer 3 (IP) network

Build SONET infrastructure

Rely on SONET for resilience

Run IP directly on SONET (POS)

Use metrics (systematically) to control flow of traffic (more on this later)




Genesis of the ATM-Core

Growth in traffic needed faster backbones (> T3/45 Mb/s)

Denser backbones metric manipulation impractical

IP routers lagged: offered only DS3 I/Fs & s/w forwarding

ATM emerged, was designed for WAN from start

In 1994-95 had OC-3, and later OC-12 I/Fs available

Allowed carriers to redesign their networks for high-speeds

As an evolutionary step, SPs moved to a switched ATM core




ATM-based Cores (mid to late 1990s)

Router

ATM

MUX

SDH/SONET

-- Service creation-- Pkt. switching-- Stat muxing-- Connectivity

-- Traffic engg.-- Hardware fwding

-- Speed match I/Fs

-- TDM transport-- Fault isolation-- Restoration




Physical Topology with an ATM Core

POP i+1

POP j

POP N

POP 1

POP 2

POP i

OC-12

OC-3

OC-3

Router

ATM Switch

ATM Core

OC-3 ATMSAR I/F




Logical Topology with an ATM Core

ATM-core (usually) fully owned by SP

Dedicated (usually) to supporting IP backbone

Utilized ATM UBR or ATM VBR-rt/CBR, depending on classes of traffic

B

A C

D

Primary PVC Mesh

Secondary PVC Mesh

PVC 1'

PVC 2'

PVC 4'

PVC 3'

PVC 5'

PVC 1

PVC 2

PVC 3

PVC 4

PVC 5B

AC

D

ATM PVC Layout L3 Logical Topology




Genesis of the IP-over-SONET/SDH Approach

Desire to minimize # of network layers

Easier management

Simpler operation

Potentially scalable

Belief that high-speed SONET/SDH I/Fs would become available with advances in components (vindicated with time)

Dictated partly by how (in time) a carrier’s network evolved




SONET/SDH-based Cores (mid-to-late 1990s and beyond)

Router

MUX

SDH/SONET

-- Service creation-- Pkt. switching-- Stat muxing-- TE w/ metrics or MPLS

-- Speed match I/Fs

-- Framing-- Fault isolation-- Restoration(moving away)

DWDM

-- B/w on existing plant




Physical Topology with SONET/SDH Core

POP i+1

POP j

POP N

POP 1

POP 2

POP i

OC-3/12 orSTM-1/4

Router

SONET/SDH Core

OC-3 SONET/STM-1 SDH I/F

OC-48 BLSR/MS-SPRing

OC-3/13 UPSR/SNCP Ring

Point-to-point SONET/SDH framed link

DXC




Ckt. 1

Ckt. 2

Ckt. 3

Ckt. 4

Ckt. 5

Logical Topology with SONET/SDH Core

SONET/SDH infrastructure (usually) owned by SP

Logical links between POP routers realized over a physical SONET/SDH circuit going over a fiber path

Parallel logical links (physically disjoint) provisioned b/w each router pair

A

B D

C

B

A C

D

Parallel Logial Links forLoad Sharing

Each logical link provisionedfor 2X the bandwidth

SONET/SDH Circuit LayoutL3 Logical Topology




Global Crossing IP Backbone Network

100,000 route miles 27 countries 250 major cities5 continents200+ POPs

Courtesy: Thomas Telkamp, GBLX




Global Crossing (GBLX): A Bit of History

First independent global fiber network

Launched operations -- March 1997

First segment turned on -- May 1998

Expanded network & svcs. by acquisitions & JVs

Frontier Telecommunications, Sept 1999

Racal Telecom, Nov 1999

Hutchison Global Crossing, Jan 2000

IXNET/IPC, June 2000

International network, worldwide reach

100,000 route miles, 27 countries, 250 major cities

195 POPs (mid 2001)




Global Crossing IP Network

OC-48c/STM-16c (2.5Gbps) IP backbone

Selected 10Gbps links operational (e.g. Atlantic)

Services offered

Internet access & Transit services

IP VPNs -- Layer 3 and Layer 2

MPLS and DiffServ deployed globally

Edge Equipment

Core Equipment

Cisco GSR 12000/12400[12.0(17) SI]

Cisco 7500/7200 ESR, OSRJuniper M10/20/40




Global Crossing: Network Design Philosophy

Ensure there are no bottlenecks in normal state

On handling congestion

Prevent via MPLS-TE

Manage via Diffserv

Over-provisioning

Well traffic engineered network can handle all traffic

Can withstand failure of even the most critical link(s)

Avoid excessive complexity & features

Makes the network unreliable/unstable




Global Crossing’s Approach: Big Picture

WebServer

HR

DR BR

AR

CR

WR

DR

HR BR

AR

CR

WR

DR

HR BR

AR

CR

WR

EthernetSwitch

ModemBank

To other ISPs

To Customers

POP1

POP2

POP3

AR = Access Router

BR = Border Router

CR = Core Router

HR = Hosting Router

WR = WAN Router

DR = DSL Aggregation

OC-3/OC-12

OC-12/OC-48

OC-48/OC-192




TE in the US IP Network: Deployment Strategy

Decision to adopt MPLS for traffic engineering & VPNs

Y2000: 50+ POPs, 300 routers; Y2002: 200+ POPs

Initially, hierarchical MPLS system 2 levels of LSPs

Later, a flat MPLS LSP full mesh only between core routers

Started w/ 9 regions -- 10-50 LSRs/region 100-2500 LSPs/region

Within regions: Routers fully-meshed

Across regions: Core routers fully-meshed

Intra-region traffic ~Mb/s to Gb/s, Inter-region traffic ~ Gb/s

Source [Xiao00]




Design Principles: Statistics Collection

A

B

C

LSP1 = 15 Mb/s

LSP2 = 10 Mb/s

LSP3 = 10 Mb/s

Statistics on individual LSPs can be used to build matrices

A

B

C

25 Mb/s

25 Mb/s

Using packets, we do not know traffic individually to B & C




Design Principles: LSP Control & Management

B

A

D

D

B

OC-48

OC-192

10% in usebefore new req.

New RequestA to D = 2.2 Gb/s

New LSP takeslonger path

Links utilization ismore balancedManually move traffic away from

potential congestion via ERO

B

A

D

D

B

B

A

D

D

B

OC-192

2 LSPs of 1.2Gb/s each

LSPs split acrossalternate routes

Lowered load, greaterheadroom to grow

Load splittingratio = 0.5 each

OC-48

Adding new LSPs with a configured load splitting ratio




Global Crossing’s Current LSP Layout and Traffic Routing

Region 1 Region 2

Region 3

Region 4

POP1POP3

POP4

POP5POP2

Full LSP Meshin Core

Core LSP betweenWRs in POPs 1 & 5

Source

Destination




Sprint (FON): A Bit of History

A century of evolution ...

1899: Brown Telephone Co., Abilene, KS

1976: United Telephone, Kansas City, MS, 3.5M customers

1984: Began building first, all-digital, US-wide, fiber-optic network

1986: United + GTE merge LD subsidiaries US Sprint

1992: United buys GTE’s stake, renaming co. to Sprint Corp.

2002: ~$23B revenue, 23M customers, 70 countries, 80,000 employees

110,000+ route miles in the long distance (LD) network

34,000+ in US, 78,000+ in rest of the world

Transport infrastructure common to voice, ATM, & IP network

Provides considerable leverage, as we’ll see later




Sprint (FON): IP Network Timeline

'92 '93 '95 '96 '97 '98 '99 '01 '02

First IXC w/ Internetsvc. on T1 network

All DS3 IP networkDEC FDDI Gigaswitch POP

Work with Cisco for nextrouter for OC-3 backbone

Service via native IP bboneOC-12 4F-BLSR deployedCisco GSR tested/deployed

GigaPOP bboneGSR in POPOC-3 WAN

Deploy OC-48POS over DWDM

Deploy GSR12016

OC-192 TAT linksOC-192 in EuropeAll bbone routersGSR12416s

Expand to Asia,South America




SprintLinkTM IP Backbone Network

19+ countries

30+ major intl. cities5 continents(reach S. America as well)

400+ POPs

Courtesy: Jeff Chaltas Sprint Public Relations

Represents connectivity only (not to scale)

110,000+ route miles (common with Sprint LD network)




SprintLinkTM IP Network

Tier-1 Internet backbone

Customers: corporations, Tier-2 ISPs, univs., ...

Native IP-over-DWDM using SONET framing

4F-BLSR infrastructure (425 SONET rings in network)

Backbone

US: OC-48/STM-16 (2.5 Gb/s) links

Europe: OC-192/STM-64 (10 Gb/s) links

DWDM with 8-40 ’s/fiber

Equipment

Core: Cisco GSR 12000/12416 (bbone), 10720 metro edge router

Edge: Cisco 75xxx series

Optical: Ciena Sentry 4000, Ciena CoreDirector




SprintLinkTM IP Design Philosophy

Large networks exhibit arch., design & engg. (ADE) non-linearities not seen at smaller scales

Even small things can & do cause huge effects (amplification)

More simultaneous events mean greater likelihood of interaction (coupling)

Simplicity Principle: simple n/wks are easier to operate & scale

Complexity prohibits efficient scaling, driving up CAPEX and OPEX!

Confine intelligence at edges

No state in the network core/backbone

Fastest forwarding of packets in core

Ensure packets encounter minimal queueing




SprintLinkTM Deployment StrategyL2 failure detection triggersswitchover before L3 converges

ZA

Parallel links 50% utilizationunder normal state

1

2

3

4

SONET framing forerror detection

LineCard

LineCard

SONETOverheadIP Data




SprintLinkTM Design Principles

Great value on traffic measurement & monitoring

Use it for

Design, operations, management

Dimensioning, provisioning

SLAs, pricing

Minimizing the extent of complex TE & QoS in the core




Sprint’s Approach to Monitoring

AccessRouter

AccessRouter

AccessRouter

BackboneLinks

Peering LinksProbe

BackboneRouter

Customers Customers Customers

Adapted from [Diot99]

Analysis platform located at Sprint ATL, Burlingame, CA




Sprint Approach to TE

Aim: Thoroughly understand backbone traffic dynamics

Answer questions such as:

Composition of traffic? Origin of traffic?

Between any pair of POPs

What is the traffic demand?

Volume of traffic?

Traffic patterns? (In time? In space?)

How is this demand routed?

How does one design traffic matrics optimally?




Obtaining Traffic Matrices between POPs

A

B

C

D

1.1.1.1

1.1.1/24

SADA

IP Packet DestinationSubnet

POP1POP2

POP3 POP4

DA

1.1.1.1

Exit POP

POP4POP1

POP2

POP3

POP4

ProtocolExitPOP

# pktsBuildTable

City A

City B

City C

City D

City A City B City C City D

City A

City B

City C

City D

City A City B City C City D

TrafficMatrices

ByProtocol

By Timeof Day

Combine data,Obtain matrix




A Peek at a Row of a Traffic Matrix

Summary of Data CollectedAdapted from [Bhattacharya02]

Distribution of aggregate access traffic across other POPs in the Sprint backbone

Peer 1

Peer 2

Web 2

Web 1

ISP

To Backbone

Sprint POP under study




Applications of Traffic Matrices

Traffic engineering

Verify BGP peering

Intra-domain routing

SLA drafting

Customer reports




Acknowledgements

Thomas Telkamp, Global Crossing

Robert J. Rockell, Jeff Chaltas, Ananth Nagarajan, Sprint

Steve Gordon, Cable and Wireless

Jennifer Rexford, Albert Greenberg, Carsten Lund, AT&T Research

Wai-Sum Lai, AT&T

Fang Wu, NTT America

Arman Maghbouleh, Alan Gous, Cariden Technologies

Yufei Wang, VPI Systems

Modern Carrier Strategies for Traffic Engineering

Technology

atm core pvc

critical systems thinkinglate

connectivity atm

atm coreoc

carrier networks

atm network use

atm ubr

emergence of atm