Global R&E Network Interconnects: AMPATH Update Ernesto Rubi, Sr. Network Engineer AMPATH International Exchange Florida International University Miami,

Global R&E Network Interconnects:AMPATH Update

Ernesto Rubi, Sr. Network EngineerErnesto Rubi, Sr. Network EngineerAMPATH International ExchangeAMPATH International ExchangeFlorida International UniversityFlorida International University

Miami, FLMiami, FL

Quilt Peering WorkshopSt. Louis, Missouri

October 11 - 12, 2006

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Quilt Peering Workshop, October 2006

OutlineOutline

• Network OverviewNetwork Overview• Layer 1/2 Peering - UpdateLayer 1/2 Peering - Update• GLIF/GOLE Activities/Infrastructure:GLIF/GOLE Activities/Infrastructure:

• The WHREN-LILA projectThe WHREN-LILA project• AtlanticWaveAtlanticWave

• Open Exchange Points in the Western Open Exchange Points in the Western HemisphereHemisphere

2

Quilt Peering Workshop, October 2006

AMPATH International AMPATH International Exchange Point CharacteristicsExchange Point Characteristics

Current Technologies:Current Technologies: Various 10 GigE circuits currently in place to other Various 10 GigE circuits currently in place to other

networks include:networks include: 10 GigE to FLR ( Florida’s RON )10 GigE to FLR ( Florida’s RON ) 10 GigE to UltraLight’s 10 GigE to UltraLight’s 10 GigE to AtlanticWave ( Week ending Oct. 13th )10 GigE to AtlanticWave ( Week ending Oct. 13th ) 10 GigE to SC 06 SCInet - Delivering A-Wave to SC.10 GigE to SC 06 SCInet - Delivering A-Wave to SC.

NRENs peer directly with AMPATH via international NRENs peer directly with AMPATH via international circuits.circuits.

NRENs peer with Abilene via multi-hop eBGP; however this NRENs peer with Abilene via multi-hop eBGP; however this will change with A-Wave implementation. will change with A-Wave implementation.

Direct peering to other NRENs available via AtlanticWave, Direct peering to other NRENs available via AtlanticWave, WHREN-LILA circuit, FLR or UltraLight.WHREN-LILA circuit, FLR or UltraLight.

Located at Miami’s Tier-1 NAP: NAP of the AmericasLocated at Miami’s Tier-1 NAP: NAP of the Americas

Quilt Peering Workshop, October 2006

AMPATH International AMPATH International Exchange PointExchange Point

AMPATH7609

Lucent CBX 500

Cisco ONS15454

STM-16 PortCurrently configured as VC4-8c with

1.22 Gbps effective bandwidth.

MIAMI

FoundryFastIron

RNP

ANSP

CLARA

Abilene

A-Wave

NLR

UltraLight

WHREN-LILA Cisco ONS 15454

SONET

Gigabit Ethernet

10 Gigabit Ethernet

EtherChannelAll peer VLANs

A -WaveTo SC 2006

SCINet/Show Floor

WHREN-LILA WHREN-LILA IRNC Award 0441095 IRNC Award 0441095

5-year NSF Cooperative Agreement5-year NSF Cooperative Agreement Florida International University (IRNC awardee)Florida International University (IRNC awardee) Corporation for Education Network Initiatives in California Corporation for Education Network Initiatives in California

(CENIC)(CENIC) Project support from the Academic Network of Sao Paulo Project support from the Academic Network of Sao Paulo

( ANSP ) ( ANSP ) CLARA, Latin AmericaCLARA, Latin America CUDI, MexicoCUDI, Mexico RNP, BrazilRNP, Brazil REUNA, ChileREUNA, Chile

Links Interconnecting Latin America (LILA) aims to Links Interconnecting Latin America (LILA) aims to Improve connectivity in the Americas through the Improve connectivity in the Americas through the establishment of new inter-regional linksestablishment of new inter-regional links

Western-Hemisphere Research and Education Networks Western-Hemisphere Research and Education Networks (WHREN) serves as a coordinating body whose aim is to (WHREN) serves as a coordinating body whose aim is to leverage participants’ network resources to foster leverage participants’ network resources to foster collaborative research and advance education collaborative research and advance education throughout the Western Hemisphere throughout the Western Hemisphere

5

Links Interconnecting Latin AmericaLinks Interconnecting Latin America

Miami - Sao Paulo link: Miami - Sao Paulo link: 1.2Gbps by year end, evolving 1.2Gbps by year end, evolving to 2.5Gbpsto 2.5Gbps

Connects State of Sao Paulo Connects State of Sao Paulo academic network (ANSP) and academic network (ANSP) and Exchange Point, regional Exchange Point, regional network (CLARA), Brazilian network (CLARA), Brazilian NREN (RNP), other NREN (RNP), other international networksinternational networks

San Diego - Tijuana link: San Diego - Tijuana link: operating at 2 x 1 Gbps, operating at 2 x 1 Gbps, providing dedicated GigE links providing dedicated GigE links to regional network (CLARA) to regional network (CLARA) and Mexican NREN (CUDI)and Mexican NREN (CUDI)

East and west coast East and west coast connectivity to I2 Abilene and connectivity to I2 Abilene and other US and global R&E other US and global R&E networksnetworks

http://www.whren-lila.net

Current RedCLARA topologyCurrent RedCLARA topology

backbone ring: backbone ring: 155 Mbps155 Mbps(Global Crossing + (Global Crossing + Cisco)Cisco)

Access links of 10 Access links of 10 to 45 Mbpsto 45 Mbps

Connection to Connection to Europe (GÉANT) at Europe (GÉANT) at 622 Mbps from 622 Mbps from BrazilBrazil

WHREN-LILA Miami-Sao Paulo WHREN-LILA Miami-Sao Paulo Distributed Exchange PointDistributed Exchange Point

Sao PauloCisco 15454

RedCLARArouter

RNPswitch

ANSP switch

RNPswitch

RNProuter

WHREN-LILA

RedCLARA

MiamiCisco 15454

Redundant dark fiber

USP

HEPGridUERJ

SPRACEUNESP

BARUERICOTIA

SDH/SONET

N X GbE

Sao Paulo has become a distributed exchange point, interconnecting ANSP, CLARA, RNP, supporting a distributed CMS Tier-2 facility,

with inter-regional connections to U.S. R&E networks and E.U. GEANT

Quilt Peering Workshop, October 2006

Sao Paulo Exchange Point Sao Paulo Exchange Point CharacteristicsCharacteristics

Built using Cisco ONS 15454 SDH Chassis Built using Cisco ONS 15454 SDH Chassis STM circuit between Miami and Sao Paulo is configured STM circuit between Miami and Sao Paulo is configured

currently as VC4-8c ( ~ 1.22 Gbps ) linear circuitcurrently as VC4-8c ( ~ 1.22 Gbps ) linear circuit Drawback: Static Provisioning, tear-down required if peers Drawback: Static Provisioning, tear-down required if peers

require additional bandwidthrequire additional bandwidth VCAT and sw-LCAS can be used to accommodate network VCAT and sw-LCAS can be used to accommodate network

dynamics ( SC ‘06 )dynamics ( SC ‘06 ) STS-v ( VCAT ) circuits along with sw-LCAS allows network STS-v ( VCAT ) circuits along with sw-LCAS allows network

reconfiguration without service interruption and provides reconfiguration without service interruption and provides greater circuit granularitygreater circuit granularity

Ethernet L2 End to End Service currently provided using Ethernet L2 End to End Service currently provided using ML-1000-2 cards installed on ONSML-1000-2 cards installed on ONS Currently 2 ML cards provide a total of 4 GigE ports ( can add Currently 2 ML cards provide a total of 4 GigE ports ( can add

more if needed )more if needed ) GigE ports can be configured with various 802.1Q VLANs as GigE ports can be configured with various 802.1Q VLANs as

well as transport user-defined VLANs using QinQ mapping well as transport user-defined VLANs using QinQ mapping Bridge groups provide the necessary mappings via internal Bridge groups provide the necessary mappings via internal

ONS cross-connect fabric to previously provisioned SDH ONS cross-connect fabric to previously provisioned SDH circuits circuits

CoS/QoS and other policing methods can be applied to ML CoS/QoS and other policing methods can be applied to ML ports to conform to requirements as well as QinQ VLAN ports to conform to requirements as well as QinQ VLAN mappingsmappings

Quilt Peering Workshop, October 2006

RNPJuniperM320

RedCLARA RNP

COTIA

USP

RNPCatalyst

3550

ANSP

1 x GigaEth

SDH/SONET

2 x GigaEth

Sao Paulo / Miami - Via 2 x STM-4 / VC4-8cSao Paulo / Miami - Via 2 x STM-4 / VC4-8c(( Current as of 8/14/06 ) Current as of 8/14/06 )

ML-1000-2Slot 5 / Port 0

AMPATH 7609

SDH Circuit:VC4-8C 1.2 Gbps

WHREN-LILA ONS 15454

CLARA GSR

RNP 3550

ML-1000-2Slot 5 / Port 0

VLAN 2008VLAN 2007

VLANs 2006-2008

VLAN 2006

ANSP Foundry

Quilt Peering Workshop, October 2006

Future Implementation GoalsFuture Implementation Goals

11

Flexibility for end user requirementsFlexibility for end user requirements ONS ML GigE ports assigned to individual end users to ONS ML GigE ports assigned to individual end users to

accommodate end2end flows ( LHC experiments, network accommodate end2end flows ( LHC experiments, network research )research )

To accommodate 2 x 1 Gbps flows and an additional 300 To accommodate 2 x 1 Gbps flows and an additional 300 Mbps flow:Mbps flow: Circuit provisioned using 2 x STS-12c-2v (1 Gbps) VCAT Circuit provisioned using 2 x STS-12c-2v (1 Gbps) VCAT

groups, an additional STS-3c-2v ( 300 Mbps ) VCAT group groups, an additional STS-3c-2v ( 300 Mbps ) VCAT group and/or multiple STS-1c-2v circuits over the STM-16 circuit. and/or multiple STS-1c-2v circuits over the STM-16 circuit.

sw-LCAS can be used to accommodate network dynamics. sw-LCAS can be used to accommodate network dynamics. This means bandwidth can be re-provisioned while not This means bandwidth can be re-provisioned while not affecting data flows.affecting data flows.

Achieves: Granularity to 51 Mbps ( I.e: efficient use of Achieves: Granularity to 51 Mbps ( I.e: efficient use of bandwidth )bandwidth )

Traffic EngineeringTraffic Engineering Peers are can be guaranteed bandwidth at port speeds or Peers are can be guaranteed bandwidth at port speeds or

given CoS via policing which ensures better than best given CoS via policing which ensures better than best effort availability for sensitive data flowseffort availability for sensitive data flows

VLAN mappings can be preserved throughout; a VLAN mappings can be preserved throughout; a combination of Ethernet Wire Services or Ethernet Relay combination of Ethernet Wire Services or Ethernet Relay Service can be implemented using ML card to ensure flow Service can be implemented using ML card to ensure flow separation and data security/integrityseparation and data security/integrity

Quilt Peering Workshop, October 2006

Sao Paulo / Miami - Via VC4-8c or Full STM-16Sao Paulo / Miami - Via VC4-8c or Full STM-16(( Near Future ) Near Future )

RNPJuniperM320

RedCLARA RNP

COTIA

USP

RNPCatalyst

3550

ANSP

1 x GigaEth

SDH/SONET

2 x GigaEth

Slot 5, Port 0

Slot 5, Port 1Slot 6, Port 0

To AMPATH 7609Using EtherChannel?

VLAN 2008VLAN 2007

VLAN 2006

Policing/CoS Applied at Input of ML Card in SP

STS-v-x VCAT Circuit using sw-LCAS

Quilt Peering Workshop, October 2006

Thank You!Thank You!

• Questions/CommentsQuestions/Comments

Ernesto RubiErnesto [email protected]@cs.fiu.edu

AMPATH Engineering GroupAMPATH Engineering [email protected]@ampath.net

Visit us at:Visit us at:http://www.ampath.nethttp://www.ampath.net

http://www.whren-lila.nethttp://www.whren-lila.net

13