Global R&E Network Interconnects:AMPATH Update
Ernesto Rubi, Sr. Network EngineerErnesto Rubi, Sr. Network EngineerAMPATH International ExchangeAMPATH International ExchangeFlorida International UniversityFlorida International University
Miami, FLMiami, FL
Quilt Peering WorkshopSt. Louis, Missouri
October 11 - 12, 2006
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Quilt Peering Workshop, October 2006
OutlineOutline
• Network OverviewNetwork Overview• Layer 1/2 Peering - UpdateLayer 1/2 Peering - Update• GLIF/GOLE Activities/Infrastructure:GLIF/GOLE Activities/Infrastructure:
• The WHREN-LILA projectThe WHREN-LILA project• AtlanticWaveAtlanticWave
• Open Exchange Points in the Western Open Exchange Points in the Western HemisphereHemisphere
2
Quilt Peering Workshop, October 2006
AMPATH International AMPATH International Exchange Point CharacteristicsExchange Point Characteristics
Current Technologies:Current Technologies: Various 10 GigE circuits currently in place to other Various 10 GigE circuits currently in place to other
networks include:networks include: 10 GigE to FLR ( Florida’s RON )10 GigE to FLR ( Florida’s RON ) 10 GigE to UltraLight’s 10 GigE to UltraLight’s 10 GigE to AtlanticWave ( Week ending Oct. 13th )10 GigE to AtlanticWave ( Week ending Oct. 13th ) 10 GigE to SC 06 SCInet - Delivering A-Wave to SC.10 GigE to SC 06 SCInet - Delivering A-Wave to SC.
NRENs peer directly with AMPATH via international NRENs peer directly with AMPATH via international circuits.circuits.
NRENs peer with Abilene via multi-hop eBGP; however this NRENs peer with Abilene via multi-hop eBGP; however this will change with A-Wave implementation. will change with A-Wave implementation.
Direct peering to other NRENs available via AtlanticWave, Direct peering to other NRENs available via AtlanticWave, WHREN-LILA circuit, FLR or UltraLight.WHREN-LILA circuit, FLR or UltraLight.
Located at Miami’s Tier-1 NAP: NAP of the AmericasLocated at Miami’s Tier-1 NAP: NAP of the Americas
Quilt Peering Workshop, October 2006
AMPATH International AMPATH International Exchange PointExchange Point
AMPATH7609
Lucent CBX 500
Cisco ONS15454
STM-16 PortCurrently configured as VC4-8c with
1.22 Gbps effective bandwidth.
MIAMI
FoundryFastIron
RNP
ANSP
CLARA
Abilene
A-Wave
NLR
UltraLight
WHREN-LILA Cisco ONS 15454
SONET
Gigabit Ethernet
10 Gigabit Ethernet
EtherChannelAll peer VLANs
A -WaveTo SC 2006
SCINet/Show Floor
WHREN-LILA WHREN-LILA IRNC Award 0441095 IRNC Award 0441095
5-year NSF Cooperative Agreement5-year NSF Cooperative Agreement Florida International University (IRNC awardee)Florida International University (IRNC awardee) Corporation for Education Network Initiatives in California Corporation for Education Network Initiatives in California
(CENIC)(CENIC) Project support from the Academic Network of Sao Paulo Project support from the Academic Network of Sao Paulo
( ANSP ) ( ANSP ) CLARA, Latin AmericaCLARA, Latin America CUDI, MexicoCUDI, Mexico RNP, BrazilRNP, Brazil REUNA, ChileREUNA, Chile
Links Interconnecting Latin America (LILA) aims to Links Interconnecting Latin America (LILA) aims to Improve connectivity in the Americas through the Improve connectivity in the Americas through the establishment of new inter-regional linksestablishment of new inter-regional links
Western-Hemisphere Research and Education Networks Western-Hemisphere Research and Education Networks (WHREN) serves as a coordinating body whose aim is to (WHREN) serves as a coordinating body whose aim is to leverage participants’ network resources to foster leverage participants’ network resources to foster collaborative research and advance education collaborative research and advance education throughout the Western Hemisphere throughout the Western Hemisphere
5
Links Interconnecting Latin AmericaLinks Interconnecting Latin America
Miami - Sao Paulo link: Miami - Sao Paulo link: 1.2Gbps by year end, evolving 1.2Gbps by year end, evolving to 2.5Gbpsto 2.5Gbps
Connects State of Sao Paulo Connects State of Sao Paulo academic network (ANSP) and academic network (ANSP) and Exchange Point, regional Exchange Point, regional network (CLARA), Brazilian network (CLARA), Brazilian NREN (RNP), other NREN (RNP), other international networksinternational networks
San Diego - Tijuana link: San Diego - Tijuana link: operating at 2 x 1 Gbps, operating at 2 x 1 Gbps, providing dedicated GigE links providing dedicated GigE links to regional network (CLARA) to regional network (CLARA) and Mexican NREN (CUDI)and Mexican NREN (CUDI)
East and west coast East and west coast connectivity to I2 Abilene and connectivity to I2 Abilene and other US and global R&E other US and global R&E networksnetworks
http://www.whren-lila.net
Current RedCLARA topologyCurrent RedCLARA topology
backbone ring: backbone ring: 155 Mbps155 Mbps(Global Crossing + (Global Crossing + Cisco)Cisco)
Access links of 10 Access links of 10 to 45 Mbpsto 45 Mbps
Connection to Connection to Europe (GÉANT) at Europe (GÉANT) at 622 Mbps from 622 Mbps from BrazilBrazil
WHREN-LILA Miami-Sao Paulo WHREN-LILA Miami-Sao Paulo Distributed Exchange PointDistributed Exchange Point
Sao PauloCisco 15454
RedCLARArouter
RNPswitch
ANSP switch
RNPswitch
RNProuter
WHREN-LILA
RedCLARA
MiamiCisco 15454
Redundant dark fiber
USP
HEPGridUERJ
SPRACEUNESP
BARUERICOTIA
SDH/SONET
N X GbE
Sao Paulo has become a distributed exchange point, interconnecting ANSP, CLARA, RNP, supporting a distributed CMS Tier-2 facility,
with inter-regional connections to U.S. R&E networks and E.U. GEANT
Quilt Peering Workshop, October 2006
Sao Paulo Exchange Point Sao Paulo Exchange Point CharacteristicsCharacteristics
Built using Cisco ONS 15454 SDH Chassis Built using Cisco ONS 15454 SDH Chassis STM circuit between Miami and Sao Paulo is configured STM circuit between Miami and Sao Paulo is configured
currently as VC4-8c ( ~ 1.22 Gbps ) linear circuitcurrently as VC4-8c ( ~ 1.22 Gbps ) linear circuit Drawback: Static Provisioning, tear-down required if peers Drawback: Static Provisioning, tear-down required if peers
require additional bandwidthrequire additional bandwidth VCAT and sw-LCAS can be used to accommodate network VCAT and sw-LCAS can be used to accommodate network
dynamics ( SC ‘06 )dynamics ( SC ‘06 ) STS-v ( VCAT ) circuits along with sw-LCAS allows network STS-v ( VCAT ) circuits along with sw-LCAS allows network
reconfiguration without service interruption and provides reconfiguration without service interruption and provides greater circuit granularitygreater circuit granularity
Ethernet L2 End to End Service currently provided using Ethernet L2 End to End Service currently provided using ML-1000-2 cards installed on ONSML-1000-2 cards installed on ONS Currently 2 ML cards provide a total of 4 GigE ports ( can add Currently 2 ML cards provide a total of 4 GigE ports ( can add
more if needed )more if needed ) GigE ports can be configured with various 802.1Q VLANs as GigE ports can be configured with various 802.1Q VLANs as
well as transport user-defined VLANs using QinQ mapping well as transport user-defined VLANs using QinQ mapping Bridge groups provide the necessary mappings via internal Bridge groups provide the necessary mappings via internal
ONS cross-connect fabric to previously provisioned SDH ONS cross-connect fabric to previously provisioned SDH circuits circuits
CoS/QoS and other policing methods can be applied to ML CoS/QoS and other policing methods can be applied to ML ports to conform to requirements as well as QinQ VLAN ports to conform to requirements as well as QinQ VLAN mappingsmappings
Quilt Peering Workshop, October 2006
RNPJuniperM320
RedCLARA RNP
COTIA
USP
RNPCatalyst
3550
ANSP
1 x GigaEth
SDH/SONET
2 x GigaEth
Sao Paulo / Miami - Via 2 x STM-4 / VC4-8cSao Paulo / Miami - Via 2 x STM-4 / VC4-8c(( Current as of 8/14/06 ) Current as of 8/14/06 )
ML-1000-2Slot 5 / Port 0
AMPATH 7609
SDH Circuit:VC4-8C 1.2 Gbps
WHREN-LILA ONS 15454
CLARA GSR
RNP 3550
ML-1000-2Slot 5 / Port 0
VLAN 2008VLAN 2007
VLANs 2006-2008
VLAN 2006
ANSP Foundry
Quilt Peering Workshop, October 2006
Future Implementation GoalsFuture Implementation Goals
11
Flexibility for end user requirementsFlexibility for end user requirements ONS ML GigE ports assigned to individual end users to ONS ML GigE ports assigned to individual end users to
accommodate end2end flows ( LHC experiments, network accommodate end2end flows ( LHC experiments, network research )research )
To accommodate 2 x 1 Gbps flows and an additional 300 To accommodate 2 x 1 Gbps flows and an additional 300 Mbps flow:Mbps flow: Circuit provisioned using 2 x STS-12c-2v (1 Gbps) VCAT Circuit provisioned using 2 x STS-12c-2v (1 Gbps) VCAT
groups, an additional STS-3c-2v ( 300 Mbps ) VCAT group groups, an additional STS-3c-2v ( 300 Mbps ) VCAT group and/or multiple STS-1c-2v circuits over the STM-16 circuit. and/or multiple STS-1c-2v circuits over the STM-16 circuit.
sw-LCAS can be used to accommodate network dynamics. sw-LCAS can be used to accommodate network dynamics. This means bandwidth can be re-provisioned while not This means bandwidth can be re-provisioned while not affecting data flows.affecting data flows.
Achieves: Granularity to 51 Mbps ( I.e: efficient use of Achieves: Granularity to 51 Mbps ( I.e: efficient use of bandwidth )bandwidth )
Traffic EngineeringTraffic Engineering Peers are can be guaranteed bandwidth at port speeds or Peers are can be guaranteed bandwidth at port speeds or
given CoS via policing which ensures better than best given CoS via policing which ensures better than best effort availability for sensitive data flowseffort availability for sensitive data flows
VLAN mappings can be preserved throughout; a VLAN mappings can be preserved throughout; a combination of Ethernet Wire Services or Ethernet Relay combination of Ethernet Wire Services or Ethernet Relay Service can be implemented using ML card to ensure flow Service can be implemented using ML card to ensure flow separation and data security/integrityseparation and data security/integrity
Quilt Peering Workshop, October 2006
Sao Paulo / Miami - Via VC4-8c or Full STM-16Sao Paulo / Miami - Via VC4-8c or Full STM-16(( Near Future ) Near Future )
RNPJuniperM320
RedCLARA RNP
COTIA
USP
RNPCatalyst
3550
ANSP
1 x GigaEth
SDH/SONET
2 x GigaEth
Slot 5, Port 0
Slot 5, Port 1Slot 6, Port 0
To AMPATH 7609Using EtherChannel?
VLAN 2008VLAN 2007
VLAN 2006
Policing/CoS Applied at Input of ML Card in SP
STS-v-x VCAT Circuit using sw-LCAS
Quilt Peering Workshop, October 2006
Thank You!Thank You!
• Questions/CommentsQuestions/Comments
Ernesto RubiErnesto [email protected]@cs.fiu.edu
AMPATH Engineering GroupAMPATH Engineering [email protected]@ampath.net
Visit us at:Visit us at:http://www.ampath.nethttp://www.ampath.net
http://www.whren-lila.nethttp://www.whren-lila.net
13