ICFA Standing Committee on ICFA Standing Committee on Interregional Connectivity (SCIC) Interregional Connectivity (SCIC) Harvey B. Newman Harvey B. Newman California Institute of Technology California Institute of Technology CHEP2009, Prague, March 23 2009 CHEP2009, Prague, March 23 2009 Global Networks for HEP in 2009 World Internet Connection Density 2008 http://www.chrisharrison.net/projects/InternetMap/ The World At Night http://www.bertc.com/subfour/truth/nightworld.htm http://monalisa.caltech.edu:8080/Slides/Public/SCICReports2009Final http://monalisa.caltech.edu:8080/Slides/Public/SCICReports2009Final
39
Embed
ICFA Standing Committee on Interregional Connectivity (SCIC)
World Internet Connection Density 2008 http://www.chrisharrison.net/projects/InternetMap/. The World At Night http://www.bertc.com/subfour/truth/nightworld.htm. ICFA Standing Committee on Interregional Connectivity (SCIC). Global Networks for HEP in 2009. Harvey B. Newman - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Three 2009 Reports: An Eventful YearThree 2009 Reports: An Eventful YearRapid Progress, Dawn of an Era; Deepening Digital DivideRapid Progress, Dawn of an Era; Deepening Digital Divide
Main ReportMain Report: “Networking for HEP”: “Networking for HEP” [HN, A. Mughal et al.][HN, A. Mughal et al.] Includes Updates on the Digital Divide, World Network Includes Updates on the Digital Divide, World Network
Status; Brief Updates on Monitoring, Advanced Status; Brief Updates on Monitoring, Advanced Technologies; Technologies; Focus on Digital Divide IssuesFocus on Digital Divide Issues
3939 Annexes: A World Network Overview Annexes: A World Network Overview Status and Plans of Nat’l & Regional Networks, HEP Labs, Status and Plans of Nat’l & Regional Networks, HEP Labs, & Optical Net Initiatives & Optical Net Initiatives [[35 Updated in 200935 Updated in 2009]]
Monitoring Working Group ReportMonitoring Working Group Report [R. Cottrell, U. Kalim] [R. Cottrell, U. Kalim] Also See: Also See: TERENA TERENA 2008 Compendium2008 Compendium ((www.terena.nlwww.terena.nl)): : R&E Networks in EuropeR&E Networks in Europe http://internetworldstats.comhttp://internetworldstats.com: Worldwide Internet Use: Worldwide Internet Use OECD Broadband PortalOECD Broadband Portal http://www.oecd.org/sti/ict/broadbandhttp://www.oecd.org/sti/ict/broadband
Broadband: the new Digital Divide ?Broadband: the new Digital Divide ? SCIC 2003 Digital Divide ReportSCIC 2003 Digital Divide Report [A. Santoro et al.][A. Santoro et al.]
SCIC Report 2009 - Main Trends Accelerate:SCIC Report 2009 - Main Trends Accelerate: Dark Fiber Nets, Dynamic Circuits, 40-100GDark Fiber Nets, Dynamic Circuits, 40-100G
SCIC Report 2009 - Main Trends Accelerate:SCIC Report 2009 - Main Trends Accelerate: Dark Fiber Nets, Dynamic Circuits, 40-100GDark Fiber Nets, Dynamic Circuits, 40-100G
Current generation of 10 Gbps network backbones and major Int’l Current generation of 10 Gbps network backbones and major Int’l links arrived in 2002-8 in US, Europe, Japan, Korea; Now links arrived in 2002-8 in US, Europe, Japan, Korea; Now China China
Bandwidth Growth: from 16X to Bandwidth Growth: from 16X to 10,000X in 7 Yrs. >> Moore’s Law10,000X in 7 Yrs. >> Moore’s Law Proliferation of 10G links across the Atlantic & Pacific since 2005Proliferation of 10G links across the Atlantic & Pacific since 2005
Bandwidth for LHC well above 100 Gbps, in aggregateBandwidth for LHC well above 100 Gbps, in aggregate Rapid Spread of “Dark Fiber” and DWDM: Emergence of Rapid Spread of “Dark Fiber” and DWDM: Emergence of Continental, Continental,
Nat’l, State & MetroNat’l, State & Metro N X 10G “Hybrid” Networks in Many NationsN X 10G “Hybrid” Networks in Many Nations Demand may still exceed capacity by ~2010, in some regionsDemand may still exceed capacity by ~2010, in some regions
Point-to-point “Light-paths” for HEP and “Data Intensive Science”Point-to-point “Light-paths” for HEP and “Data Intensive Science” Now Dynamic Circuits, and Managed Bandwidth ChannelsNow Dynamic Circuits, and Managed Bandwidth Channels
Technology continues to drive Performance Higher, Costs Lower Technology continues to drive Performance Higher, Costs Lower Commoditization of GE now 10 GE ports on serversCommoditization of GE now 10 GE ports on servers Multicore processors with Multi-Gbyte/sec interconnectsMulticore processors with Multi-Gbyte/sec interconnects Cheaper and faster storage ($ < 100/Tbyte); 100+ Mbyte/sec disksCheaper and faster storage ($ < 100/Tbyte); 100+ Mbyte/sec disks
TransitionTransition to 40G, 100G links to 40G, 100G links from 2011 (on land), ~2012 (sub-sea) from 2011 (on land), ~2012 (sub-sea) OutlookOutlook:: Continued growth in bandwidth deployment & useContinued growth in bandwidth deployment & use
Core NREN Capacity of EU/EFTA
Countries 2004-2008 TERENA Compendium 2008:
www.terena.org/activities/compendium/100G
10G
1G
0.1G
20G
atbe
bgcy
czdk
ee
frde
grhu
isie
it ltlu
mtnl
noplfi lv
sksi
esse
chuk
ptro
LHC Optical Private Network (The LHCOPN)
LHC Optical Private Network (The LHCOPN)
Edoardo Martelli (CERN)
Close collaboration with CERN & GEANT2 (11 10G Links)
Working with GEANT2 to achieve fiber diversity for European circuits to/from CERN
Inter-Tier1 links for resilience, redundancy:
BNL, FNAL: US LHCNetCNAF, SARA, GridKa, IN2P3:
Cross-border dark fiberRAL (UK): 2nd 10G in ProcessNDGF, ASGC, Taiwan, TRIUMF:
Reduced BW backupPIC (Spain): Still no backup
Evolution to N X 10 Gbps: Fermilab, BNL, US LHCNet; SURFnet, GARR-X
GÉANT2 Pan-European BackboneDark Fiber Core
Among 19 Countries: Austria Belgium Croatia Czech Republic Denmark Finland France Germany Hungary Ireland Italy Netherlands Norway Slovakia Slovenia Spain Sweden Switzerland United KingdomC. Stover (DANTE)
34 NRENs, ~30M Users; 50k km Leased Lines 12k km Dark Fiber; Point to Point ServicesGN3 Next Gen. Network Proposed 9/2008
Projected Start Q2 2009
SNLL
PNNL
LIGO
INEEL
LANL
MIT
ANLAMES
LLNL
GA
DOE-ALB
JGI
LBNL
SLACCHI
IARC
DenverLNOC
NERSC
BNL
FNAL
CH
I-SL
NYC
NYC
CERN
Amsterdam
By 2006
US LHCNet 10Gb/sUS LHCNet 20 Gb/sInternational (high speed)10 Gb/s SDN core10G/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less
ESnet4 IP Core (10 Gbps)
ESNet4 Science Data Net Core (n*10 Gbps)
Redundant “light-paths” to BNL and FNAL; dark fiber to FNAL
Connections to ESnet MANs in NYC & Chicago 10 Gbps peerings with Abilene (2) and GEANT2
US LHCNet (4x10Gbps)
US LHCNet + ESnet4 Today
SURFNet
In 2007
US-LHCNet Plan 2008-10: 40, 60, 80
GbpsNY-CHI-GVA-AMS
In 2008
A Global Partnership of R&E Networks and Advanced R&D Projects Supporting the LHC Program
14 to 16 10G Trans- Atlantic Links in 2009
Partnership with Internet2, NLR, ESnet in the US
Computing Model Progress CMS Internal Review of Software and Computing
Computing Model Progress CMS Internal Review of Software and Computing
Implementation: US LHCNet Plan Phase 8 Implementation: US LHCNet Plan Phase 8 (2014): Transition to Full Use of 100G(2014): Transition to Full Use of 100G
Using OTU-4 Using OTU-4 (100G) Links(100G) Links+ Next-Gen. + Next-Gen.
Optical MuxesOptical Muxes
4 X 100G4 X 100GTrans-Trans-
AtlanticAtlantic
FollowingFollowingan 8 Phasean 8 Phase
Plan Plan 2007-20142007-2014
US LHCNet Bandwidth Roadmap Matches LHC Storage Roadmap
US LHCNet Bandwidth Roadmap Matches LHC Storage Roadmap
20082009
20102011
20122013
Atlas Disk Ratio wrt 2008
AVG Disk Ratio wrt 2008
USLHCNet Bandwidth
CMS Disk Ratio wrt 2008
0
1
2
3
4
5
6
7
8
9
10Atlas Disk Ratio wrt 2008
AVG Disk Ratio wrt 2008
USLHCNet Bandwidth
CMS Disk Ratio wrt 2008
2008 2009 2010 2011 2012 2013
Atlas Disk Ratio wrt 2008 1 1.85 3.39 4.77 6.16 9.51
AVG Disk Ratio wrt 2008 1 1.55 2.63 3.62 4.62 6.60
Czech Tier2:Czech Tier2: 1 Gigabit Lightpaths to the Tier1s 1 Gigabit Lightpaths to the Tier1s at Fermilab, BNL, Karlsruhe and Taiwan;at Fermilab, BNL, Karlsruhe and Taiwan;
Similar scheme: Similar scheme: it, br, nl, de, ….it, br, nl, de, ….
H. Sverenyak
2002 - 2004: Dark Fiber Links to Austria, Czech Republic, Poland
2005-6: Complete 1 GbE links to all main sites 2006: 10 GbE Cross-Border Dark Fiber to Austria & Czech
Republic; 8 X 10G over 224 km with Nothing In-Line Deonstrated
2007-8: Transition Backbone to 10G Done; All CB Dark Fibers to 10G
~10,000x Increase Since
2002
WeisHorvath http://www.sanet.sk/en/index.shtm
SLOVAK Academic Network January 2009:
Now ~All 10 GbE Switched Ethernet
SANET to Schools 1GE to
500 Schools
In 54 Cities
By 2012
The Emergence of “Hybrid” Networks The Emergence of “Hybrid” Networks With Dynamic Circuits with BW GuaranteesWith Dynamic Circuits with BW Guarantees
W. Johnston, ESnet W. Johnston, ESnet On Circuit-Oriented Network ServicesOn Circuit-Oriented Network Services
Traffic Isolation; Security; Deadline Scheduling; Fairness; High UltilizationTraffic Isolation; Security; Deadline Scheduling; Fairness; High Ultilization
US LHCNet Configuration (2H 2009)Non-stop Operation; Circuit-oriented Services US LHCNet Configuration (2H 2009)
Non-stop Operation; Circuit-oriented Services
Robust fallback at layer 1 + next-generation hybrid optical network: Robust fallback at layer 1 + next-generation hybrid optical network: Dynamic Dynamic circuit-oriented network services with BW guarantees circuit-oriented network services with BW guarantees
Emerging Emerging StandardsStandards
VCAT, LCASVCAT, LCAS
CIENA Core CIENA Core DirectorsDirectors
Equipment Equipment and link and link
RedundancyRedundancy
Also Internet2 and SINET3
(Japan)
2007 Outbound Traffic13.3 PBytes
77% on Circuits: Peak 18 Gbps
6.7 Gbps Average
2007 Outbound Traffic13.3 PBytes
77% on Circuits: Peak 18 Gbps
6.7 Gbps Average
Large Scale Flows Mostly Handled by Dynamic
CircuitsUsing Software by
Fermilab and Caltech
Large Scale Flows Mostly Handled by Dynamic
CircuitsUsing Software by
Fermilab and Caltech
Traffic on CircuitsTraffic on Circuits
CMS data transfer between FNAL and UNL using Internet2's
DCN and LambdaStation Software (FNAL + Caltech) Cumulative transfer volume (top) and data rates (bottom)
CMS data transfer between FNAL and UNL using Internet2's DCN and LambdaStation Software (FNAL + Caltech)
Cumulative transfer volume (top) and data rates (bottom)
Caltech and CIENA: 191 Gbps Avg., 199.90 Gbps Max on An OTU4
(Standard 100G) Wave at SC2008
10 X 10G Waves at the Caltech HEP Booth
Used Fully, in Both Directions with Caltech’s
FDT (TCP-Based Java Application)
Previewing the US LHCNet Transition to
4 X 100G by ~20141.02 Petabytes Overnight
SC08
Plots Courtesy the PingER Project
Data from http://internetworldstats.com
Digital Divide: North Vs. South East Vs. West
Work on the Digital Dividefrom Several PerspectivesWork on the Digital Dividefrom Several Perspectives
Share Information: Share Information: Monitoring,Monitoring, Tracking Tracking BW ProgressBW Progress;; Dark Fiber Projects & Pricing Dark Fiber Projects & Pricing Track Planning (focus on LHC) and Leading Edge ProgressTrack Planning (focus on LHC) and Leading Edge Progress Model Cases: Poland, Slovakia, Czech Rep., Brazil, China …Model Cases: Poland, Slovakia, Czech Rep., Brazil, China … Encourage Access to Dark Fiber; Modern technology choicesEncourage Access to Dark Fiber; Modern technology choices
Technical Help with Modernizing the Infrastructure: Technical Help with Modernizing the Infrastructure: Provide Tools for Effective Use: Provide Tools for Effective Use: Data Transport, Monitoring, Data Transport, Monitoring,
Remote Collaboration and e-LearningRemote Collaboration and e-Learning Design, Commissioning, Development Design, Commissioning, Development India “Knowledge Network”, Brazil’s Nat’l and Int’l NetworksIndia “Knowledge Network”, Brazil’s Nat’l and Int’l Networks
Encourage, and Work on Inter-Regional Projects Encourage, and Work on Inter-Regional Projects GLORIAD,GLORIAD, Russia-China-Korea-US-Europe Optical Ring Russia-China-Korea-US-Europe Optical Ring Latin America: Latin America: CHEPREO/WHRENCHEPREO/WHREN (US-Brazil); (US-Brazil); RedCLARARedCLARA Mediterranean: Mediterranean: EUMEDConnect; EUMEDConnect; Asia-Pacific: Asia-Pacific: TEIN3; TEIN3; Asia:Asia: ORIENT ORIENT
SCIC Monitoring WG PingER (Also IEPM-BW) SCIC Monitoring WG PingER (Also IEPM-BW)
Measurements from 1995 OnMeasurements from 1995 OnReports link reliability & qualityReports link reliability & quality
Countries monitoredCountries monitored Contain Contain 95.5%95.5% of world pop. of world pop. 99% of World’s Internet Users99% of World’s Internet Users
890890 remote nodes at remote nodes at 650650 sites in sites in 165 nations165 nations; ; 45 monitoring nodes;45 monitoring nodes; 95 Sites in 50 African countries95 Sites in 50 African countries
Strong Collaboration with ICTP Strong Collaboration with ICTP Trieste and NUST/SEECS (Pakistan)Trieste and NUST/SEECS (Pakistan)
Excellent, Vital WorkExcellent, Vital Work
Monitoring & Remote Nodes (1/09)
Countries: N. America (3), Latin America (21), Europe (33), Balkans (10), Africa (50), Middle East (13), Central Asia (9), South Asia (8), East Asia
(4), SE Asia (10), Russia (1), China (1) and Oceania (4)
R. Cottrell
Number of Hosts MonitoredBy Region: 1998 – End 2008 Number of Hosts MonitoredBy Region: 1998 – End 2008
R. Cottrell
AfricaAfrica
End
SCIC Monitoring WG - Throughput Improvements 1998-2008
Bandwidth of TCP < MSS/(RTT*Sqrt(Loss)) Matthis et al., Computer Communication Review 27(3), July 1997Matthis et al., Computer Communication Review 27(3), July 1997
30% annual improvement
~14X Per 10 yrs
Progress: but Digital Divide is Mostly Maintained
Behind Europe3-6 Yrs: Russia, Latin America, SE Asia 9-12 Yrs: South Asia, Central Asia16 Years: Africa
South & Central Asia, EspeciallyAfrica are Falling Farther Behind:
100X-1000X Worse by 2019 ?
R. Cottrell
http://www-iepm.slac.stanford.edu/
98
99
00
01
02
03
04
05
06
07
08
09
10k
100k
1M
10M
Th
rou
gh
pu
t (b
ps)
Brazil: RNP2 Next-Generation “Ipê” Backbone
Brazil: RNP2 Next-Generation “Ipê” Backbone
M. Stanton
New vs. OldBandwidth 70 to 300X
2007-9 Buildout of dark fiber nets in 27 cities: Connecting 200 Univ. & Institutes at 1 Gbps
2009: Upgrade 2.5G Sao Paulo –Miami Link to 2 X 10G (RNP+ANSP+NSF)10G to Grid-UNESP
Tier1, Tier2s in Rio and Sao Paulo
Dark Fiber (622 Mbps) across the Amazon Porto Velho to Manaus
Extending the Ipê Fiber footprint to the East and South
Three undersea cables cut: traffic greatly disturbed between Europe and Asia/Near
East zone France Telecom observed Dec. 19 that 3 major underwater cables were cut:
“Sea Me We 4” at 7:28am, “Sea Me We3” at 7:33am and FLAG at 8:06am. Causes of the cut, located in the Mediterranean between Sicily and Tunisia,
on sections linking Sicily to Egypt, remain unclear. Most traffic between Europe and Asia rerouted through the USA. Traffic from Europe to Near East & Asia interrupted to greater or lesser extent.First estimate of percentage of out-of-service capacity): Saudi Arabia: 55% out of service Djibouti: 71% out of service Egypt: 52% out of service UAE: 68% out of service India: 82% out of service Lebanon: 16% out of service Malaysia: 42% out of service Maldives: 100% out of service Pakistan: 51% out of service Qatar: 73% out of service Syria: 36% out of service Taiwan: 39% out of service Yemen: 38% out of service Zambia: 62% out of service
Computing Model Progress CMS Internal Review of Software and Computing
Cable Cuts on Dec. 19.
Partial Recovery by Dec. 23
PingER Measurementsof the Impact of the Cuts on Throughput
Vs. Time
1 3 5 7 9 11 1315 17 19
21 23 9 2727
SCIC: Networks for HEPMain Conclusions for 2009
SCIC: Networks for HEPMain Conclusions for 2009
The major R&E networks serving HEP have progressed rapidly over The major R&E networks serving HEP have progressed rapidly over the last few years; to N X10G in many casesthe last few years; to N X10G in many cases Worldwide transition to 40G & 100G links will occur in 2010 – 14Worldwide transition to 40G & 100G links will occur in 2010 – 14
Our BW usage has kept pace; may soon outstrip the planned capacityOur BW usage has kept pace; may soon outstrip the planned capacity Groups in HEP have developed state of the art methods to use these Groups in HEP have developed state of the art methods to use these
networks most effectivelynetworks most effectively Widespread deployment, end-to-end monitoring, training requiredWidespread deployment, end-to-end monitoring, training required
Adapting the LHC Computing Models to fully exploit networks Adapting the LHC Computing Models to fully exploit networks would have a profound positive impact on the LHC programwould have a profound positive impact on the LHC program
It is more urgent than ever that we act It is more urgent than ever that we act to Close the Digital Divideto Close the Digital Divide To make physicists in all regions full partners in the discoveries To make physicists in all regions full partners in the discoveries
We are learning to do this effectively, in some cases, in partnership We are learning to do this effectively, in some cases, in partnership with many agencies & HEP groups:with many agencies & HEP groups: Central Europe Central Europe andand Brazil; India Brazil; India and and PakistanPakistan
But we are leaving other countries and regions behind, for example:But we are leaving other countries and regions behind, for example: the Middle East, the Rest of Latin America; Africa the Middle East, the Rest of Latin America; Africa
A great deal of work remainsA great deal of work remains