OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY The DOE UltraScience Network (aka UltraNet) GNEW2004 March 15, 2004 Wing/Rao
Dec 19, 2015
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
The DOE UltraScience Network(aka UltraNet)
GNEW2004
March 15, 2004
Wing/Rao
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Talk Outline
What are we trying to do? Why do it? How (we all sort of know don’t we)? How can we get there from here? …Where will that be?
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
DOE Science
DOE has bigger problems (drivers) … (and less money) than any other three-letter agency
But with it, we are trying to do things like: Climate Modeling Supernova modeling Genomics (and proteomics)
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Climate Modeling
North American model at 400Kmresolution
North American model at 60Kmresolution
The Rockies block the atmospheric circulation and are importantgeneses regions for the weather systems that form the North AtlanticStorm Track. They are poorly represented in low resolution models.
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Type II CoreCollapse Supernova
QuickTime™ and aVideo decompressor
are needed to see this picture.
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Molecular dynamics of a lipopolysaccharide (LPS)
Nanosecond molecular dynamics of the LPS membrane of Pseudomonas aeruginosa
QM/MM molecular dynamics of membrane plus mineral
Computational Molecular BiologyComputational Molecular BiologyMembrane with possible uses for bioremediationMembrane with possible uses for bioremediation
.25TF
11TF
Energy transduction across membranes is still out of reach at 1.0PF
400 TF>1.0 PF
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Billions of Bases in GenBank
1982 1986 1990 1994 1998 2002
According to the GOLD database, there are 146 published genomes, 344 prokaryotic ongoing genomes projects, and 243 eukaryotic ongoing genome projects.
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Growth of Proteomic Data vs. Sequence Data
0.00001
0.0001
0.001
0.01
0.1
1
10
100
1000
198819901992199419961998200020022004200620082010201220142016
Years
PetaBytes
Proteomic data
GenBank
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
PetaScale Science Climate modeling today => 50-100 TB data sets
Still adding carbon chemistry Still refining resolution
You saw DOE Road Map table showing PB requirements in 2-3 years
Supernova modeling just going from 2 1/2 dimensions to 3D - this is a PB problem today
Genomics and Proteomics will dwarf these …and then of course there is High Energy
…but
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
We run hard into network limits…
There is some VERY nice work being done on TCP But that will just emphasize router delay And expose congestion, even where it coexists with it
There is nice work being done to extend MPLS But that does not eliminate jitter And imposes additional overhead
And there is still the tyranny of layers
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
2.5 Gbps
Science Applications - Bit Rate BudgetEnd-to-end
40 Gbps
Optical Backbone Host
SONET(One Channel)
Site SecurityFirewall
10 Gbps
5.0 Gbps
1.0 Gbps
TCP/IP(One Flow)
DWDM(One
Lambda)
IP Network Services
OC-768
100 Gbps
Research Opportunities
Bit-Rate
OC-48
OC-192
GigE
Apps
ESnet.65 Gbps
End-to-end Performance
Gap
ESnet
SITE LAN
Host
ESnet Cluster
One Host
ESnet
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
What are we trying to do?
We are trying to break out of the limitations of conventional networks TCP/IP Firewalls, and layers Small-frame technologies
Take advantage of the real throughput available in optical networks today at the SONET layer
Develop the techniques needed within 2-3 years for DOE Peta-scale science
Use switched-circuits as a solution we can start on today
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Restating the Problem -
Networks will soon offer 40 Gbs channels Typical multi-stream throughput limited to
5-6Gbs, and programming multi-steam is hard - hero efforts yield 9-10Gbs
Typical single-steam throughput to a single application ~1Gbs, even on a tuned network - hero effort yields 5-6Gbs
Firewalls throw it away anyway
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Solution - We won’t have line-rate optical packet-switching anytime
soon No photonic transistors No photonic memory buffers
We can do optical circuit switching Useless for most commercial networks
Doesn’t aggregate for economy of scale But bulk P2P purchases can be extremely attractive
A good match for DOE high-performance science applications which require both bandwidth and precision Petabyte file transmission in 2-3 years Tera-scale interactive visualization ASAP Remote real-time instrument control now
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
UltraNet Research Testbed
Build a sparse, lambda-switching, dedicated channel-provisioning testbed
Connect hubs close to DOE’s largest Science users (but let the user labs pay last-mile costs)
Provide an evolving matrix of switching capabilities
Separately fund research projects (e.g., high-performance protocols, control, visualization) that will exercise the network and directly support applications at the host institutions
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
The Resources:
Off hours capacity from Esnet Probably 2 x OC48 between Sunnyvale and
Chicago in the northern route
Dedicated Lambdas on NLR Two 10G lambdas between Chicago and
Sunnyvale Possibly more in year two or three
Two dedicated lambdas on the ORNL Chicago-Atlanta connector
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Resources Cont.
A progression of switching technologies Start with Ciena, Cisco, or Sycamore Migrate to Calient, Glimmerglass all-optical or a hybrid
Some Local Storage Logistical Storage Depot (prior to local application
development) Progression of experimental point-to-point
transport technologies (plus legacy) Fiber channel Infiniband
Migrate to the Production ESnet environment
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
CERN
StarLight
Sunnyvale
SOX
ANLFNAL ORNL
CalTech
SLAC
LBLNERSC
PNNL2 x10 Gbps
2 x10 Gbps
Seattle
BNL
JLab
Research Exchange Point
DOE University Partners
DOE National LabAccess via production networks
Research 10 Gbps lambda
Science UltraNet Phase I
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
The Physical View From 50,000 feet
NLR
ESnet
CERN etc.
ORNL Connector
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Network OverviewC
hic
ag
oC
hic
ag
o
Sc
hn
eid
er
Sc
hn
eid
er
Mo
no
nM
on
on
Cra
wfo
rds
vil
leC
raw
ford
sv
ille
Ind
ian
ap
oli
sIn
dia
na
po
lis
Bo
wli
ng
Gre
en
Bo
wli
ng
Gre
en
Na
sh
vil
leN
as
hv
ille
Ch
ris
tia
na
Ch
ris
tia
na
Se
ym
ou
rS
ey
mo
ur
Lo
uis
vil
leL
ou
isv
ille
Up
ton
Up
ton
Ste
ve
ns
on
Ste
ve
ns
on
Ch
att
an
oo
ga
Ch
att
an
oo
ga
Da
lto
nD
alt
on
Ac
wo
rth
Ac
wo
rth
Ata
lnta
Ata
lnta
CICI
SOADM conversionSOADM conversion
2 x OC192CHI – ATL
Tx ETF
2 x OC192CHI – ATL
Tx ETF
OLA1OLA1
KnoxvilleKnoxville
OakridgeOakridge
CICI
2 x OC192CHI – ORNL
Ultranet
2 x OC192CHI – ORNL
Ultranet
OLA2OLA2OLA - 2OLA - 2
MemphisMemphis
OLA -1OLA -1
OLA - 3OLA - 3
20 x Gig-E???20 x Gig-E???
GbE / 10GbE / OC102GbE / 10GbE / OC102
Intro 10GLAN PHY – CDIntro 10GLAN PHY – CD
10GbEOC192
10GbEOC192
10GbEOC192
10GbEOC192
10GbEOC192
10GbEOC192
10GbEOC192
10GbEOC192
OC192 10GbE
OC192 10GbE
10GbEOC192
10GbEOC192
10GbEOC192
10GbEOC192
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
The Logical View
Dedicated Lambdas, NLR
Dedicated Lambdas, NLR
Scheduled Lambdas, ESnet
MSPP
MSPP
MSPP SONET switch
SONET ADM
MSPP
MSPP
ORNL
Chicago
Atlanta
PNNL
Sunnyvale
SONET ADM
SONET switch
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Schedule
Oak Ridge Connector to be up in June Fall-back circuits via NLR if necessary
Chicago-Sunnyvale paced by NLR PNNL fiber schedule will pace their connection
Expect first user-traffic in June
OAK RIDGE NATIONAL LABORATORYU. S. DEPARTMENT OF ENERGY
Future Plans
ORNL is half the team building CHEETAH Nagi Rao and Malathi Veeraraghavan are PI’s Funded by NSF Will Connect ORNL, Atlanta, NC State, possibly CUNY Explicitly developing signaling protocols
Want to partner with HOPI …and all of you