Next Generation Distributed Environments For Global
Science
Joe Mambretti, Director, ([email protected])
International Center for Advanced Internet Research (www.icair.org)
Northwestern University
Director, Metropolitan Research and Education Network (www.mren.org)
Director, StarLight, PI StarLight IRNC SDX,Co-PI Chameleon, PI-iGENI, PI-
OMNINet (www.startap.net/starlight)
Asia Pacific Advanced Network (APAN)
Co-Located With Supercomputing Asia
March 26-29, 2018
Singapore
Sloan Digital Sky
Survey
www.sdss.org
Globus Alliance
www.globus.org
LIGO
www.ligo.org TeraGrid
www.teragrid.org
ALMA: Atacama
Large Millimeter
Array
www.alma.nrao.edu
CAMERA
metagenomics
camera.calit2.net
Comprehensive
Large-Array
Stewardship System
www.class.noaa.gov
DØ (DZero)
www-d0.fnal.gov
ISS: International
Space Station
www.nasa.gov/statio
n
IVOA:
International
Virtual
Observatory
www.ivoa.net
BIRN: Biomedical
Informatics Research
Network
www.nbirn.net
GEON: Geosciences
Network
www.geongrid.org
ANDRILL:
Antarctic
Geological
Drilling
www.andrill.org
GLEON: Global Lake
Ecological
Observatory
Network
www.gleon.orgPacific Rim
Applications and
Grid Middleware
Assembly
www.pragma-
grid.net
CineGrid
www.cinegrid.orgCarbon Tracker
www.esrl.noaa.gov/
gmd/ccgg/carbontrack
er
XSEDE
www.xsede.org
LHCONE
www.lhcone.net
WLCG
lcg.web.cern.ch/LCG/publi
c/
OOI-CI
ci.oceanobservatories.org
OSG
www.opensciencegrid.org
SKA
www.skatelescope.o
rg
NG Digital
Sky Survey
ATLAS
Compilation By Maxine Brown
New Science Communities Using LHCONE
• Belle II Experiment, Particle Physics Experiment Designed To
Study Properties of B Mesons (Heavy Particles Containing a
Bottom Quark).
• Pierre Auger Observatory, Studying Ultra-High Energy Cosmic
Rays, the Most Energetic and Rarest of Particles In the Universe.
• In August 2017 the PAO, LIGO and Virgo Collaboration Measured a
Gravitational Wave Originating From a Binary Neutron Star Merger.
• The NOvA Experiment Is Designed To Answer Fundamental
questions in neutrino Physics.
• The XENON Dark Matter Project Is a Global Collaboration Investing
Fundamental Properties of Dark Matter, Largest Component Of The
Universe.
• ProtoNUMA/NUMA – Collaborative Research On Nutrinos
iCAIR Basic Research Topics
• Transition From Legacy Networks To Networks That Take Full Advantage
of IT Architecture and Technology
• Extremely Large Capacity (Multi-Tbps Streams)
• Specialized Network Services, Architecture and Technologies for Data
Intensive Science
• High Degrees of Communication Services Customization
• Highly Programmable Networks
• Network Facilities As Enabling Platforms for Any Type of Service
• Network Virtualization
• Tenet Networks
• Network Virtualization
• Network Programming Languages (e.g., P4) API (e.g., Jupyter)
• Disaggregation
• Orchestrators
• Highly Distributed Signaling Processes
• Network Operations Automation (Including Through AI/Machine Learning)
• SDN/SDX/SDI/OCX/SDC/SDE
Issues
• 21st Century Scientists Encounter Unprecedented Opportunities As Well
As Deeply Complex Challenges By Utilize New Knowledge Discovery
Techniques Based On Exploring Hidden Patterns In Exabytes of Globally
Distributed Data.
• Scientific Data is Growing Exponentially
• Growth Will Further Accelerate As New Major Instrumentation Is
Implemented.
• Also, New, Highly Sophisticated Analytic Techniques For Big Data Based
Investigations Are Being Created.
• The Challenges That Arise From This Global Scale Research Are Being
Addressed Through the Creation of Powerful, Innovative Globally
Distributed Computational Science Platforms, Such As the Global
Research Platform (GRP).
Global Research Platform (GRP).
• The Global Research Platform (GRP) Provide Next Generation Services.
Architecture, Techniques, and Technologies For World-Wide Data
Intensive Scientific Research.
• This Platform Provides Exceptional Support For Capturing, Managing,
Indexing, Analyzing, Storing, Sharing, Visualizing and Transporting
Exescale Data.
• Innovations Being Developed Include Those Related To:– Heterogeneity
– Virtualization,
– Segmentation
– Open Infrastructure Architecture and Components
– Orchestration
– Sliceability
– Multi-level Resource Abstraction and programmability,
– Granulated Customization,
– Fluid Data Distribution
– AI/machine Learning/Deep Learning
• Built-In Preconfigured Examples/Templates To Establish Infrastructure
Foundation Workflows
• Orchestration
• Zero-Touch “Playbooks” For Different Segments of Infrastructure
Foundation Workflows After Implementing Initial Suites (e.g., Using
Jupyter)
• Interactive Control Over Running Workflows
• Portability for Different Infrastructure Foundation Workflows
• Options/Capabilities for Specialized Customization
• Options For Real Time Visualization Of Individual Workflows, At Highly
Granulated Levels
Emerging Capabilities/Technologies
Network Research: Kernel Bypass
Networks
Drivers:
• AI Training/DL
• Distributed Storage Systems
• Virtual Networking
• Programmable Networks/SDN
• NFV
• In-Memory Caching
• Low Latency
• High Capacity (e.g., Big Data)
• Low Latency and High Capacity
Motivation For KBNets & Issues
• Need To Minimize Network CPU Overhead
• Sub Performance of Traditional OS Stack
Issues To Be Addressed
• Control Planes
• Data Planes (Transport)
• Management Plane
• Virtualization
• Programming Languages
• NIC/Backplane/Switch/Chip Architecture
Techniques
• RDMA (Remote Direct Memory Access)
• DPDK (Data Plane Development Kit)
• SmartNICs
• New Backplane/Switch Fabrics
• RoCE (RDMA Over Converged Ethernet)
• Et Al
• New High Performance OS Stacks
Programmable Network Techniques and
Devices
• SDN/SDX & Network Programming Languages
• Programmable Switch ASICs
• Programmable Network Processors
• FPGAs
• Programmable NICs
• Ref: Barefoot Tofino, Intel FlexPipe, Cavium XPliant,
Netronome Agilio.
• P4 Based In-Network Telemetry
• AI/ML/DL Integrated With Network Programming
• Jupyter
• DTNs
iCAIR: Founding Partner of the Global Lambda Integrated Facility
Available Advanced Network Resources
Visualization courtesy of Bob Patterson, NCSA; data compilation by Maxine Brown, UIC.
www.glif.is
AutoGOLE
International Multi-Domain Provisioning Using AutoGOLE
Based Network Service Interface
(NSI 2.0)
* Network Service Interface (NSI 2.0)
* An Architectural Standard Developed By the *Open Grid
Forum (OGF)
* OGF Pioneered Programmable Networking (Initially Termed
“Grid Networking”)
Techniques That Made Networks ‘First Class Citizens” in Grid
Environments – Programmable With Grid Middleware
* Currently Being Placed Into Production By R&E Networks
Around the World
IRNC: RXP: StarLight SDX A Software Defined
Networking Exchange for Global Science Research and
Education
Joe Mambretti, Director, ([email protected])
International Center for Advanced Internet Research (www.icair.org)
Northwestern University
Director, Metropolitan Research and Education Network (www.mren.org)
Co-Director, StarLight (www.startap.net/starlight)
PI IRNC: RXP: StarLight SDX
Co-PI Tom DeFanti, Research Scientist, ([email protected])
California Institute for Telecommunications and Information Technology (Calit2),
University of California, San Diego
Co-Director, StarLight
Co-PI Maxine Brown, Director, ([email protected])
Electronic Visualization Laboratory, University of Illinois at Chicago
Co-Director, StarLight
Co-PI Jim Chen, Associate Director, International Center for Advanced Internet
Research, Northwestern University
National Science Foundation
International Research Network Connections Program
Workshop
Chicago, Illinois
May 15, 2015
IRNC SDX
IRNC SDX
IRNC SDX
IRNC SDX
GENI
SDX
UoM SDX
Emerging US SDX Interoperable Fabric
GENI
SDX IRNC SDX
Global LambdaGrid Workshop 2017
Demonstrations, Sydney Australia
International Multi-Domain Provisioning Using AutoGOLE Based
Network Service Interface (NSI 2.0)
Using RNP MEICAN Tools for NSI Provisioning
Large Scale Airline Data Transport Over SD-WANs Using NSI and
DTNs
Large Scale Science Data Transport Over SD-WANs Using NSI
and DTNs
SDX Interdomain Interoperability At L3
Transferring Large Files E2E Across WANs Enabled By SD-WANs
and SDXs
ESA Service Prototype
Source; John Graham UCSD
Source; Jim Chen, iCAIR
Implementing a SCinet DTN
SUPERMICRO 24X NVMe SUPER
SERVER
NVMe Type A: 8 X Intel P3700 800G
NVMe Type B: 8 X SamSung 950 512G
+ M.2 to U.2 Adopter
Dell 14G Solution Configuration
(Co-Research & Development in Collaboration
with Dell)
Intel® Xeon®
processorE5-2600 v4
Intel® Xeon®
processorE5-2600 v4
QPI2 Channels
DDR4
LANUp to
4x10GbE
PCIe* 3.0, 40 lanes
Intel® C610
series
chipset
WBG
DDR4
DDR4
DDR4
DDR4
DDR4
DDR4
DDR4
PowerEdge R740XD Server
2 X Intel® Xeon® Gold 6136 3.0G,12C/24T,10.4GT/s 2UPI,24.75M Cache,Turbo,HT (150W)
192G DDR4-2666
PCI-e Configuration Investigation:
2 X Mellanox ConnectX-5 100GE VPI
4 X Kingston/Liqid AIC NVMe PCI-e X8 SSD Drives
Optional SAS/SATA Drives
Recent WAN DTN Testing
• Preparation – Tests Conducted By Se-Young Yu (iCAIR)
• DTNs:
• @iCAIR : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz, Mellanox
ConnectX-5 100G NIC
• @PACWAVE - LA : Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz,
Mellanox ConnectX-5 100G NIC
• @UvA : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz, Mellanox
ConnectX-3 40G NIC
• @CERN : Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Intel 82599ES
10G NIC
• Tuning Parameters :
• BIOS, CPU, NIC, TCP Stack, O/S and MTU Tuning Applied
Memory-to-Memory Test Results
Disk To Memory Test Results
Wenji Wu, Phi Demar et al
Source: Wenji Wu
Source: Wenji Wu
Source: Wenji Wu
Source: Wenji Wu
Source: Wenji Wu
L2 SW Disaggregation: Falconwitch
Prototype ConfigurationNVMe Type A: Intel P3700 800G x 8
NVMe Type B: M.2 to U.2 Adopter with
SamSung 950 Pro 512G X 8
SamSung 960 Pro 1T X 8
SamSung 960 Pro 2T X 8
4 X Mellanox ConnectX-5 100GE
2 X PCI-e X16 host adapter to host
Host node: SuperWorkstation 7048GR-TR
2 X E5-2667 V4
2 X PCI-e X16 host adapter
Falconwitch PCI-e Switching Fabric
Selected ML Frameworks (Of Many):
• Apache Singa
• Caffe
• H2O
• MLlib (Apache Spark)
• Scikit-Learn (Python)
• Shogun (C++)
• TensorFlow
• Theano (Python)
• Torch (~ Scientific Computing)
• Veles (C++, w/ Some Python)
Summary
• Data Intensive Science Can Benefit From A GRP, Including
Enhanced Services/Techniques/Technologies For High
Performance WAN Data Transport
• One Approach: Relying On L2 WAN Transport Channels
• A Complementary Enabling Capability: Using DTNs Integrated With
Specialized WAN Paths To Optimize E2E Data Flows
• Core Components Can Be Supplemented By Enhancing Software
Stacks, e.g., Jupyter, NSI, MEICAN, P4 Programs, BDE, AI/ML/DL,
etc
• Today, Many Components Exist To Create An E2E Services For
Data Intensive Science
• Major Opportunity: Creating This Service and Placing It Into
Production
www.startap.net/starlight
Thanks to the NSF, DOE, DARPA,
NIH, USGS, NASA,
Universities, National Labs,
International Partners,
and Other Supporters