Slide 1
100 Gbps Services, Technologies, and Facilities: Basic Issues,
Early Implementations, Demonstrations
Wendy Huntoon, Director Advanced Networking,
([email protected])Pittsburgh Supercomputing Center/3Rivers Optical
Exchange
Abdella Battou, Interim Executive Director,
([email protected])
Joe Mambretti, Director,
([email protected])International Center for Advanced
Internet Research (www.icair.org)Northwestern UniversityDirector,
Metropolitan Research and Education Network
(www.mren.org)Co-Director, StarLight
(www.startap.net/starlight)
Quilt Annual MeetingPortland OregonSeptember 20111Need for
Massive Additional Capacity Support for Capacity Much Beyond
Aggregation of Millions of Small FlowsSupport for Extremely Large
Individual Stream (Including End-To-End)Communications for Data
Intensive (e.g., Petascale Science) Communications for Specialized
Highly Distributed EnvironmentsEnvironments Directly Controlled By
Edge Processes (Application Specific Network Services)Highly
Controllable Science WorkflowsBegin Migration From Centralized
NOCsScience Clouds (vs Consumer and Enterprise)Many New
Applications and Services That Cannot Be Supported Today
Drivers for 100 Gbps Services2National Science Foundation
Academic Research Infrastructure Program (ARI)Traditionally, This
Program Has Funded Structures - e.g., Creating and Rehabbing
Specialized Buildings and LaboratoriesCurrently, For the First
Time, The Program Allows Communications Infrastructure To Be
Included As StructureARIs Are Limited To One Per Campus, Requiring
Local Organization CompetitionThe Funding Is Directed At Supporting
Science Research Not General R&E Services
NSF Advanced Research Infrastructure Program3Four Worlds of 100
Gbps
100 Gbps Routing100 Gbps Ethernet100 Gbps OpticalOthere.g.,
Fiber Bundles, Interconnections, Control Planes, etc.4100 Gbps
RoutingAvailable Today Based on Proprietary Technology Optimal
Network Designs Place Such Devices At the Network Edge vs Network
Core
100 Gbps Services: Routing5100 GigE L2 Switching
Standard: IEEE 802.3baTechnical changes Finalized In July
2009Formal Final Approval Took Place In July 2010Beta Products
Available Q4 20101st Commercial Products End of Q4 2011Provides for
a Rate of 103.125 Gbps
100 Gbps Services6100 Gbps Optical Switching
Standard: ITU G.709 v3 (ODU4 100G)ODU4/OTU4 Format -- Designed
to Transport 100GbE (OTU4 = the ODU4 With FEC IncludedFormal Final
Approval Took Place In Dec 2009Beta Products Available Today Ref:
Demonstrations at SC101st Commercial Products Available End of Q3
2011
100 Gbps Services: WAN Side/Line Side 7100 Gbps Switch Design
IssuesOptical/Electrical InterfacesData Plane ProcessingControl
Plane Processing Switch FabricsModulation Techniques: e.g.,
Proprietary Dual Polarization Quadrature Phase Shift Keying (QPSK),
a Format Used With 50GHz Channel Spacing (4 Signals In the Same
Channel Spacing)PowerEtc.
100 Gbps Services: WAN Side/Line Side8100 GigE Physical Layer
Standard (PHY) Objectives
Preserve 802.3 / Ethernet Frame Format Based On 802.3
MACPreserve Min/Max Frame Size of 802.3 StandardProvide PHY
Specifications For Single-Mode Optical Fiber, Multi-Mode Optical
Fiber (MMF), Copper Cables, Backplanes. Support Bit Error Ratio
(BER) Better Than or Equal to 10 12 at the MAC/PLS Service
InterfaceProvide Appropriate Support for Optical Transport Network
(OTN) Standard
100 Gbps Services: Client Side 9Physical and Link Layer100G
Long-Distance DWDM Transmission Framework The Framework Project
Documents High Level System Objectives for Initial Implementations
of 100G Long-Haul DWDM Transmission. It Identifies Transceiver
Module Functional Architecture, and Decomposes That Architecture
Into a Number of Technology Building Blocks. This Project Is
Directed At Developing a Consensus Among a Critical Mass of Module
and System Vendors On Requirements For Specific 100G technology
Elements To Create Largest Possible Market for These Components.
This Project Complements and Builds on Work Defining 100G Ethernet
in the IEEE, and the Optical Transport Hierarchy (OTH) in the
ITU-T.
Optical Internetworking Forum (OIF)100 Gbps Standards10100G Long
Distance DWDM Integrated Photonics Receiver This Project Specifies
Key Aspects of Integrated Receivers for Coherent DWDM Applications.
Initially Targeting 100G PM-QPSK Applications, Project Intends to
Remain Modulation Format and Data Rate Agnostic Whenever Practical
to Maximize Applicability to Future Market Requirements. Key
Aspects of Project Include Definition of: (1) Required
Functionality. (2) High Speed Electrical Interfaces. (3) Low Speed
Electrical Interfaces. (4) Optical Interfaces. (5) Mechanical
Requirements. (6) Environmental Requirements. One Example: Optical
Internetworking Forum100 Gbps Standards11100 Gbps Initiative
PartnersPittsburgh Supercomputing Center3Rivers Optical
ExchangeMid-Atlantic Cross RoadsMetropolitan Research and Education
Network (MREN)StarLight International/National Communications
ExchangeInternational Center for Advanced Internet Research
(iCAIR), Northwestern UniversityUniversity of Illinois at
ChicagoArgonne National LaboratoryFermi National Accelerator
LaboratoryNational Center for Supercomputing ApplicationsCHI MAN
Chicago Metro Area Optical Transport Facility for national Research
Labs and StarLight (Multiple 100 Gbps)Illinois Wired/Wireless
Infrastructure for Research and Education (I-WIRE)University of
ChicagoEnergy Science Network 100 G Advanced Network
InitiativeEnergy Science NetworkNASA Goddard Space Flight
CenterCANARIE, SURFnetNational Oceanographic and Atmospheric
Administration etc123ROX ARI ProjectRegional Network Renovation:
Upgrading the 3ROX Virtual Research Environment Start date -
9/2010Duration - 36 monthsObjectivesUpgrade 3ROX DWDM
infrastructure Upgrade Layer2/Layer2 infrastructure to be 100 GE
capableProvide resource pool to researchers133ROX ARI StatusDWDM
UpgradeCompleted in March 2011Features included:Increased lambda
capacity - 4 lambdas per segment to up to 40 lambdas per
segmentAdded ACM as a PoP on the ringReduced the number of
transponders required to traverse the metro area Support 40 GE and
100GE lambdas
3ROX ARI Status - DWDM Ring
3ROX ARI Status100 GERFP out for 100 GE optical
equipmentResponses due September 23 - significant
interestLayer2/Layer3Minimum capability is to support 100 GE
Internet2 connectionIdeally support switched 100 GESupport for
OpenFlow optionalRFP to be released by the end of
SeptemberResponses due 3 weeks after release
3ROX ARI StatusCreate Research Resource PoolEstablishes a pool
of resources available to 3ROX associated researchersCurrent
Resources10 GE transponders10 GE switch port and optics3ROX PoP
co-location space and power - as availablePending100 GE
transponders100 GE switch port and optic 3ROX ARI
StatusProcessSubmit a brief proposal requesting resourcesReviewed
by the 3ROX RRP Review committeeIf approved, resources allocated
for a 6 to 12 month periodResearchers can request a time extension
for longer projectsProgram GoalReduce the barrier for researchers
to utilize network infrastructure, including 100 GEProvide
mechanism for researchers to experiment with network resources
before committing funds MAX ARI ProjectNSF 09-562 submitted
08/24/2009OIA Academic Research InfrastructureStarting Date :
05/03/10Duration 24 monthsObjectivesEnhance existing Research and
Education metro network with 100G equipment to support data
intensive science exploration, modeling, and discoveryUpgrade the
10G connection to Internet2 with 100GUpgrade the NGIX-East to
100G
19MAX 100G StatusPacket Optical Switching RFI issued on
4/6/2010Selected Fujitsu on 5/27/2010Acquired 6 Fujitsu 9500 Packet
Optical on 9/20/2010Deployed them on 2/28/2011100G transponders
expected end of September, 2011100G L2 Ethernet Switching RFI
issued on 6/6/2011Internally Selected Brocade MLX-E on
08/22/2011Order for 4 100G ports waiting approval from NSF20MAX
100G Research Network
21E-science ApplicationsIt is the nature of science problems
that they deal with a continuum of time and space parameters, which
need to be quantized to adapt to available network throughput. For
e-science experiments, it is never the question of how much
throughput is needed to support an experiment, but what the
accuracy of an experiment is given the network throughput. For
example, a climate modeling experiment will be able to work with
temperature and pressure gradients over a 3-D mesh which is ten
times finer in each spatial dimension if the network throughput is
improved by three orders of magnitude.
Thus, credible science experiments tend to, and in fact are
designed to consume as much network throughput as possible
We better start working on the next higher rate : 1 Tbps
interface !22IP Router Port Cost Improvement
Router ARouter BRouter CRouter DRouter ARouter BRouter CRouter
DBefore: After:
Expensive 100Gbs router ports are reduced 50% Traffic is
efficiently routed to appropriate destination (ex. A-C, vs.
A-B-C)2310G consolidations
Any 10G waves along the same path could be aggregate into a 100G
using OTN 24MREN Submitted ARI Proposal Early in 2010Award
Announced In October 2010Award Granted To Metropolitan Research and
Education Network/StarLight International/National Communications
Exchange FacilityStarWave, A New Advanced Multi-100 Gbps Facility
and Services Is Currently Being Designed It Will Be Implemented
Within the StarLight International/National Communications Exchange
FacilityThe Optical Components Will Be Implemented September 22-24,
2011Demonstration Showcases Planned for Oct, 2011, Nov 2011 (SC11,
Seattle)
Metropolitan Research and Education Network (MREN) Multi-100
Gbps Facility25StarWave: A Multi-100 Gbps FacilityStarWave Is Being
Funded To Provide Services To Support Large Scale Data Intensive
Science Research InitiativesFacilities Components Will Include: An
ITU G. 709 v3 Standards Based Optical Switch for WAN Services,
Supporting Multiple 100 G ConnectionsAn IEEE 802.3ba Standards
Based Client Side Switch, Supporting Multiple 100 G Connections,
Multiple 10 G ConnectionsMultiple Other Components (e.g., Optical
Fiber Interfaces, Measurement Servers, Test
Servers26CA*net/Ciena/StarLight/iCAIR 100 Gbps Testbed Sept-Oct
2010
Source: CANARIE100 Gbps~850 Miles, 1368
Km27CA*net/Ciena/StarLight/iCAIR 100 Gbps Testbed Sept-Oct 2011
Source: CANARIE100 Gbps~850 Miles, 1368 Km28
29
30
DOE ESnet Advanced Networking Initiative: 100 GbpsSource :
ESnet31Future International 100 Gbps Services Via Global Lambda
Integrated FacilityAvailable Advanced Network
ResourcesVisualization courtesy of Bob Patterson, NCSA; data
compilation by Maxine Brown, UIC.www.glif.is
iCAIR is a co-founder of the GLIF, a consortium of institutions,
organizations, consortia and country National Research &
Education Networks who voluntarily share optical networking
resources and expertise to develop the Global LambdaGrid for the
advancement of scientific collaboration and discovery. 32t
Component of Next Gen Control Framework: Fenius
GLIFDemonstrationsGlobal Lambda Grid WorkshopSC10HK GLIF Technical
Workshop33SC11 Demonstration PlansConnect MAX and MREN to SCinet to
support 100 GE demonstrationsBackbone connectivityUtilize existing
backbone (ESnet, Internet2) to connect MAX and MREN to
ScinetInvestigating dark fiber path between MAX and MREN for
additional pathWould light with Fujitsu optical equipmentSCInet
SCInet NOC, NASA, Ciena and LAC/iCAIR
100 GE WAN ConnectivitySeattleChicagoMcLeanWANWANNOCLA/iCAIR
CienaNASAS/ROpticalOpticalOpticalARI AwardsOrganizationTitle
Awarded Amount University Corporation For Atmospheric ResearchThe
Bi-State Optical Network (BiSON) Upgrade $ 430,000 University of
HawaiiEnabling Hawaii to COMPETE: Seizing the Opportunity for
Equitable Connectivity $ 9,838,151 Northwestern UniversityARI:
Enhancing MREN Optical Communications Infrastructure for e-Science
$ 1,374,132 University of Maryland College ParkRenovating
Mid-Atlantic Crossroads Network Facilities $ 1,442,919 North Dakota
State University FargoNorthern Tier Network--North Dakota and South
Dakota Access Improvement $ 751,244 North Dakota State University
FargoEnabling Inclusive Western States NSF Research via Next
Generation Networking - the Northern Tier $ 1,157,498 University of
New MexicoA Dedicated Advanced Science and Engineering Enterprise
Network: The National Backbone Component $ 2,992,000 Rensselaer
Polytechnic InstituteNYSERNet Research and Education network
renovations $ 390,050 Carnegie-Mellon UniversityRegional Network
Renovation: Upgrading the 3ROX Virtual Research Environment $
1,535,012 Clemson University Research FoundationExpanding and
Improving the C-Light Regional Optical Network $ 1,431,340 $
21,342,346