NetherLight day-to-day experience APAN lambda networking BOF

Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

NetherLight day-to-day experienceNetherLight day-to-day experienceAPAN lambda networking BOFAPAN lambda networking BOF

NetherLight day-to-day experienceNetherLight day-to-day experienceAPAN lambda networking BOFAPAN lambda networking BOF

Erik Radius

Manager Network Services

SURFnet bv


SURFnetSURFnetSURFnetSURFnet

• Provides the Dutch National Research Network

• Not for profit company, 50 employees

• 150 connected organizations, 500,000 users

• Turnover (2002): 30 million Euro

• Infrastructure services:– innovation paid for by government

– cost effective exploitation for higher education and research


SURFnet5SURFnet5SURFnet5SURFnet5

• Partners BT and Cisco

• 15 PoPs connected by thirty 10 Gbit/s lambdas

• Dual stack IPv4 and IPv6 since 2001

• More than 100 institutes connected at Gbit/s level

• 500,000 users


NetherLightNetherLightNetherLightNetherLight

• Optical Internet exchange point in Amsterdam

• Built and operated by SURFnet

• Experiments with light path provisioning in a multi domain environment

• www.netherlight.net


NetherLight Network 2001NetherLight Network 2001NetherLight Network 2001NetherLight Network 2001

• 2.5Gbit/s lambda between StarLight, Chicago, USA and NetherLight, Amsterdam, NL

• Lambda terminated on Cisco ONS15454 muxes, – WAN side: SONET framed: OC48c

– LAN side: GbE interfaces to computer clusters

StarLight NetherLight

2.5G lambdaGbE

GbE

GbE


NetherLight Network at iGrid2002NetherLight Network at iGrid2002NetherLight Network at iGrid2002NetherLight Network at iGrid2002

2.5 Gbit/s SURFnet

2.5 Gbit/s SURFnet

10 Gbit/sLevel3

10 Gbit/s IEEAF

2.5 Gbit/sCERN

DWDM SURFnet

ChicagoStarLightChicagoStarLight

New YorkNew York

GenevaCERN

GenevaCERN

DwingelooAstron/JiveDwingelooAstron/Jive

AmsterdamNetherLightAmsterdamNetherLight


NetherLight equipment in A’damNetherLight equipment in A’damNetherLight equipment in A’damNetherLight equipment in A’dam

10GBase-LR (SM fiber, = 1310nm) 1000BaseT (copper)1000BaseSX (MM fiber, = 850nm)1000BaseLX (SM fiber, = 1310nm)

DWDM line terminal(Cisco ONS15252)

Ethernet switch(Cisco 6509)

optical cross connect(Calient PXC)

L1 (SONET) multiplexer

(Cisco ONS15454)

Lambdas to Chicago, Geneva, Prague etc

DWDM over dark fiber to JIVE

Computer clusters

UvA/NIKHEF


International lightpath network 1Q2004International lightpath network 1Q2004International lightpath network 1Q2004International lightpath network 1Q2004

DWDM SURFnet

lambda service path

IP service path

10 Gbit/s

SURFnet10 Gbit/s

SURFnet10 Gbit/s

IEEAF10 Gbit/s

DwingelooASTRON/JIVE

DwingelooASTRON/JIVE

PragueCzechLight

PragueCzechLight

2.5 Gbit/s

NSF10 Gbit/s

LondonUKLightLondonUKLight

StockholmNorthernLight

StockholmNorthernLight

2.5 Gbit/s

New YorkMANLANNew YorkMANLAN

TokyoWIDETokyoWIDE

10 Gbit/s

10Gbit/s

10 Gbit/s

2x10 Gbit/s

IEEAF10 Gbit/s

2x10 Gbit/s

10 Gbit/s

2.5 Gbit/s

2.5 Gbit/s

TokyoAPANTokyoAPAN

GenevaCERN

GenevaCERN

ChicagoChicago AmsterdamAmsterdam


Day-to-day lightpath useDay-to-day lightpath useDay-to-day lightpath useDay-to-day lightpath use

• Part of lambda capacity statically configured for IP production traffic

– Router-to-router SONET circuits between Amsterdam and Chicago for SURFnet and others (e.g. NSF link)

– Link & interface state monitored by SURFnet NOC

• Part of lambda capacity for lightpath research community

– Changed on weekly/monthly basis upon request

– Has a ‘test’ status; is hardly monitored by NOC


NSF funded 10G lambda US - EUNSF funded 10G lambda US - EUNSF funded 10G lambda US - EUNSF funded 10G lambda US - EU


Application for a lightpathApplication for a lightpathApplication for a lightpathApplication for a lightpath

• Researchers can apply for a lightpath by an e-mail to [email protected]

• Information in the application should include:– Test purpose

– End points A and Z [CERN, StarLight, Stockholm]

– Interface type [1GbE: SX or LX, 10GbE WAN PHY]

– Number of lightpaths needed

– Lightpath capacity in multiples of SONET STS-6 (311Mb/s):[STS-6 / STS-12 / STS-24 / STS-48 / STS-192] *

• Full GbE capacity: STS-24 *

– VLAN ID (if applicable)

– Start date, end date

– Contact details (e-mail, phone)

* Bandwidth granularity depends on transport equipment, e.g. Cisco ONS15454


Day-to-day operations (1)Day-to-day operations (1)Day-to-day operations (1)Day-to-day operations (1)

• SURFnet NOC manages production network plus NetherLight

– Part of NetherLight has production status– NetherLight test circuits get lower priority than SURFnet5

IP network

• Some learning curve to get NOC engineers familiar with transport (SONET) troubleshooting

– ONS15454 network operations: craft terminals only

• DWDM link to Dwingeloo (ASTRON): very stable, no pro-active management of optical system yet

– DWDM (ONS15252) craft terminal is not very user-friendly



• International lambda network monitoring – SNMP monitoring by NOC for SONET alarms and port errors

– BER level monitoring on ONS15454 SONET ports• BER parameter is part of SLA with carriers

– GbE ports: errors mostly attributable to LAN troubles

– No IP traffic monitoring possible on SONET ports• ONS15454 cannot look inside SONET circuits, only provide

overall quality information (BER, errored-seconds etc)

• No trouble with clocking observed yet– SONET network should have one master clock

– ONS15454 internal Stratum III clock is sufficient, no external clock needed



• Layer0: optical cross connect– Calient Photonic Cross Connect, automated patch panel– Research status only, managed by University of

Amsterdam– Topology discovery difficult

• Layer1: SONET layer– Very flexible GbE path setups on +- biweekly basis– 10G end-to-end also possible, for couple of months at a

time– Multi-domain setup works very well by building half-circuits– ONS15454 networks from Canet and SURFnet pre-wired at

StarLight


What did we learn / need to work onWhat did we learn / need to work onWhat did we learn / need to work onWhat did we learn / need to work on

• Point to point lambdas are a powerful service– High-performance research groups are pulling at us after getting a

taste of 10GE all-the-way…

• Straightforward to build a hybrid network architecture on top of optical:

– Some lambdas to interconnect your IP prod. routers

– Some lambdas for new high speed few-to-few usage

• Current optical-electrical-optical equipment can allocate sub-lambdas to individual applications

– Ethernet-over-SONET based, this will get more advanced soon (we need to learn what LCAS can do here)

• Management is still cumbersome, learning curve for all of us– How to make this dynamic? Or self-provisionable…?


Light Path Provisioning in SURFnet6Light Path Provisioning in SURFnet6Light Path Provisioning in SURFnet6Light Path Provisioning in SURFnet6

SURFnet6Layer 2 / Layer 1 network

Light Path Provisioningvia Control Plane

Layer 3 flowdata collectionand analysis

OperationsManagement

OGSAApplications

Webservice

Broker

SURFnet6Routers

Light Path


[email protected]

www.netherlight.net www.surfnet.nl

NetherLight day-to-day experience APAN lambda networking BOF

Documents