2003 © SWITCH
SWITCHlambda - Experiences with national dark fibers @ SWITCH
Willi Huber / Urs Eppenberger
2003 © SWITCH 2
The SWITCHlambda Project
Project goals:
• Provide fore front network services to the Swiss Universities at low recurring cost
• Network scalable to multiples of 10 Gbps with low additional cost
• Long living infrastructure, no need to change provider every couple of years
• Independence of carrier market
2003 © SWITCH 3
SWITCHlambda Project
Basic decisions:
• Replacement of the STM-1 links by GigabitEthernet connections based on dark fiber
• Long term lease of dark fiber, indefeasible right of use (IRU) for at least 10 years
• Use of DWDM on main trunk lines
• Redundant wide area links
• Simple network design (no SDH, no ATM): Use of Ethernet-Interface all over the place
2003 © SWITCH 4
SWITCHlambda Network Design
• Hierarchy of technologies – DWDM on main trunk lines (16 possible today, expandable if
later needed)– CWDM or single GE to smaller sites off the backbone
• Transparent optical path between Zurich and Geneva– No regeneration between the two main sites of SWITCH
• Optical path on main trunks designed for 10 Gbps / – Chromatic dispersion compensated– Polarization mode dispersion under control
• Bi-directional use of fibers– lower fiber costs
2003 © SWITCH 5
SWITCHlambda Network end of 2003
dark fiber based links (approx. 1200 km)
leased GE link
350 km
Zurich
Bern
Basel
Geneva
Lausanne
2003 © SWITCH 6
Bi-directional Use of Fiber
• Only one fiber used for the DWDM system
• Second fiber free to connect smaller sites inbetween Universities with cheap GE technology
P2P GigabitEthernet-connections
n * 1 GE / 10 GE implemented with DWDM
RouterRouter
One pair of fiber
DWDM node DWDM node
Router
Router
2003 © SWITCH 7
Rx Tx
GigEMux/Demux
Rx Tx
GigEMux/Demux
dispersioncompensator
add/dropcircuit
opticalamplifiers
opticalamplifiers
dispersioncompensator
directionalcoupler
directionalcoupler
to/fromsite C
to/fromsite A
localconnections
site B
Bi-directional transmission mode: i.e. a single fiber is used for signals in both directions, leaving the second fiber free for future use.
Typical SWITCHlambda node
2003 © SWITCH 8
Sorrento Gigamux
2003 © SWITCH 9
Sorrento CWDM
JumpStart-400 CWDM System
– 8 wavelengths (4 receive, 4 transmit)
– Bi-directional operation
– Optional backup path
SWITCH uses currently 1 pair of Jumpstarts for two p2p GE connections over on single fiber.
2003 © SWITCH 10
Bi-directional GigabitEthernet
Cheap solution for p2p GE connection over one single fiber with colored GBICs and passive optical devices
Splitter
OADM
Coloured GBIC
Switch /Router
Splitter
OADM
Coloured GBIC
Switch /Router
Signal reflection (e.g. bad connector)
2003 © SWITCH 11
Testing 10 Gbps/channel capability
Goals• verify equipment capability under real life conditions• verify optical design (dispersion compensation)• verify fiber quality for 10 Gbps application (PMD and
other adverse characteristics)
Realization• DWDM interface cards on loan from Sorrento (10GE
WAN-PHY)• IP equipment from Force10 (2002) and Cisco (2003)• traffic generation with Linux based workstations and
the free software “mgen”
2003 © SWITCH 12
10 Gbps tests
10 Gbps/, 360 km, bi-directional transmission
1. Test with signal regeneration optical - electrical - optical in the middle (at Bern)
2. Test with transparent optical signal Zurich - Geneva
3. Same as 2., but with additional attenuation of 3 dB of optical signal at the sender
Each test: 8 Gbps of traffic during 35 hours, 0 bit errors BER < 8 * 10-15
SWITCHlambda ready for 10 Gbps !
2003 © SWITCH 13
Pros and Cons: Technical View
• Stable, long living infrastructure
• High future potential
• This new stuff is fun
• none
2003 © SWITCH 14
Is this reliable?
• Stable infrastructure with occasionally long interruptions
– Interruptions mainly due to planned maintenance
– True fiber cuts are rare, but last for hours if not days (e.g. fiber on high voltage power lines, fiber along railway)
– Experience of a big Swiss carrier: Fiber cuts happen mostly in the local loop (city)
• Our experiences so far– Several interruptions due to planned maintenance (lasting up
to 1 day)
– Broken optical amplifier (interruption of 4 hours)
» Redundant links are mandatory for a good sleep!
2003 © SWITCH 15
No connection, what now?
• 24/7 monitoring by Ascom Netwatch– ping, SNMP– alarm SWITCH staff– basic analysis
• SWITCH NOC for analysis and action decision• University IT staff
– power problems– physical equipment check
• Onsite support by Ascom Netwatch– localize cable problems with OTDR (optical time domain
reflectometer)– Cisco hardware
• Onsite support by Deltanet for Sorrento DWDM
2003 © SWITCH 16
Pros and Cons: Operational View
• we are the boss • Big effort for negotiations with a lot of contract partners
• no single provider to blame in case of connection loss
• need a car to fix problems instead of telnet
2003 © SWITCH 17
Network Cost, general Remarks
• Cost for wide area links– Well known in advance
• Local loop cost– Vary a lot from place to place
– Good (say cheap) solutions need a lot of planning and patience
– Building own infrastructure is economically better than renting, but not possible in all cities
– In some cities there is a de facto monopole of the city’s industrial works
– The railways are an excellent partner for wide area links: low local loop cost because the schools are mostly located near railway stations
2003 © SWITCH 18
Network Cost, general Remarks (2)
• Cost for optical equipment– DWDM is expensive
– Long 10 Gbps/ DWDM trunks are very expensive (dispersion compensation, amplification)
– CWDM stuff is getting cheap
• Routers / Switches– Ethernet interfaces are cheaper by factors than SONET ones
2003 © SWITCH 19
Cost Structure
Total yearly cost: 2.05 MEURCost include: Investment years 2001-2004
10 Gbps between the major sites maintenance & support
wide area fiber
local loop fiber
DWDM equipment
CWDM equipment
maintenance & support
Depreciation period used for the calculations:Fiber optic infrastructure (wide area & local loop):
10 yearsFiber optic equipment DWDM & CWDM:
5 years 31 %
12 %
6 %
44 %7 %
2003 © SWITCH 20
Comparison with predecessor
SWITCHng SWITCHlambda
Years covered: 1999-2002 2001-2004
Technology: ATM/STM-1 dark fiber, GE & 10 GE
Sites connected: 13 universities 21 universities
Yearly cost: 2.95 MEUR 2.05 MEUR
2003 © SWITCH 21
Isn’t it cheapter to buy it?
We have not done a serious market review lately or even negotiated for capacity to get the real prices
But we have a figure we assume is the best current offer on the market for high capacity links: 100 kEUR
– limited to 300 Mbit/s
– local loop must exist
For 21 sites: 2.1 MEUR
Similar cost as SWITCHlambda,but less flexibility and performance.
2003 © SWITCH 22
Pros and Cons: Economic View
• Low recurring cost
• Low marginal cost for additional bandwidth
• High investment
• Uncertainity about development of bandwidth prices