High Speed Optics – The road to 400G and Beyond - Cisco Live

Errol Roberts – Distinguished Systems Engineer

Mark Nowell – Distinguished Engineer

BRKOPT-2005

High Speed Optics –The road to 400G and Beyond

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Webex Teams

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session

Find this session in the Cisco Events Mobile App

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

1

2

3

4

3

cs.co/ciscolivebot#BRKOPT-2005

BRKOPT-2005

Agenda

© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4

• Optics Landscape

• Optics and Optical technology

• Standards and Solutions – 400G

• Future developments on 400G and beyond

• Conclusion

BRKOPT-2005


Setting the stage ..High-speed optics landscape

5BRKOPT-2005

ASIC

Market & Network Architecture

Optics & Optical Technology

Solution


Setting the stage...... Optics deployment context

6BRKOPT-2005

• Within a building or campus or city

• Grey optics

• Ethernet

• Solve for Cost, Power, Density

Transport Networkcarrying Ethernet Traffic

TxP

Regen

Across country (100’s to 1000s km)

Multiple channels / Fiber (DWDM)

Colored Optics

Solve for Spectral eff. and Perf.

OA

Mux/DemuxROADM

DC & Client Optics Line Optics

IEEE defined Ethernet

Optics Landscape


High-speed optics landscape

8BRKOPT-2005

Industry Standards

and Consortia

ASIC



Solution


Ethernet Roadmap

9BRKOPT-2005

6 new speeds in 1st 35 years of Ethernet

6 new speeds in a 2 year span

• 2.5 GbE – 2016

• 5 GbE – 2016

• 25 GbE – 2016

• 50 GbE – late 2018

• 200 GbE – 2017

• 400 GbE – 2017

• 25/50GE Consortium

• MSAs

• QSFP-DD, OSFP

• 100G Lambda

https://ethernetalliance.org/the-2018-ethernet-roadmap/


Ethernet Speed Transitions

11BRKOPT-2005

Source:

Millions

•

•

•


400G Market Adoption

12

• 400G expected to be a rapid adoption

• Driven by both Cloud and Service Provider markets

Source: LightCounting

BRKOPT-2005


Importance of Interconnect

13BRKOPT-2005

Data Center to User

14.9%

Within Data Center

71.5%

Data Center to

Data Center

13.6%

Total East-West Traffic Will Be 85%

(Rack-local traffic would add another slice twice the size of “Within Data Center”)

Interconnect is #2 HW cost category in Cloud DC (behind servers)

Cost of interconnect can accelerate or delay upgrade transition

We need to focus on lowering optics cost to accelerate transition to 400 GbE

Some of East-West Traffic interconnects buildings (DCI)

Source: Cisco Global Cloud Index, 2016–2021

20.6 ZBin 2021


Data Center Fabric

14BRKOPT-2005

• High bandwidth Fabric

• Redundancy Model

• Highly Scalable

• Distributed Data Center

Multiple Spines

Wider Spine

Higher Scale Compute

100G 400G

100G 400G

25G 50G 100G

• Hash Efficiency Improvements w/high

speed links

• 400G deployments on horizon

• Copper Cables

• Multimode

• Single-mode

• DWDM (DCI)


2017 2018 2019/2020

40 GE 100 GE 100 GE 100 GE 200/400 GE

40 GE 100 GE(40 GE optics for cost

40 GE 100 GE 100 GE

40 GE 100 GE(40 GE optics for cost)

40 GE 100 GE 100 GE/400 GE

40 GE 40 GE 100 GE40 GE optics for cost

100 GE

100 GE 100 GE 400 GE

Source: Dell’Oro Jan 2018

Ethernet Speed Migration: Uplinks / Core

15BRKOPT-2005


Access Network Trends

17BRKOPT-2005

• Services, content and apps are moving to the edge

• Traffic is growing faster at the Metro

• Higher capacity systems are being deployed

• High speed optics is being adopted in client form factors

• Networks are being simplified, optimized and automated

long haul

DCI

Co-lo

Router /

Switch

An Access Network

Distributed

DC


Optical Transport Coherent market trends

18BRKOPT-2005

DCI drove edge growth in 2017

• Next wave coming in 2020

• 100G/400G “ZR” pluggable

market

10G Migration

• 10G Long Haul migration to

Coherent complete

• 45% of all 10G links are

<80km

• This will migrate to 100G

coherent at the right price100G and above, in 100G equivalents

Edge: < 120km

Metro: < 600km

LH: >600kmCourtesy of Andrew Schmitt, Cignal AI



19BRKOPT-2005

Industry Standards

and Consortia

ASIC



Solution


ASIC Trends: Historical Perspective Shows What’s Coming

20BRKOPT-2005

• Historical curve fit to highest rate switch products introduced to market (blue squares)

• Single ASIC IO capacity doubling every ~ 2 years

• IO speed has to increase due to package limitations

Total Switch IO BW

High Speed IO Electrical

Ratification


Ethernet’s consistent trend: Narrower/Faster and re-use

22BRKOPT-2005

1

10

1 10 10025Gb/s

50Gb/s

100Gb/s

10Gb/s

2.5Gb/s

1x

2x

4x

8x10x

16x

# lanes

IO (SERDES) speeds

Few

er la

nes

* At the right time

Higher speed

Optics and Optical Technologies



24BRKOPT-2005

Industry Standards

and Consortia

ASIC



Solution



25BRKOPT-2005


• Grey optics

• Ethernet



TxP

Regen



Colored Optics


OA

Mux/DemuxROADM




Optical technology…

26BRKOPT-2005

…Is the key enabler for systems to maintain pace with the requirements of the bandwidth pressures

Form Factor Size reduction

Power reduction

Reach

Packaging simplification

The next higher speed

Port Density increase

Port Density increase

System/network scaling

Increased Yield, lowers cost

System Scale, lower costs

Challenge Goal


Anatomy of a module

27BRKOPT-2005

PHY/CDR n:m Gearbox

TIA

LD

Host-Module SERDESinterface

High Speed optical side SERDES

interfaces

• Ethernet’s architecture allows these sets of interfaces to evolve independently. • Enables continual optimization of cost to occur.

Optimum when n = m

n m


Client Optics Goals – how do get there?

29BRKOPT-2005

Lower Power

Lower Cost

Smaller

👍

• Less lanes – higher speed per lane

• More integration

• More integration• Advanced technology

Fewer components:• Less lanes• More integrationEconomies of Scale• Volume


Optical Technology Trends

30BRKOPT-2005

IntegrationGoals:

Lower Cost Size

Power

Faster Optics (fewer needed)

Better materials for integration

e.g. Silicon Photonics

e.g. PAM modulation

Higher Yields Relaxed

specifications

e.g. Forward Error Correction


What is Silicon Photonics?

31BRKOPT-2005

• Ability to control optical signals in silicon

• Utilizes CMOS manufacturing infrastructure and capabilities

• Wafer scale manufacturing of optics!

• Promise to meet current and future optical communication requirements with:

Higher BW (density), Lower Power, and Lower Cost

Processed 8” SOI wafer

Silicon Photonic Die

(Image from Samsung website)


Silicon Photonics

32BRKOPT-2005

Potential for “transceiver on a chip”

• Optical devices can be made cheaply using standard semiconductor CMOS fabrication techniques

• Optics can be integrated with microelectronic chips.

• Silicon integrated optical chips that can generate, modulate, process and detect light signals

Silicon Photonics is the most promising optical technology for:

Solving for Cost, Power, Density for DC & Client optics

Solve for Spectral efficiency and Performance for DCI & Long Haul optics


Faster optics

33BRKOPT-2005

Parallelization enables high speed interfaces to be implemented using parallel lanes (fibers or wavelengths)

400 Gb/s 25 Gb/s x 16 50 Gb/s x 8 100 Gb/s x 4Multiple options

Increasing the speed of each lane, reduces the numbers of lanes needed resulting in lower cost in the long run

NRZ PAM-4

10 Gb/s 25 Gb/s 50 Gb/s 100 Gb/s


Higher Order ModulationSame Data and Data Rate; but lower frequency (baud rate).

34BRKOPT-2005

• Twice the data capacity for same “speed” components• Enables lower bandwidth components and materials• Reduces wavelengths & fibers compared to NRZ• More complex transmitters and receivers

Impact of higher order modulation

PAM-2(1-bit per symbol)

PAM-4(2-bit per symbol)

PAM-2

PAM-4

1-bit Symbols

2-bit Symbols

0 1 1 0 1 0 0 0 1 1

0 Level

1 Level

0 Level

1 Level

2 Level

3 Level

0 1 1 0 1 0 0 0 1 1

0 (0 level)

1 (1 level)

1 1 (3 level)

1 0 (2 level)

0 1 (1 level)

0 0 (0 level)

(But 4 levels)

(aka NRZ)


Forward Error Correction - overview

35BRKOPT-2005

Historically used in “difficult” transmission applications (e.g. long-haul optical transmission, satellite)

Adopted by clients optics starting @ 25 Gb/s and above

Block of data Protected data

Math happens. Create extra “check

bits”Adds some latency

Original dataReceived data

Data is transmittedSpeed increase due to

extra bitsErrors occur

Using extra check bits, errors are corrected.Adds some latency

Enables use of lower grade optical specs (knowing transmission errors will happen) but that can be corrected with simple low-cost digital logic


Forward Error Correction Benefits far out-weigh the drawbacks

36BRKOPT-2005

OSNR (dB)(Quality of signal)

(Higher quality = higher cost)

Bit E

rror R

ate

-3-4-5-6-7-8-9-10-11-12-13

Pre-FEC BER

Post-FECBER

• Incremental latency impact is dependent on implementation and data rate.• For common Ethernet interfaces latency increase in range of ~50 to 100 ns (equivalent

to time of flight over 5-10m of fiber)

Link operation Point:With FECW/o FEC

Usage of lower quality optical specifications significantly reduces cost and power of solutions

Different FEC algorithms can be used all with different performance properties• Reed-Solomon: most common in

Ethernet• Higher performance FECs (e.g. used in

Coherent optics)


Dual-Rate 40/100G BiDi QSFP28

37BRKOPT-2005

100G Capable

port

Tx

Rx

Tx

Rx

l

filter

l

filter

850 nm

Gearbox

900 nm

850 nm900 nm

20G NRZ

20G NRZ

QSFP+ module

Tx

Rx

Tx

Rx

l

filter

l

filter

850 nm

Gearbox+

FEC (100G mode)

900 nm

850 nm900 nm

20G NRZ or 50G PAM4

20G NRZ or 50G PAM4

QSFP28 module – supporting both 40G or 100G BiDi

40G Capable

port


Client Optics – Technology summary

39BRKOPT-2005

Increase baud rate

(e.g. 10G to 25G)SR, LR, etc.

Increase number of

fibers

(Parallel)SR4, PSM4, DR4,

Increase number of

wavelengths

(WDM)

LR4, ER4, BiDi, CWDM4, FR4

Change modulation

format

(e.g. NRZ to PAM4)

100G-FR, 100G BiDi, 400G-DR4

Enhance Bit Error Rate

with Forward Error

Correction (FEC)

Everything above 40G

(except 100G-LR4)

Block of data Protected data

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwiOxY2u9I7TAhWZ0YMKHfrND1cQjRwIBw&url=http://www.fiber-optic-transceiver-module.com/tag/40gbase-sr4&bvm=bv.151426398,d.amc&psig=AFQjCNG7wbSN5kvC2OmaWYsgB4Cy2cOqWA&ust=1491536315661363



40BRKOPT-2005


• Grey optics

• Ethernet



TxP

Regen



Colored Optics


OA

Mux/DemuxROADM




Ethernet & Transport Evolution

41BRKOPT-2005

• Ethernet dominates traffic coming into the transport networks.

• Transport systems work to align with Ethernet evolution and to stay ahead

• Solutions have developed enhanced capabilities in order to optimize for the diverse transport infrastructures

1985 1990 1995 2000 2005 2010 2015

Ethernet

Transport

Standard FE GE 10GE 40/100GE

StandardEth Payload

Demand and Innovation continue

SDH PayloadDemand and Innovation

continueOTU1/2 OTU3 OTU4

OTN aligned with Ethernet speeds

200/400GE

OTUcnFlexO


New approaches to increase fiber carrying capacity

Gain more from existing InfrastructureMaximize fiber usage

Optimize Data Rate vs Distance vs CostFigure of Merit: $$/Gb-KmReduce cost of optics

Reduce Operational Costs, Simplify through Automation

Optimize network continuously through Flexibility across Layers

Automation & Simplification via SW

Drive to Next Gen Data Rates 600G, 800G, 1T, etc…

Increase bits per port to reduce cost per bit,

leveraging Silicon advancesIncrease interface capacity

Enabling Transport Optical Networks to go Farther and Faster at Lower Cost

42BRKOPT-2005

Go Further, Faster at

Lower Cost












42BRKOPT-2005


Lower Cost

Coherent Optical Technology












42BRKOPT-2005


Lower Cost

Coherent Optical Technology

Control Plane and Automation


Coherent optics breaks the barrier on long reach fiber transmission

45BRKOPT-2005

Transmitted optical signal

(Contains Amplitude and

Phase information) Fiber impairments:

• Loss• Chromatic dispersion• Polarization dispersion• Amplifier noise• and more…

Photo-detector • PD generates the square of incoming signal

• Phase information lost. Unable to electronically compensate for fiber impairments

• Pushes compensation to optical domain ($$)

Photo-detector • PD generates the square of incoming signal

• Combination of local laser light and received light results in phase information retained.

• Electronic compensation for fiber impairments possible with much cheaper DSP technology

Local laser

Direct Detection

Coherent Detection


Coherent Detection

46BRKOPT-2005

Direct Detection

• Must correct for impairments in the physical domain (insert DCU’s)

• Forced to live with non-correctable impairments via network design (limit distance, regenerate, adjust channel spacing)

Coherent Detection

• Moves impairment correction from the optical domain into the digital domain

• Allows for digital correction of impairments (powerful DSP) vs. physical correction of impairments (DCU’s). Adds advanced FEC.

• Massive performance improvements over Direct Detection.

DDDD

DCU DCU DCU

Regen

CD

DCU – Dispersion Compensation Unit


Key building blocks along with Coherent Detection

48BRKOPT-2005

Payload

Framing and high performance FEC

• Standardized ways to map different clients together

Tunable lasers

• Tune transmitters and receivers to available wavelength channel


Tools to Increase and Optimize System Capacity

49BRKOPT-2005

• Speed up & WDM & Modulation

Multi-Modulation

Adjust spectral efficiency by transmitting more or less bits per symbol at a given baud rate

Superchannel Formation

Optimize spectral efficiency by tightly spacing sub-carriers using Flex-Spectrum ROADM

Hybrid Modulation

Mix different modulation formats in the time domain to achieve a greater bandwidth/distance granularity

Flex Baud Rate

Increase the bit rate by increasing the rate at which symbols are transmitted


Line side optics evolution

• How to go faster with less pieces?

• Double Baud Rate from 32GB to 64GB

• More Modulations

• QPSK, 8QAM, 16QAM, 32QAM, 64QAM

• Hybrid Modulation

• Integration – Fewer Components

• Line Rate Encryption

NG DSP for 400G DWDM and Beyond

50

DSP 200, 250, 300, 350, 400, 500, 600{

SW Configurable Rate / Modulation

SW Controlled Reach

BRKOPT-2005


Continuous Control with NG DSP

51BRKOPT-2005

Baud rateFEC Hybrid mode

15% 27%

nQAM mQAMmQAM nQAM…

9.6 Tbps12.8 Tbps

16 Tbps

19.2 Tbps

22.4 Tbps

25.6 Tbps

28.8 Tbps

32 Tbps

35.2 Tbps

38.4 Tbps

24 - 72 GBd/S

QPSK

8QAM16QAM

32QAM

Time Hybrid combination

0.008 bits/symbol

control

BPSK

64QAMMaximize capacity in 50Gbps increments for Reach required

NOTE: 64Chs C-Band @ 75GHz


Higher order modulation schemes

52BRKOPT-2005

Increasing the capacity per WDM channel requires sending more complex signals; trading between spectral efficiency and reach

3x (64-QAM vs QPSK) gain

QPSK 16-QAMBPSK 64-QAM

12842

Bits/Symbol(2 pol.)

6

8-QAM

Capacity@32Gbaud

50G 100G 150G 200G 300G


Dynamic Data Rates

53BRKOPT-2005

• Optimize Capacity vs Reach

• Avoid wasted capacity

BPSK

QPSK

8QAM

16QAM

32QAM

64QAM

0 1 2 3 4 5 6 7 8

100

1000

10000

0 10 20 30 40

Re

ach

(km

)

C-band capacity (Tbit/s)

Spectral efficiency (bit/s/Hz)

Simulations for 100km spans of NDSF, 0.2dB loss, 5dB NF EDFA

BPSK

QPSK

8QAM

16QAM

32QAM

64QAM


Example: Optimizing Performancewith Modulation and baud rate

75Ghz 37.5Ghz

16QAM64QAM 32QAM

75Ghz

69Gbd/s 56Gbd/s 32Gbd/s

600x64 = 38.4Tbps 400x64 = 25.6Tbps 200x128 = 25.6Tbps

BRKOPT-2005 54


Network Optimization .......

55BRKOPT-2005

Flex Mod, Hybrid Mod 50G/100G/200/250G/+

DWDM interfaces

Flex SpectrumFlexible nx12.5Ghz

channels

NPU Transponder/DCI DWDM

2x50G 2x200G

Superchannels

Flexible, efficient and dynamic mapping of packet services to optical transport

FlexSpectrum6.25/12.5/50GHz/100

GHz DWDM systems


• Gain more from existing Infrastructure

• New approaches to increase fiber carrying capacity

• Support Next Gen Data Rates 400G, 600G, 800G, 1T, etc…

• Increase bits per port to reduce cost per bit, leveraging Silicon advances

• Automation and Simplification via SW

• Reduce Operational Costs, Simplify through Automation

Go Further, Faster at Lower Cost!!

Flexible data rates require flexible spectrumSpectrum Switched Optical Networks

• Spectrum Switched Optical Networks

https://tools.ietf.org/html/rfc7698

BRKOPT-2005 56

https://tools.ietf.org/html/rfc7698


Each 50GHz carrier provisioned and switched individually

Superchannel switched through the ROADM network as a single entity

This requires Flexible Spectrum allocation – Flex Spectrum

50 GHz ITU Grid 50 GHz ITU Grid12.5 GHz Slices

Rigid Spacing, Wasted SpectrumSuperchannel with Minimal Spacing

Efficient Spectrum Use

Superchannels require a new kind of ROADM, one that can switch chunks of bandwidth larger and yet more granular than 50GHz

DWDM Network capacity limited by channel spacing imposed with fix ITU Grid

57BRKOPT-2005


Baud rate increase vs increasing sub-carrierswith super-channels

58BRKOPT-2005

• Higher baud rates

• 2x rate (1/2 carriers), but with Nyquist spectral

• density improvements only below 5%

• While higher NLI and implementation

• impairment might indeed affect reach/bandwidth

• Is higher baud rate necessary for increased bandwidth?

• Not necessary; transport over multiple wavelengths will be supported by new protocols (FlexE, OTUCn)

Meng Qiu, Qunbi Zhuge, Xian Xu, M. Chagnon, M. Morsy-Osman, and David V. Plant ‘Subcarrier Multiplexing Using DACs for Fiber Nonlinearity Mitigation in Coherent Optical Communication Systems,’ inProc. OFC 2014, paper Tu3J.2, San Francisco (CA), Mar. 2014.


Towards More Flexible High bit Rates DWDM Optical Network Control Plane Evolution

59

ASONAutomatic Switched Optical Networks

WSON

SSON

Wavelength Switched Optical Networks

Spectrum Switched Optical Networks

SDNSoftware Defined Networks

BRKOPT-2005


Control Plane Innovations – Flex spectrum

• Wavelength Switched Optical Networks (WSON) follows the standard ITU-T grid specification which defines the 50GHz constant channel spacing

• To enable Flexibility in the spectrum assignment, ITU-T also defines an Extended Granularity in the channel spacing, down to 6.25GHz

• Increasing the granularity of the spectrum grid, the complexity of running such network is increased and determines more requirements for the Optical Control Plane

• How to optimize the Transmission layer to cope with Distance and Capacity needed?

• How to optimize / de-frag provisioned Channels to save overall Spectrum

• FLOW: A new Control Plane supporting Flex-Spectrum networks extending GMPLS control plane and compliant with SSON standard

• FLOW allows to provision, protect, restore the new set of Optical Connectivity requirement coming by the dismount of fixed grid network

• Enhanced Optical Calculation algorithm implemented to take in account of new physical entities

BRKOPT-2005 60


SSON - FLOW - Optical Signals Hierarchy

61BRKOPT-2005

Media-Channel MCH1

Super-Channel SCH1

Super-Channel

SCH2

Super-Channel SCH3

Carriers Carriers

Carrier

Media-Channel: Continuous spectrum section

allocated from Sour. to Dest. (with Path) to bring

by default one S-CH

Super-Channel: set (1 or more) of homogeneous

(same type) optical carrier(s)

Carrier: Optical Channel (i.e. Trunk) carrying part

or all client payload

By default one MCH shall be associated to each

SCH

By defual each MCH can be switched/routed

independently

• The MCH has the information on Optical BW

allocated and the Path along the network

• The SCH has information on the channels

contained, and all the optical data

Media-Channel MCH2

Media-Channel MCH3

In order to maximize the spectral efficiency

Several MCHs can be aggregated to form a

MCH-GROUP.

When aggregated into a MCH-GROUP shall

have the same Src/Dest/Path and they shall be

managed in term of routing as a single entity.

Media-Channel Group


Possible Application: Automatic Line Rate DetectionMaximize utilization based on actual network autosensing capabilities

SSON FS CDC Network

100Ghz circuit

40Gbd 200G

40Gbd 200G

50Gbd 300G

50Gbd 300G

BRKOPT-2005 62


High Speed Client ImplementationsInterface independent functionality

63

SRGrey optics

(Short Reach)

Tunable Transponder

DWDMROADM

G.709FEC

Router

SR

G.709FEC

IP-over-DWDM

Colored opticsG.709

wrapper

OTN

G.709FEC

• IP-over-DWDM

• Pre-FEC error threshold is monitored directly by router

• RP initiates fast re-route based on internal trigger directly from PLIM

• Gray Client - Ethernet

• Pre-FEC error threshold is monitored by transponder

• Ethernet trigger is generated by transponder and sent to router which initiates fast re-route

• Gray Client - OTN

• Pre-FEC error threshold is monitored by transponder

• OTN PF-FDI trigger is generated by transponder and sent to router which initiates fast re-route

• OTN interface monitors end-to-end path

OTN ClientEthernet Client

OTN Trigger

Ethernet Trigger

BRKOPT-2005


Line side optics moving to pluggable modules

64BRKOPT-2005

• Optics purchased on-demand

• Replaceable if defective

• Tuneable laser

Integrated Coherent Optical Transmitter

Integrated Coherent Optical Receiver

Coherent DSP

4

4

TX

RX

Optical Signal

Analog Coherent Optics (ACO)

e.g. CFP2-ACO Module (100-250G)

DSP on linecard

On Host Board



HOST ASIC

4

4

TX

RX

Optical Signal

Digital Coherent Optics (DCO)

e.g. QSFP-DD DCO Module (400G)

DSP in module

On Host Board

DSP

Solution - Standards and 400G



66BRKOPT-2005

Industry Standards

and Consortia

ASIC



Solution


Industry Standards Groups

67BRKOPT-2002

IEEE

OIF

ITU IEEEOIF

Transport NetworksLayer 1/0 interoperability

Client InterfacesLayer 2/1 interoperability

Hardware VendorsComponent Interoperability,

Commonality

Services, Control Plane, MIBs, YANG, SDN APIs

IETF, MEF, ONF, Broadband Forum


Putting it together

ASIC

Non-Standard

MSA

Standards - IEEE

Host (linecard) Connector

Pluggable Module

SystemModule

Interface

Interfaces: • Point of interoperability• DR4, FR4 etc

Modules:• System design choice• Density, portfolio• Does not affect interoperability

System:• Responsible for integration

of solution. Power, Cooling etc

MA

CCisco linecard


Ethernet Standards, Consortiums, MSAs… wow!

69BRKOPT-2005

It may seem chaotic (and it is) but it is necessary and accelerates solutions

StandardsIEEE defines the

foundational interoperable

specifications and interface specs that industry can build

on.

MSAs

Consortiums

Interop Agreements

• Generally built on IEEE specs

• Define implementations or follow-on specifications

• Often membership based. Companies with common goal, cooperating to enable market advantage

• Move quicker (see above)

• QSFP-DD, OSFP, 100G l, COBO, PSM4, CWDM4, …


Current (400 GbE) Industry Activities – not just one place!

70BRKOPT-2005

Standards IEEE 802.3bs ✅ 400 GbE MAC & Interfaces (AUI, 400G-DR4, 400G-FR8, 400G-LR8) [Also 200GE…]

IEEE 802.3cd ✅ 100G-DR, 50G-CR [Also 50GE SMF/MMF, 200GE MMF]

OIF 400ZRIEEE 802.3ct

400G Coherent 100km 100G & 400G Coherent 100km

IEEE 802.3cm 400 GbE MMF (BiDi and SR8)

MSAs* 100G Lambda MSA100G Lambda IEEE

100G-FR, 100G-LR, 400G-FR4, 400G-LR4

QSFP-DD MSA 400G Form factor

OSFP MSA 400G Form factor

SFP-DD MSA 100G Form factor

COBO ✅ Embedded 400G/800G module

Industry aligned (except FR8/LR8)

Industry aligned

Industry aligned

Industry aligned

Industry aligned

* Multi-Source Agreements – new ones all the time. Not all get wide industry adoption

Limited Industry traction

✅ Complete

Industry aligned

Industry aligned

Limited Industry traction


Why the module pluggable form factor matters

71BRKOPT-2005

ASIC MA C

Everything hinges off optical module form factor

System Density # ports, cooling

Optical Module investment focus

Cost. Economy of scale

Time to Market New vs. Leverage known solutions

End User Investment Protection

Backwards compatibility enables reuse of optics, reuse of architecture

De-risk technology transition

Backwards compatibility enable 400 GbE systems to use 100G optics if 400GE Optics are delayed/expensive


400 GbE Industry Leadership

72

Which pluggable module form factor will dominate has been THE industry contention point over last 2 years.

Courtesy: TE

BRKOPT-2005

QSFP-DD OSFP

Latest update is that all system OEMs are building products based on QSFP-DD. OSFPs “advantages” proving to not be impactful for 400 GbE


QSFP-DD MSA

73BRKOPT-2005

• Initiated by Cisco in 2016

• Stable Specification

• D1.0 in Sept 2016

• D2.0 in March 2017

• D3.0 in Sept 2017

• 60+ member companies

• Mgmt Interface Spec written and shared with other industry groups

• Supports ASIC interfaces - 400G AUI-8 (8x 50G PAM4)

• Support network requirements for system density: 32 & 36 ports

• Support necessary thermal/SI for implementation (all optical and copper reaches)

• Backwards compatibility with QSFP28

www.QSFP-DD.com


Optics Innovation – QSFP-DD

• QSFP plus a 2nd row of pins

• Drop-in upgrade for 100G networks – same port count

• Maintains 36 ports per RU w/ backward compatibility

• Same faceplate, slightly deeper

• QSFP56-DD for 400G

• 8 electrical lanes at 50G (56 w/ overhead)

• QSFP28-DD for 200G or 2x 100G

• 8 electrical lanes at 25G (28 w/ overhead)

• Can support breakouts


Market Transition Barriers

75BRKOPT-2005

Optics is a barrier to transition

Reduction of ASIC cost-per-bit is outpacing that of optics

Critical to 400 GbE transition that optics cost reductions are accelerated


Accelerating 400 GbE optics volume on single form factor

76BRKOPT-2005

o System & network requirements do not change. Same port density per RU to maintain proven fabric designso Limited impact on system ecosystem – strong leverageo Multi-speed switch port options – slower optics in higher speed ports

Lessons from the Past1G 10/25G or 40G 100G transitions resulted in same high volume form factor being adopted. Why?

SFP28SFP

1G to 10/25GJourney

XEN PAK

X2

XFP

QSFP QSFP28

40G to100GJourney

CFP

CPAKCFP2

CFP4


QSFP-DD 400G Module ComparissonQSPF-DD provides the highest BW density of any pluggable module

77BRKOPT-2005

QSFP-DDCFP

86

130

16

21

29

16.2

CPAK

91

35

11.6

CFP2

92

42

12.4

QSFP28

18

50

13.5

CFP4

22

76

9.5

Lin

e c

ard

fa

ce

pla

te

CXPuQSFP

14

50

CFP8

92

41

12.4

18

50

13.5

100G Modules 400G Modules

83

22

13.5

OSFP

75


Avoiding the form factor journey for 400 GbE

78BRKOPT-2005

0

2

4

6

8

10

12

10 GbE 100 GbE 400 GbE

Years

Years to Achieve High Volume form

• 1st 5 yrs of 400 GbE forecast to be

40x larger than 1st five years of 100

GbE

Cu & OpticalHigh Density & volume

Longer reachLower volume

Lower System Density

QSFP QSFP28

40G to100GJourney

CFP

CPAKCFP2

CFP4

Higher volumesMedium System

Density

Largely ignored(Expensive & no

backwards compatible)


400 GbE: New Technologies enables 1st

generation optics convergence

79BRKOPT-2005

Deep industry experience has enabled cooling

solutions compatible with existing system designs

Cooling of 20W in stacked cages

Advanced Cooling

Coherent technology leveraging advanced CMOS

nodes and optical integration

Direct Attach Copper Cables (DAC) with 3m reach using 26 AWG

400ZR in same package as Cu/AOC

All reaches in same package size

Backwards compatibility enables decoupling of

Switch product deployment from optics deployment

De-risks technology transition


QSFP-DD Thermal performance

Taking QSFP-DD to 20W cooling capabilities from the 3.5W of QSFP28 required some impressive innovation!!

• Advanced cage design

• Advanced heat-sink design

• Advanced heat-sink retention

80

Latest testing show > 20W cooling very feasible

BRKOPT-2005


Accelerating 400 GbE optics volume on single form factor

81BRKOPT-2005

Priority with wide range of customers including cloud customersBackwards Compatibility

Optics cost is becoming a barrier to transition. Cost VolumeLower Cost

Industry experience with proven form factorDesign Flexibility

Investment ProtectionCabling infrastructure is an expensive and cumbersome process

Customer Care-about (aka path to volume)


400 GbE – Driving major technology innovations

Three main innovations driving technology transitions

• QSFP-DD pluggable module

• Coherent optics migrating to pluggable modules

• OIF 400ZR/ZR+ – 400G Coherent DWDM with reaches to > 1000 km

• Advances in CMOS technology (7nm) enables implementation in QSFP-DD

• 100 Gb/s per wavelength optics

• Faster optical direct detect modulation

• Reduces component count (cost/power)

• Aligns optics with advances in SerDes technology

82BRKOPT-2005


100G l MSA/IEEE

83BRKOPT-2005

Industry consortium and IEEE both specifying interfaces based on 100G PAM4 signaling

400 GbE: Half the lane count

100 GbE: Quarter the lane count (vs. CWDM4/LR4)

Improved cost, yield, power, scalability

100G-FR 2 km

100G-LR 10 km

400G-FR4 2 km

400G-LR4 10 km


100 GbE: Reduced complexity leads to lower cost

84BRKOPT-2005

100GBASE-LR4

Optical DMux

λ6

R

X6

λ1

R

X3

λ2

R

X4

λ3

R

X5

Optical Mux

10 km

4x CDR + DVR

4x Rx +CDR

4x 25G

λ1

λ2

λ3

λ4

RX

2

RX

3

RX

4

RX

1

Optical DMux

Optical Mux

2 km

4x CDR + DVR

4x Rx +CDR

4x 25G

100G-CWDM4

λ

RX

500 m

DSP

DSP

4x 25G

100G-DR

λ

RX

2 km

DSP

DSP

4x 25G

λ

RX

10 km

DSP

DSP

4x 25G

4x optical lane reduction

100G-FR 100G-LR


Reduced component count enables denser solutions: e.g. 100 GbE

85BRKOPT-2005

λ

DSP

4x 25G

λ

DSP

8 x 50G

λλ

λ

λ

RX

500 m

DSP

DSP

4x 25G

λ

RX

2 km

DSP

DSP

4x 25G

λ

RX

DSP

DSP

4x 25G

10 km

100 GbE

in QSFP28

Quad 100 GbE in QSFP-DD

(aka 400G-DR4)


400 GbE Duplex SMF Opticscomplexity reduction (vs. initial IEEE specs)

86BRKOPT-2005

400GBASE-FR8

λ7

RX

7

λ8

RX

8

λ1

RX

1

Optical DMux

λ6

RX

6

λ2

RX

2

λ3

RX

3

λ4

RX

4

λ5

RX

5

Optical Mux

10 km

λ7

RX7

λ8

RX8

λ1

RX1

Optical DMux

λ6

RX6

λ2

RX2

λ3

RX3

λ4

RX4

λ5

RX5

Optical Mux

2 km

λ1

λ2

λ3

λ4

RX2

RX3

RX4

RX1

Optical DMux

Optical Mux

10 km

λ1

λ2

λ3

λ4

RX2

RX3

RX4

RX1

Optical DMux

Optical Mux

2 km

400GBASE-LR8

50 Gb/s optics based

2x optical lane reduction

Moving from 8 lanes to 4 lanes further enables relaxation on wavelength grid to be considered

100 Gb/s optics based

400GBASE-FR4 400GBASE-LR4


Coherent entering the Data Center

88BRKOPT-2005

OIF’s 400ZR project will be the a key change within the data center

• Used for DCI applications (which includes for East-West traffic)

• Pluggable form factor enabled by:

• Advanced CMOS node for DSP (7nm)

• Reduced targets on reach

• Integrated optical Tx/Rx components

• Advanced cooling (20W)

• Enables same form factor as copper/client

Beyond 400G: Does coherent technology penetrate deeper into the Data Center shorter reaches?



HOST ASIC

4

4

TX

RX

Optical SignalDSP

QSFP-DD

Installation Considerations


Optics Codes (Cheat Sheet)

90BRKOPT-2005

wG-xRy.z e.g. 400G-DR4

(w) Data Rate:10 Gb/s25 Gb/s50 Gb/s100 Gb/s200 Gb/s 400 Gb/s

(x) Reach:MMF

S = 100 mSMF

D = 500 mF = 2 kmL = 10 kmE = 40 kmZ = 80 km

(y.z) Lane Count:(fiber or Wavelength)y = “ “ single fiber/wavelengthy = “4”

F/L = 4 wavelengthsD = 4 fibers

y = “4.2”e.g. SR4.2MMF: 4 fibers

2 wavelengths per fiber

Narrower =

Lower cost


Fiber Infrastructure – 10 GbE to 400 GbENotes on fiber options

SMF

• Most long term upgrade path

• Parallel fiber utilized after 100 Gb/s

• Shorter reaches (500m)

• Lower cost solutions

MMF

• Duplex and Parallel solutions available

• 300m reach @ 10 Gb/s per lane

• 100m reach >10 Gb/s per lane

• Parallel fiber utilized after 100 Gb/s

• Multiple MMF fiber types

• OM3 – 850nm optimized

• OM4 – enhanced modal bandwidth

• OM5 – wider wavelength window (supports SWDM)

BRKOPT-2005 91


Infrastructure Migration Paths

92BRKOPT-2005

1G (SFP)

10G (SFP+)

100G (QSFP28)

40G (QSFP+)

400G (QSFP-DD)

SMF Duplex

LX/LH

10 km

SX

2 km

LR

10 km

SR

400 m

LR4

10 km

4x10G LR

10 km

BiDi

150 m

SR4

150 m

LR4

10 km

PSM4

500 m

CWDM4

SM-SR

2 km

FR

2 km

BiDi

100 m

SR4

100 m

CSR4

400m

4xBiDi

(SR4.2)

100 m

DR4

500 m

FR4

2 km

LR4

10 km

LR4-L

2 km

SMF Parallel

MMF Duplex

MMF Parallel


10 GbE 25 GbE 40 GbE 50 GbE 100GbE 200 GbE 400 GbE

SMF

Duplex 10G-LR25G-LR25G-ER

40G-LR450G-FR50G-LR

100G-DR100G-FR100G-LR100G-LR4100G-ER4CWDM4

200G-FR4200G-LR4

400G-FR4400G-LR4400G-FR8400G-LR8

400ZR

Parallel (x4)

- - - - 100G-PSM4 200G-DR4 400G-DR4

MMF

Duplex 10G-SR 25G-SR 40G-BiDi 50G-SR 100G-BiDi - -

Parallel (x2/x4)

- -40G-SR4

40G-CSR4-

100G-SR4100G-CSR4100G-SR2

200G-SR4400G-SR4.2

(BiDi)

Parallel (x8/x16)

- - - - 100G-SR10 -400G-SR8

400G-SR16

Fiber Infrastructure – 10 GbE to 400 GbEUpgrade paths for different fiber options

Broadest long term support

Largest installed base of MMF

Parallel needed to

support higher data

rates

Lower cost enabled by SiPhotonics

BRKOPT-2005 93

Future developments on 400G and beyond


Ethernet Speed Evolution

95BRKOPT-2005

400 Gb/s Ethernet is the current highest speed

It is a certainty that denser & faster solutions will be needed

• SERDES moving to 100 Gb/s

• ASIC roadmaps include 25.6 Tb/s & 51.2 Tb/s


Transceivers Evolution Drive Efficiency

96BRKOPT-2005

• Modules use less power for the same bandwidth for density

10087.5

240

120

75

35 25

0

50

100

150

200

250

300

Pow

er

(W)

Transceiver Power* for 1Tb/s of

Bandwidth

0

10

20

30

40

50

60

02468

10121416

# o

f Li

ne

Car

d P

ort

s

Lin

e C

ard

Ba

nd

wid

th (

Tb

/s)

Line Card Bandwidth Enabled by

Module Form Factor

Line Card BW

Power efficiency becoming a more significant area of focus for optics

* 2km optics


Moving beyond pluggable optics

97BRKOPT-2005

Pluggable optics at 800G is looking feasible

BUT

Optics, ASIC and packaging technology is making it possible to consider moving the optics into the system or into the ASIC package.

Switch

Switch

Switch

Pluggable module

On-board optics

Co-packaged opticsThe goal is to permit greater switch radix with lower power consumption, which is really important as we continue to increase speed and bandwidth

Low

er p

ow

er

Incre

ased c

om

ple

xity


Future trends: Co-packaged Optics

98BRKOPT-2005

Co-packaged – In ASIC Package

• SiPhotonics die are compatible with ASIC die for co-packaging

• Close proximity lowers power needed to interconnect the dies – lowers overall system power

• Complex packaging issues to be resolved but underway

• Enables higher density solutions

At some point, pluggable faceplate modules will not keep up with ASIC bandwidths

Chip package w/ ASIC and

SiPhotonics die

Fiber pigtails to faceplate

Courtesy Luxtera


What Will Change in order to Continue to Scale?

99BRKOPT-2005

400G 400G

400G 400G

CoherentDSP

Multi-Terabit

Processor

600G200G

28G56G100G

Optical FibersSMF or MMF

ElectricalAnalog

DWDM Optic

MultiRateDSP

LineSystem

S&RASIC

nx800G

Electrical Lanes

28G56G100G

DPSK8QAM

+32QAM+64QAM

32Gbd64Gbd

Incre

ase

Incre

ase

Incre

ase

Incre

ase

Incre

ase

Incre

ase

Incre

ase

Incre

ase

CoherentDSP

200G400G800G

FlexMod

FlexO

integrate

Summary



101BRKOPT-2005

Industry Standards

and Consortia

ASIC



Solution


Choice of Optic Considerations

• Fiber type – SM, MM, duplex, parallel considerations factor in to optic client choices

• Balance choice of optics with cost, reach, power, density, performance and packaging

• Backward compatibility will de-risk investment cycles for server and switching

• High speed optics in the DC fabric improves application performance

• 400 GbE optics activity across industry

• Volumes will drive cost reduction – directly related to adoption of common form factor

• Line Side developments important to maximize fiber capacity Flex Spectrum, Hybrid/Flex Modulation, control plane

• Client side and line side technology merging – coherent, pluggable package – new applications

• High speed optics enables flexible bandwidth, increased system performance, drive network architecture

102BRKOPT-2005


Alignment Needed to Achieve TCO Goals

103BRKOPT-2005

ASIC BW & Port Speed

Optics Speed,

Power, SizeCabling Infrastructure/

transport infrastructure

• Innovation

• Standardization

• Open Ecosystems


Summary

Demand for 400 GbE is here

104

Different solutions and options for client and Line-side optics exist to meet the different

challenges

Uniquely for 400 GbE, all solutions will exist in common dense QSFP-

DD form factor

Industry is broadly engaged to deliver 400

GbE now

Cisco is a leader in all aspects

BRKOPT-2005


Cisco Webex Teams

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session

Find this session in the Cisco Events Mobile App

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

1

2

3

4

105

cs.co/ciscolivebot#BRKOPT-2005

BRKOPT-2005


Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com

• Please complete your Online Session Survey after each session

• Complete 4 Session Surveys & the Overall Conference Survey (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Events Mobile App or the Communication Stations

Complete your online session survey

106BRKOPT-2005


Demos in the Cisco Showcase

Walk-in self-paced

labs

Meet the engineer

1:1 meetings

Related sessions

Continue Your Education

107BRKOPT-2005

Thank you

High Speed Optics – The road to 400G and Beyond - Cisco Live

Documents