INTREPID Update ARPA-E Annual Review October 31 th , 2019 Presenters: Clint Schow: Overview and progress updates Katharine Schmidtke: Data center perspective and T2M Adel Saleh: Network architecture and modeling Representing contributions from the entire INTREPID team
31
Embed
Potential and Challenges for Silicon Photonics in …...Workshop on Silicon Photonics for High Performance Computing, Fort Collins, CO, May 2018. • C. L. Schow, “Ultra-Low Power
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Presenters:Clint Schow: Overview and progress updatesKatharine Schmidtke: Data center perspective and T2MAdel Saleh: Network architecture and modeling
Representing contributions from the entire INTREPID team
INTREPID Project Objectives
2
‣ Making Coherent Optics in the Data Center a Reality– Demonstrate energy efficient coherent engines compatible with
package integration– Establish path to low power links at 200Gb/s/λ and beyond– Leverage large sensitivity gains to enable new architectures
based on photonic routing and switching‣ Unique approach: Use Optical Phased Locked Loops (OPLLs) to
eliminate DSP for orders of magnitude reduction in power‣ Overturn the conception that coherent = high power consumption‣ Explore and model novel architectures enabled by the expanded link
budget of coherent receivers‣ Phase 2 Goal: Pave the way for transition to widespread deployment
and commercial success
Phase 1 INTREPID Team: UCSB and Facebook
3
‣ Highly multidisciplinary, expertise spanning many fields– Datacenter operation, network design and management– Requirements and roadmap for optics at cloud-scale– Global optical supply chain, expertise and ability to drive– Electronic/photonic integration and co-design– Low-power, high-speed circuit design– Photonic integrated circuits: design and fabrication– High-speed packaging, assembly, and characterization– Network architecture design and modeling
‣ Phase 2 Team will add a partner with extensive engineering and manufacturing resources to enable technology transition
Facebook: Katharine Schmidtke (co-PI), Hans-Juergen Schmidtke, Ariel Hendel, James Stewart, Todd Hollmann, Jimmy WilliamsUCSB Faculty: Clint Schow (co-PI), James Buckwalter, Larry Coldren, Jonathan Klamkin, Adel SalehUCSB Students and Post-Docs: Hector Andrade, Takako Hirokawa, Navid Hosseinzadeh, Junqian Liu, Aaron Maharry, Thomas Meissner, Stephen Misak, Luis Valenzuela, Yujie Xia, Shireesh Bhat, Fabrizio Gambini, Sergio Pinna
INTREPID Highlights from Phase 1Technology Development: Hardware• Targets: 200 Gb/s analog coherent links
– Exploring business models for ecosystem development– Developing and publishing open interfaces
• Lightcounting market research study on co-packaged optics• Co-Packaged Optics Collaboration (CPO) led by Facebook and Microsoft
Network Architecture• Developed folded Clos data center architectures with AWGRs, end-of-row
(EoR) switching for increased scalability, economy and energy efficiency• Built a Python simulation platform to investigate a wide range of
architectures/technologies• Devised various energy efficient, optical-switching-based architectures,
including a cost-effective, energy-efficient data center growth strategy using quasi-static optical switches
High-Speed Electronics and Packaging
InP PICs: Coherent RX and TX
Si PICs: Coherent RX
Coherent Link Modeling~5pJ/bit possible
50Gbd/s, dual-pol, QPSK
Tunable laser/LO
INTREPID Review, September 2019
ToR Switches with Server and Fabric InterfacesAnalog Coherent WDM links• “Up”: all switch-to-switch connections, up to 2km over SMF• OPLL (Optical Phased Locked Loop) No/very little DSP• Expanded link budgets to enable photonic routing/switching• 1310nm: uncooled lasers, dispersion not an issue• Target: 800Gb/s/fiber = 4λ @ 200Gb/s/λ (dual-pol QPSK, 50 Gbd/s)
Multimode VCSEL links• “Down”: ToR-server connections (30m)• Currently all copper today challenging cost point • 50G (Phase 1)100G (Phase 2)
Photonic interfaces integrated in first-level switch packages:
• Modular: common interface to switch ASIC• Optimum I/O architecture between ASIC and
optics to maximize efficiency and performance
5INTREPID Review October 2018
Phase 2: Focus on analog coherent transceivers
High-Speed Packaging and Test
6INTREPID, ENLITENED SPECIAL SESSION, OFC 2019
TX with custom OptiGOT VCSEL
Limiting Amplifier
OPLLTestbed
TDR of Connector Launch
Fully Functional InP Coherent RX PIC
INTREPID Review, September 2019
Signal in
Local Oscillator
Back mirror Gain
Front Mirror
PhaseSOA
SOA
2 X 4 MMIPDs
Layout
Fabricated Device1 mm
Gen 1 PICs functional• LO lasers tunable over 26 nm• PD BW >20 GHz• PD responsivity ~1 A/WGen 2 Improvements:• Epi modification to improve laser efficiency• Semi-insulating substrate for higher speed
and easier packaging
Schematic LO tuning response
Photodiode BW
Photodiode spectral response
COTS OPLL Testbeds
INTREPID Review, September 2019
Gen 1
4.34 4.35 4.36 4.37 4.38 4.39 4.4
Frequency (GHz)
-60
-55
-50
-45
-40
-35
Mag
nitu
de (d
Bm)
Stable FWHM = 6.336 MHz
Frequency locking (TIA output):
Gen 2: Oct 2019
• OPLL testbed for PIC verification and testing• COTS TIAs and XOR to be replaced with single custom ASIC
(first-gen OPLL chip received 10/18/2019)
Loop OFF39 MHz LW
Loop ONLW: 6 MHz
DP-QPSK Dual-Pol OPLL Receiver
• Multiple designs of two-channel RX and two-channel RX with OPLL in fab
• Global Foundries GFUS8XP process• Novel design consolidating functional blocks
expected to reduce power dissipation up to 2X– 1.69 pJ/bit for OPLL chip
INTREPID Review, September 2019
Chip 1
Chip 2
Generalized Block Diagram of QPSK Costas Loop Architecture
Used in Receiver OPLL Integrated Circuit
1st Generation OPLL RX
Received 10/18/2019
INTREPID Review, September 2019
Functional Gen1 Si Coherent Receiver
• All functional blocks verified• Now mounted on high-speed
• 3 coherent receivers:• Splits on 2 heater types and 4x4 MMI
vs directional couplers• 2 IQ Modulators
• Traveling wave• Segmented
• 1 MZM + monolithically integrated driver
Loaded electrode Phase shifter Full TW MZM
47 GHz 43 GHz10 GHzdue to velocity mismatch
Traveling wave modulator: Gen1 Gen26mm x 6 mm tile
Coherent Link Modeling
-10 to 40 dBm -10 to 40 dBm
Sensitivity:50 uA
Traveling wave or Segmented• Length• 𝑉𝑉π𝐿𝐿π• α𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜• Differential drive• Total capacitance
η𝑜𝑜𝑜𝑜𝑙𝑙𝑙𝑙𝑙𝑙 = 20%
η𝐿𝐿𝐿𝐿 = 20%
13 dB
20% efficiency
150 mW per TIA
204 mW for OPLL circuit R = 1.0 A/W
𝑉𝑉𝑆𝑆𝑜𝑜𝑆𝑆𝑆𝑆𝑜𝑜𝑜𝑜
Assumed:• BER 10−12• 56 GBd
INTREPID Review, September 2019
1. Find optimal LO and TX laser operating point on blue curve.2. Find energy per bit (EPB) for corresponding TX laser power along orange curve.
Full-Link Energy Efficiency
INTREPID Review, September 2019
Si TW-MZM
1
2
InP TW-MZM
56 GBdBER 10−12
• Si photonic receiver—support for dual-pol• Significant efficiency advantage for traveling-wave modulators
compared to segmented designs• Significant efficiency advantage for InP vs Si modulators
Talks and PublicationsPlenary Talks• L. A. Coldren, " Photonic Integrated Circuits for Coherent Communication and Sensing," MOC '18,
Plenary talk, Taipei, October 16, 2018 (Micro-optics Conf.)• C. Schow, “Opportunities for Si Photonics in Next-Generation Data Centers” IEEE Components and
Packaging Symposium Japan, Plenary talk, Kyoto, Japan 2018.Invited Talks and Workshop Presentations• C. L. Schow, PANEL SESSION 1: Optics Invading Copper? The Future of Backplane Communication, CSICS,
Miami, October 2017.• C. L. Schow, “Low Power Analog Coherent Links for Next-Generation Datacenters, Next-Generation Optical
Networks (NGON), San Francisco, CA, October 2017 .• C. L. Schow, “Can Coherent Optics Invade the Datacenter, and then HPC Systems?,” North American
Workshop on Silicon Photonics for High Performance Computing, Fort Collins, CO, May 2018.• C. L. Schow, “Ultra-Low Power Short-Reach Interconnects for 100G and Beyond,” 6th International
Symposium for Optical Interconnect in Data Centres, ECOC, Rome, Italy, Sept. 2018.• C. L. Schow, “Low Power Analog Coherent Links for Next-Generation Datacenters,” JIEP OPT Workshop,
Tokyo, Japan, November 2018.• C. L. Schow and K. Schmidtke, “INTREPID: Developing Power Efficient Analog Coherent Interconnects to
Transform Data Center Networks,” Open Compute Project (OCP) Global Summit 2019, San Jose, CA, March 2019.
• C. L. Schow and K. Schmidtke, “INTREPID: Developing Power Efficient Analog Coherent Interconnects to Transform Data Center Networks,” Optical Fiber Communication (OFC) Conference 2019, Paper M4D.9, San Diego, CA, USA, Mar. 2019.
• C. L. Schow, “Beyond 400G for Hyperscale Data Centers,” Optical Fiber Communication (OFC) Conference 2019, panelist, San Diego, CA, USA, Mar. 2019.
INTREPID Review, September 2019
Talks and Publications (continued)Invited Talks (continued)• C. L. Schow, “Low Power Analog Coherent Links for Next-Generation Datacenters,” Conference on Lasers and
Electro-Optics (CLEO) 2019, paper STh4N.3, San Jose, CA, USA, May 2019.• C. L. Schow, “Low Power Coherent Links to Enable New System Architectures,” 2nd North American
Workshop on Silicon Photonics for High Performance Computing, Estes Park, CO, USA, May 2019.• J. Buckwalter, “Picojoule-per-Bit Coherent Links through Co-Design of Photonic and Electronic ICs,” European
Conference on Optical Communication (ECOC), paper Tu.1.E.1, Dublin, Ireland, Sept. 2019.
Journal and Conference Papers• S. Arafin, L. A. Coldren, "Advanced InP Photonic Integrated Circuits for Communication and Sensing," JSTQE,
24 (1) Jan/Feb, 2018. Invited Paper• H. Andrade et al., “Monolithically-Integrated 50Gbps 2pJ/bit Photoreceiver with Cherry-Hooper TIA in 250nm
BiCMOS Technology,” Optical Fiber Communication (OFC) Conference 2019, Paper M3A.5, San Diego, CA, USA, Mar. 2019.
• T. Hirokawa, S. Pinna, J. Klamkin, J. Buckwalter, C. L. Schow, “Energy Efficiency Analysis of Coherent Links for Datacenters,” IEEE Optical Interconnects Conference, paper TuC1, Santa Fe, NM, April 2019.
• A. Maharry, H. Andrade, T. Hirokawa, J. F. Buckwalter, and C. L. Schow, “A Novel Architecture for a Two-Tap Feed-Forward Optical or Electrical Domain Equalizer Using a Differential Element,” IEEE Photonics Conference, paper TuH2.2, San Antonio, TX, Oct. 2019.
Workshops Organized• “Si Photonics for the data centre” ECOC, Gothenberg, Sweden, September 2017.• “Has the time come for coherent optics in the data centres?” ECOC, Rome, Italy, 2018.• “What Will Drive the Transition to Coherent Intra-Data-Center Optics?” OFC, San Diego, CA, 2020
INTREPID Review, September 2019
Coherent Optics is Coming to Datacenters
INTREPID Review, September 2019
2018
2019
2020
Tech to Market
Katharine Schmidtke
Could-Scale Drives the Market
Global Sales of Ethernet Transceivers by Application
Source: LightCounting
$ Bi
llion
0
4
8
2
6
Forecast
T2M Critical Metrics
• Power consumption
• Reliability
• Cost
• Manufacturing volume
INTREPID Review, September 2019
Business Models for Co-Packaged
• Switch driven
• Photonics driven
• OEM/ODM driven
• Customer driven
INTREPID Review, September 2019
Building an Ecosystem: Need for Open Interfaces
• INTREPID interface standards shared with the industry through open forums such as OCP, TIP or other industry MSA group
INTREPID Review, September 2019
• Build ecosystem to enable innovation• JDF Collaboration with Microsoft
Photonic Integration of Switching ASIC using INTREPID’sCo-Packaged, Coherent, WDM Optics – Schematic View
24INTREPID Review October 2018
• Analog Coherent, Dual-Pol, QPSK with WDM over SM Fiber– Basic Symbol Rate: 𝝆𝝆 = 𝟓𝟓𝟓𝟓 Gbaud(same as bit rate per server &bit rate per switch ASIC port)
– Multiplexing Factor: 𝝁𝝁 = 𝟐𝟐 × 𝟐𝟐 = 𝟒𝟒 b/symbol
– Bit Rate per Wavelength: 𝒓𝒓 = 𝝁𝝁𝝆𝝆 = 𝟐𝟐𝟓𝟓𝟓𝟓 Gb/s
– Number of Wavelengths per fiber: 𝝂𝝂 = 𝟒𝟒 (up to 𝟏𝟏𝟏𝟏)
– Bit Rate per Fiber: 𝑹𝑹 = 𝝂𝝂𝒓𝒓 = 𝟖𝟖𝟓𝟓𝟓𝟓 (up to 𝟑𝟑𝟐𝟐𝟓𝟓𝟓𝟓) Gb/s
Conventional, 3-Level, ToR-Based Data Center with 800 Gb/s per Fiberwith an Oversubscription Ratio (OSR) = 1, it Supports 131,072 Servers
25INTREPID Review October 2019
• Number of the 51.2 Tb/s Spine and Aggregation Switches = 384• Number of 6.4 Tb/s ToR switches = 2,048• If a higher OSR is used, which is often done to reduce cost, the resulting
bandwidth constriction can reduce the system utilization for multi-rack communications The above 3-D representation is similar to that used by Alexey Andreyev,
EoR-Based Data Center with 4-λ, 800 Gb/s per Fiber having only two electronic switching levels and one intermediate layer of 4 x 4 AWGRs
with an OSR = 1, it also Supports 131,072 Servers
26INTREPID Review October 2019
• Number of the 51.2 Tb/s spine and aggregation switches = 384 – same as before• The 2048 ToR switches have been replaced by the same number of 4x4 AWGRs• thus saving cost, power and latency (due to reducing the number of switching levels)• Also, if a higher OSR is used, the system utilization within an EoR cluster of racks is
markedly increased because of replacing the single-rack ToRs with multi-rack EoRs
EoR-Based Data Center with 8-λ, 1600 Gb/s per Fiber having only two electronic switching levels and one intermediate layer of 8 x 8 AWGRs
with an OSR = 1, it also Supports 131,072 Servers
27INTREPID Review October 2019
• Same number of 51.2 Tb/s switches as before, but now with 512 8x8 AWGRs• Same cost, power and latency savings as previous design, and same improvement in
the overall system utilization within an EoR cluster of racks if a higher OSR is used• An improvement over the previous design is that the degree of integration is doubled,
thus the number of fibers per switch is halved, without reducing the number of servers
EoR-Based Data Center with 16-λ, 3200 Gb/s per Fiber having only two electronic switching levels and one intermediate layer of 16 x 16 AWGRs
with an OSR = 1, it also Supports 131,072 Servers
28INTREPID Review October 2019
• The degree of integration is again doubled by using 16 λs and 16x16 AWGRs• All the previous performance improvements are maintained, but the number of fibers
per switch is halved again, without reducing the number of supported servers … But,• is it more feasible to keep increasing the degree of transceiver integration and reduce
their number, or to co-package a larger number of simpler transceivers with the ASIC ?
Integration versus Scalability of Data CentersComparing ToR-Based, 3-Level, and EoR/AWGR-Based, 2-Level, Designs
29INTREPID Review October 2019
Integration versus Scalability of Data CentersAdding EoR-Based, 2-, 3- and 4-Level Designs
30INTREPID Review October 2019
Outlook/Next Steps
‣ Finish assembly and characterization of Phase 1 VCSEL and analog coherent links
– 8XP ICs received 10/18/2019– 9WG Si photonics expected 11/12/2019
‣ Meet Phase 1 metrics‣ Update and refine link performance and power modeling‣ Transition to Phase 2‣ Carry out IP-based simulations comparing the performance
of ToR-, EoR- and AWGR-based data center designs‣ Assess the cost, performance and energy efficiency of data