The Role of the FPGA in 400GbE Technology Development

© 2013 Ethernet Alliance 1

Gordon Brebner Distinguished Engineer Xilinx, Inc.

The Role of the FPGA in 400GbE Technology Development

2 © 2013 Ethernet Alliance

Disclaimer

The views we are expressing in this presentation are our own personal views and

should not be considered the views or positions of the

Ethernet Alliance.

3 © 2013 Ethernet Alliance

400GbE PCS/MAC

  Expect first: 16 PCS lanes, each at 25.78125 Gb/s   Glueless interface to optics   Possible re-use of the 802.3ba PCS   Other options possible for PCS, maybe native FEC

  Later: 8 lanes, each at 51.56G   Or 4 lanes with 2 bits/symbol at 56Gbaud (e.g. PAM4)

  Packet size 64 bytes to 9600 bytes

  Use 100GbE building blocks where possible

4 © 2013 Ethernet Alliance

Silicon technology

  Technology nodes (silicon feature size)   130nm, 65nm, 40nm, 28/32nm, 20/22nm, 14/16nm

  Application-Specific Integrated Circuit (ASIC)   Fixed chip   Increasingly expensive: need high volumes   Best suited to post-standardization Ethernet

  Field Programmable Gate Array (FPGA)   Programmable logic chip   Suitable for prototyping and medium volumes   Best choice for pre-standardization Ethernet

5 © 2013 Ethernet Alliance

400GbE line/system bridge

500G

Interlaken

40 x 12.5G or

48 x 10G SERDES

Bridge logic

400GbE

PMA/PCS

CDFP or

4xCFP4

Optical

16 x 25G SERDES 400GbE

MAC

Wide parallel data path between blocks

ASIC or FPGA chip

Line side System side

6 © 2013 Ethernet Alliance

MAC rate = Width x Clock

400 Gb/s and 1 Tb/s Ethernet MAC options

MAC rate Silicon node Technology Data path width Clock frequency

100 Gb/s 45, 40nm ASIC 160 bits 644 MHz

100 Gb/s 45, 40nm FPGA 512 bits 195 MHz

400 Gb/s 28, 20nm ASIC 400 bits 1 GHz

400 Gb/s 28, 20nm FPGA 1024 bits 1536 bits

400 MHz 267 MHz

1 Tb/s 20, 14nm ASIC 1024 bits 1 GHz

1 Tb/s 20, 14nm FPGA 2048 bits 2560 bits

488 MHz 400 MHz

7 © 2013 Ethernet Alliance

Multiple Packets/Word

  Up to 512-bit, only one packet completed   Just need to deal with EOP then SOP in word

  Beyond 512-bit, multiple packets completed   Need to add parallel packet processing   Must deal with varying EOP and SOP positions

Bus width Max packets Max EOPs

512 2 1

1024 3 2

1536 4 3

512 * n n+1 n

8 © 2013 Ethernet Alliance

400GbE CRC Example

  All Ethernet packets carry Cyclic Redundancy Code (CRC) for error detection   Computed using CRC-32 polynomial   Critical function within Ethernet MAC

  Requirements   Computed at line rate   Deal with multiple packets in wide data path   Economical with silicon resources

9 © 2013 Ethernet Alliance

400GbE CRC Prototype

  Xilinx Labs research project   Modular: built out of 512-bit 100G units   Computes multiple CRCs per data path word   Targeting 28nm FPGA (Xilinx Virtex-7 series)

N-bit data path partitioned into 512-bit sections

512-bit unit CRC results combined to get final CRC results

10 © 2013 Ethernet Alliance

400GbE CRC Prototype

  Results:

  1024-bit width is feasible for 400GbE   Other widths:

  Less challenging clock frequencies   Demonstrate scalability beyond 400GbE

Data bus word size 1024-bit 1536-bit 2048-bit

Max clock frequency (MHz) 400 381 326

Maximum line rate (Gb/s) 409 585 668

Latency (ns) 17.5 18.4 21.5

FPGA resources (slices) 2,888 4,410 5,719

11 © 2013 Ethernet Alliance

Pre-standard 400GbE

  PCS/MAC layer implementation on FPGA:   64B to 9600B packets   Priority pause handling, two queues in egress

direction with individual flow control   Link fault handling   Full statistics   Optional 1588 timestamping   16 x CEI-28G-VSR interface

  Unknown: actual final nature of PCS

  Work under way - ready in 2014

12 © 2013 Ethernet Alliance

400GbE bridge on FPGA

  Current (28nm) generation of FPGAs   Two-FPGA implementation

Option (a) Directions separated Option (b) Functions partitioned

13 © 2013 Ethernet Alliance

400GbE bridge on FPGA

  Future (20nm) generation of FPGAs – 2014   Single-FPGA implementation

  Estimated resources:   644K logic cells (without FEC), 835K (with FEC)

14 © 2013 Ethernet Alliance

4x100GbE Aggregation

  Xilinx research prototype   Includes 32x8 128-bit crossbar switch and scheduler   Aggregates 4x100G to 400G   Targeting 28nm FPGA

  Results:

BUNDLE  0

BUNDLE  2

BUNDLE  1

BUNDLE  3

BUNDLE  4 BUNDLE  5

BUNDLE  6 BUNDLE  7

BUNDLE  8 BUNDLE  9

BUNDLE  11

BUNDLE  13

BUNDLE  10

BUNDLE  12

BUNDLE  15

BUNDLE  17

BUNDLE  19

BUNDLE  21

BUNDLE  23

BUNDLE  14

BUNDLE  20

BUNDLE  18

BUNDLE  16

BUNDLE  22

BUNDLE  24

BUNDLE  26

BUNDLE  30

BUNDLE  28

BUNDLE  25

BUNDLE  27

BUNDLE  29

BUNDLE  31

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

BRAM

CMAC

3

CMAC

2

CMAC

4

CMAC

1

Data bus word size 1024-bit

Max clock frequency (MHz) 450

Line rate (Gb/s) 460

Latency (# of clock cycles) 10

FPGA resources (logic cells) 20,851

15 © 2013 Ethernet Alliance

Conclusions

  Can anticipate 400GbE PCS/MAC standard

  Ever-increasing rates mean ever-wider internal data path width in electronics   Leading to multiple packets per data word

  Demonstrated modular Ethernet CRC block based on 100GbE units   Silicon resource scales linearly with line rate

  Possible to prototype pre-standard 400GbE PCS/MAC using today’s FPGA technology