THE LIMITS OF SWITCH BANDWIDTH Scott Kipp February 28, 2011.

THE LIMITS OF SWITCH BANDWIDTHScott Kipp

February 28, 2011

2

Increasing Bandwidth

• Switch designs are limited by:• ASICS that are getting faster with process improvements• 45 nm processes can support 25Gbps lanes

• Backplanes and connectors are getting faster and support 25 Gbps• FR4 will probably need to be replaced with Nelco13

• Front panel is set by 19” rack width and limits the port count• The height is optimized on 1.75” (1RU) increments and is a

differentiator

• Power of blade – optics usually only dissipate 50W of power max

• Higher speed designs will need new optical module and electrical interface standards

• 25-28 Gbps electrical signals is needed to increase the bandwidth of the switch

3

10GbE ASIC and Module Design

QSFP – 4 lanes of 10GbE

1st Generation

2nd Generation 3rd Generation

Electrical / Optical Interface

XSBI / 300 pin MSA

XAUI / X2, XENPAK, XPAK

XFI, SFI /XFP, SFP+, QSFP+

Electrical Lanes and Speeds

16 lanes at 644 MHz

4 lanes at 3.125Gbps

1 lane at 10.3125 Gbps

10GE Ports/ASIC 3 12 48 - 128

Throughput/ASIC 30 Gbps 120 Gbps 480 to 1280 Gbps

300 Pin MSA

X2

SFP+

XFPXENPAK XPAK

4

100GbE ASIC and Module Design

1st Generation 2nd Generation 3rd Generation

Electrical / Optical Interface

CAUI - CFP, CPPI – CXP, CXP2?

CAUI-4 - CFP2, CPPI-4 – QSFP, CFP4

?

Electrical Interface

10 lanes at 10 Gbps

4 lanes at 25Gbps

?

100GE Ports/ASIC 4 with CFP32 with CXP

12 ?

Throughput/ASIC 480 to 1280 Gbps

1200 to 2560 Gbps

?

CFP QSFP+CFP2

CXP

CFP4

Under DevelopmentBy CFP MSA

CXP2?

5

Switch Design• Switches are typically designed for a 19” rack

and this allows about 16-17” of port space for ports• Highest densities so far are 64 mini SFP+ (mSFP)

ports at 10GE or 48 SFP+ and 4 QSFP ports• Blades inserted horizontally or vertically

1U Switch

Modular Switch with 64 mSFP+ ports

Modular Switch with 48 SFP+ ports

6

Switch Designs with 10Gbps interfaces

48 SFP+ Design - 480 Gbps of throughput, 48 Watts* of powerLimited by size of SFP+ with 15mm pitch

4 CFP Design - 400 Gbps of throughput, 60 Watts** of powerLimited by ASICs now, but soon limited by size of CFP Port with 84mm pitch

*Based on 1 Watt of power/SFP per SFF-8431 *** Based on 2 Watts of power/QSFP+**Based on 15 Watts/CFP for 10km links ****Based on 2.5 Watts of power / CXP

CFP CFP CFP CFP

40 QSFP+ Design - 1600 Gbps of throughput, 80 Watts*** of powerLimited by ASICs now, size of QSFP+ with 21mm pitch

32 CXP Design - 3200 Gbps of throughput, 80 Watts**** of powerLimited by ASICs, CXP has 27mm pitch

7

Summary of 10 Gbps Switch Designs

• Support of single-mode applications requires larger modules because of higher power dissipation

• Multimode modules support less power and are denser Throughput Max Power

/ Module10km Support

SFP+ 48*10Gbps = 480 Gbps

1.5 W Yes

CFP for 100GbE 4*100Gbps = 400 Gbps

20 W for 10km 32W for 40km

Yes

QSFP+ for 40GbE

40*40Gbps = 1.6Tbps

2 W Not without exotic cooling

CXP for 100GbE 32*100Gbps = 3.2 Tbps

2 W Not without exotic cooling

8

Switch Designs with 25-28 Gbps interfaces48 SFP+ Design - 1.3 Tbps of throughput at 32GFC, 48 Watts* of powerLimited by size of SFP+ with 15mm pitch

8 CFP2 Design - 800 Gbps of throughput, 96 Watts** of powerLimited by ASICs now, but soon limited by size of CFP2 Port with 42mm pitch

*Based on 1 Watt of power/SFP per SFF-8431 **Based on 12 Watts/CFP2 for 10km links

CFP2 CFP2 CFP2 CFP2

40 QSFP+ Design - 4 Tbps of throughput, 80 Watts*** of powerLimited by ASICs and power, QSFP+ pitch is 21mm

16 CFP4 Design - 1.6 Tbps of throughput, 96 Watts**** of powerLimited by ASICs, CXP pitch is 27mm

CFP2 CFP2 CFP2 CFP2

***Based on 2 Watt of power/QSFP****Based on 6 Watts/CFP2 for 10km links

9

Summary of 25Gbps Switch Designs

• Support of single-mode applications requires larger modules because of higher power dissipation

• Multimode modules support less power and are denser Throughput Max Power

/ Module10km Support

SFP+ for 32GFC 48*28Gbps = 1.3Tbps

1.5 W Yes, but challenging at 1.5W

CFP2 for 100GbE

8*100Gbps = 800 Gbps

6 W Yes

QSFP+ for 100GbE

40*100Gbps = 4Tbps

2 W Not without exotic cooling

CFP4 for 100GbE

16*100Gbps = 1.6 Tbps

3 W Probably 2014

Tbps Bandwidths require faster speeds

• 10Gbps lanes will limit the throughput of the switch/blade in the next few years• Difficult to expand beyond 128 ports / ASIC

• Module size limits bandwidth and causes long trace lengths

• In the next few years, 25-28Gbps signaling needed to support Tbps switch/blade bandwidths• 4X25Gbps easier to support than 10X10Gbps in near

future

• Cost of 25-28G signaling compared to 10Gbps will be an interesting battle in the next few years

• Terabit Ethernet will require major innovation• 100x10G or 40x25G not practical for pluggable modules

• 10X100G not practical from signaling perspective

11

Thank You