Breaking Open Linux Switching Drivers Andy Gospodarek Cumulus Networks 16 Feb 2015 netdev01
Breaking Open Linux Switching Drivers
Andy Gospodarek
Cumulus Networks16 Feb 2015
netdev01
cumulusnetworks.com 2
Agenda
• Why• History• Design proposal• Future possibilities
cumulusnetworks.com 3
Why am I here?
cumulusnetworks.com 4
Linux kernel should enable others to
create the next generation of forwarding
devices
cumulusnetworks.com 5
Integrate support for offload hardware
directly into the the Linux kernel
cumulusnetworks.com 6
Hardware Platform History
cumulusnetworks.com 7
Market dominated by switch and
router vendors providing expensive
proprietary solutions
cumulusnetworks.com 8
Proprietary software running on switches
and routers was not open for developers
and users to enhance
cumulusnetworks.com 9
Today’s hardware platforms are
significantly higher-performance and more
generally available
cumulusnetworks.com 10
Spare CPU cycles are available for
applications to run directly on the switch
cumulusnetworks.com 11
Bare-metal platforms are now appealing
and available to commercial Linux
vendors, developers, and users
cumulusnetworks.com 12
Software History
cumulusnetworks.com 13
15+ years with Linux as a viable OS for host
processor on switches/routers
cumulusnetworks.com 14
10+ years Linux “support” by ASIC vendors
cumulusnetworks.com 15
Basic in-kernel switching/offload layer support in v3.19
cumulusnetworks.com 16
Software architecture to control ASICs
has not fundamentally
changed in the last decade
cumulusnetworks.com 17
What exactly does that look like?
Typical packet path
cumulusnetworks.com 18
Switching ASIC
vendordriver
Userspace Applicationand
Vendor SDKSocket applications
Virtual devices (tun/tap)
NIC tools and consumers
NIC Counters
Control Plane Programming
cumulusnetworks.com 19
iproute2 and friends
vendor driver
Routing suites, bridge controllers, etc
Init
Virtual devices (tun/tap)
Linux kernel structures
FDB/FIB
Switching ASIC
Userspace Applicationand
Vendor SDK
cumulusnetworks.com 20
Painful for those developing switches as
management applications need to talk to
kernel/netlink and SDKs
Netlink Control Path
cumulusnetworks.com 21
vendor driver
iproute2 and friends
Routing suites, bridge controllers, etc
Init
Virtual devices (tun/tap)
Linux kernel structures
FDB/FIB
Switching ASIC
Userspace Applicationand
Vendor SDK
cumulusnetworks.com 22
Much better design, but each SDK
supported needs a new translation
between netlink and SDK
cumulusnetworks.com 23
Kernel hackers and distribution vendors
see a simple solution
cumulusnetworks.com 24
Get rid of all closed-source SDKs
cumulusnetworks.com 25
Great idea!
cumulusnetworks.com 26
Vendors do not want to open
source their SDKs
cumulusnetworks.com 27
Can we use a userspace SDK and a
kernel driver at the same time?
cumulusnetworks.com 28
cumulusnetworks.com 29
Not if you want it upstream!
cumulusnetworks.com 30
OK...how do I get started?
cumulusnetworks.com 31
Phased Approach
• Participate!• Pick a hardware platform• Write and post a switchdev-compatible
network driver• Enhance that driver to add
ndo/offload_ops to driver
cumulusnetworks.com 32
Attend conferences, participate on
mailing-lists, and post patches
cumulusnetworks.com 33
Write and post a switchdev-compatible
network driver
cumulusnetworks.com 34
Advantages
• Provide network access via front panel ports
• Phased approach to working upstream• Applications can developed without need
for hardware offload
cumulusnetworks.com 35
What might that look like?
Base Switchdev Driver
cumulusnetworks.com 36
Socket applications
Ethtool
NIC tools and consumers
switchdev compatible driver
Switching ASIC
CountersLink MgmtPort Init RxTx
cumulusnetworks.com 37
Great, we are upstream!
Are we done?
cumulusnetworks.com 38
No!
cumulusnetworks.com 39
Add offload support to driver as
upstream infrastructure is developed
cumulusnetworks.com 40
What might that look like?
Switchdev Offload Driver
cumulusnetworks.com 41
Socket applications
Ethtool
NIC tools, Routing suites, bridge controllers, etc
switchdev compatible driver
Switching ASIC
CountersLink MgmtPort Init Offload Ops
iproute2 and friends
RxTx
cumulusnetworks.com 42
“If you are the first you will be so cool.”
-DaveM
cumulusnetworks.com 43
Get coding
cumulusnetworks.com 44
“..and we’ll help you maintain it”
-DaveM
© 2013 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
▪Thank You!
cumulusnetworks.com 45