Accelerating Snabb Programs€¦ · Accelerating Snabb Programs Fabian Bonk May 1, 2018 Chair of Network Architectures and Services Department of Informatics Technical University

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Accelerating Snabb Programs

Fabian Bonk

May 1, 2018

Chair of Network Architectures and ServicesDepartment of Informatics

Technical University of Munich

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb

Snabb is an extensible, virtualized, Ethernet networkingtoolkit.

Programmers write Lua modules that handle packets and connect themtogether.

Fabian Bonk – Accelerating Snabb Programs 2

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb

Snabb is an extensible, virtualized, Ethernet networkingtoolkit.

Programmers write Lua modules that handle packets and connect themtogether.

Fabian Bonk – Accelerating Snabb Programs 2

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Naming conventions

• Apps (= Modules)• Programs (= Configs)

Fabian Bonk – Accelerating Snabb Programs 3

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Naming conventions

• Apps (= Modules)

• Programs (= Configs)

Fabian Bonk – Accelerating Snabb Programs 3

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Naming conventions

• Apps (= Modules)• Programs (= Configs)

Fabian Bonk – Accelerating Snabb Programs 3

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Fabian Bonk – Accelerating Snabb Programs 4

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Fabian Bonk – Accelerating Snabb Programs 5

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb Core

Snabb Core = Snabb runtime

• initialize apps• set up links between apps• memory management• main loop

Fabian Bonk – Accelerating Snabb Programs 6

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb Core

Snabb Core = Snabb runtime

• initialize apps• set up links between apps• memory management• main loop

Fabian Bonk – Accelerating Snabb Programs 6

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb Core

Snabb Core = Snabb runtime

• initialize apps

• set up links between apps• memory management• main loop

Fabian Bonk – Accelerating Snabb Programs 6

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb Core

Snabb Core = Snabb runtime

• initialize apps• set up links between apps

• memory management• main loop

Fabian Bonk – Accelerating Snabb Programs 6

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb Core

Snabb Core = Snabb runtime

• initialize apps• set up links between apps• memory management

• main loop

Fabian Bonk – Accelerating Snabb Programs 6

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb Core

Snabb Core = Snabb runtime

• initialize apps• set up links between apps• memory management• main loop

Fabian Bonk – Accelerating Snabb Programs 6

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb API

Appsmyapp:new(arg)myapp:push()myapp:pull()

Programslocal c = config.new()config.app(c, "app1", myapp, "params for app 1")config.app(c, "app2", myapp, "params for app 2")config.link(c, "app1.output -> app2.input")

engine.configure(c)engine.main()

Running

$ snabb run myprogram

Fabian Bonk – Accelerating Snabb Programs 7

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb APIAppsmyapp:new(arg)myapp:push()myapp:pull()

Programslocal c = config.new()config.app(c, "app1", myapp, "params for app 1")config.app(c, "app2", myapp, "params for app 2")config.link(c, "app1.output -> app2.input")

engine.configure(c)engine.main()

Running

$ snabb run myprogram

Fabian Bonk – Accelerating Snabb Programs 7

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb APIAppsmyapp:new(arg)myapp:push()myapp:pull()

Programslocal c = config.new()config.app(c, "app1", myapp, "params for app 1")config.app(c, "app2", myapp, "params for app 2")config.link(c, "app1.output -> app2.input")

engine.configure(c)engine.main()

Running

$ snabb run myprogram

Fabian Bonk – Accelerating Snabb Programs 7

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Snabb APIAppsmyapp:new(arg)myapp:push()myapp:pull()

Programslocal c = config.new()config.app(c, "app1", myapp, "params for app 1")config.app(c, "app2", myapp, "params for app 2")config.link(c, "app1.output -> app2.input")

engine.configure(c)engine.main()

Running

$ snabb run myprogramFabian Bonk – Accelerating Snabb Programs 7

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

DPDK

DPDK = Data Plane Development KitDPDK is a set of libraries and drivers forfast packet processing.

Fabian Bonk – Accelerating Snabb Programs 8

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

DPDK

DPDK = Data Plane Development Kit

DPDK is a set of libraries and drivers forfast packet processing.

Fabian Bonk – Accelerating Snabb Programs 8

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

DPDK

DPDK = Data Plane Development KitDPDK is a set of libraries and drivers forfast packet processing.

Fabian Bonk – Accelerating Snabb Programs 8

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

libmoon

LuaJIT + DPDK = fast and flexible packet processing atspeeds above 100 Gbit/s.

Lua wrapper around DPDK

Fabian Bonk – Accelerating Snabb Programs 9

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

libmoon

LuaJIT + DPDK = fast and flexible packet processing atspeeds above 100 Gbit/s.

Lua wrapper around DPDK

Fabian Bonk – Accelerating Snabb Programs 9

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

What we have

Fabian Bonk – Accelerating Snabb Programs 10

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

What we want

Fabian Bonk – Accelerating Snabb Programs 11

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Solution

Build snabb-libmoon-compat

• Reimplement Snabb Core to call into libmoon• Expose DPDK devices as Snabb Apps

Fabian Bonk – Accelerating Snabb Programs 12

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Solution

Build snabb-libmoon-compat

• Reimplement Snabb Core to call into libmoon• Expose DPDK devices as Snabb Apps

Fabian Bonk – Accelerating Snabb Programs 12

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Solution

Build snabb-libmoon-compat

• Reimplement Snabb Core to call into libmoon

• Expose DPDK devices as Snabb Apps

Fabian Bonk – Accelerating Snabb Programs 12

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Solution

Build snabb-libmoon-compat

• Reimplement Snabb Core to call into libmoon• Expose DPDK devices as Snabb Apps

Fabian Bonk – Accelerating Snabb Programs 12

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Caveats

• no multiprocessing/multithreading (yet?)• only partial API compatibility

Fabian Bonk – Accelerating Snabb Programs 13

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Caveats

• no multiprocessing/multithreading

(yet?)• only partial API compatibility

Fabian Bonk – Accelerating Snabb Programs 13

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Caveats

• no multiprocessing/multithreading (yet?)

• only partial API compatibility

Fabian Bonk – Accelerating Snabb Programs 13

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Caveats

• no multiprocessing/multithreading (yet?)• only partial API compatibility

Fabian Bonk – Accelerating Snabb Programs 13

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

snabb-libmoon-compat

• Apache 2.0 License• Lua only (no C)• https://github.com/Reperator/snabb-libmoon-compat

• https://fabianbonk.de/snabb-libmoon-compat/

Fabian Bonk – Accelerating Snabb Programs 14

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

snabb-libmoon-compat

• Apache 2.0 License

• Lua only (no C)• https://github.com/Reperator/snabb-libmoon-compat

• https://fabianbonk.de/snabb-libmoon-compat/

Fabian Bonk – Accelerating Snabb Programs 14

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

snabb-libmoon-compat

• Apache 2.0 License• Lua only (no C)

• https://github.com/Reperator/snabb-libmoon-compat

• https://fabianbonk.de/snabb-libmoon-compat/

Fabian Bonk – Accelerating Snabb Programs 14

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

snabb-libmoon-compat

• Apache 2.0 License• Lua only (no C)• https://github.com/Reperator/snabb-libmoon-compat

• https://fabianbonk.de/snabb-libmoon-compat/

Fabian Bonk – Accelerating Snabb Programs 14

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

snabb-libmoon-compat

• Apache 2.0 License• Lua only (no C)• https://github.com/Reperator/snabb-libmoon-compat

• https://fabianbonk.de/snabb-libmoon-compat/

Fabian Bonk – Accelerating Snabb Programs 14

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Performance

Performance target was achieved

Snabb’s maximum packet rate: 14.88 Mpps (no support for faster NICs)

snabb-libmoon-compat: up to 18.65 Mpps (using 40GbE NIC)

Fabian Bonk – Accelerating Snabb Programs 15

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Performance

Performance target was achieved

Snabb’s maximum packet rate: 14.88 Mpps (no support for faster NICs)

snabb-libmoon-compat: up to 18.65 Mpps (using 40GbE NIC)

Fabian Bonk – Accelerating Snabb Programs 15

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Performance

Performance target was achieved

Snabb’s maximum packet rate: 14.88 Mpps (no support for faster NICs)

snabb-libmoon-compat: up to 18.65 Mpps (using 40GbE NIC)

Fabian Bonk – Accelerating Snabb Programs 15

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Performance

Performance target was achieved

Snabb’s maximum packet rate: 14.88 Mpps (no support for faster NICs)

snabb-libmoon-compat: up to 18.65 Mpps (using 40GbE NIC)

Fabian Bonk – Accelerating Snabb Programs 15

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Test setup

Fabian Bonk – Accelerating Snabb Programs 16

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Test systems

CPU CPU freq. L3 cache NICIntel Xeon E5-2620 v3 3.2 GHz 15 MiB Intel XL710

Intel 82599ES2 Intel Xeon E5-2630 v4 3.1 GHz 25 MiB Intel XL710

Intel 82599ESIntel i7-3770K 4.5 GHz 8 MiB Intel 82599ESAMD Threadripper 1950X 4.2 GHz 32 MiB Intel 82599ES

Fabian Bonk – Accelerating Snabb Programs 17

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Intel Xeon E5-2620 v3 + Intel XL710/Intel 82599ES

2 4 6 8 10 12 14 16 18 200

10

20

30

10GbE maximum

~XL710 maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)snabb-libmoon-compat(40GbE)

Fabian Bonk – Accelerating Snabb Programs 18

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Intel Xeon E5-2620 v3 + Intel XL710/Intel 82599ES

10 20 30 40 50 60 70 80 90 1000

10

20

30

10GbE maximum

~XL710 maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)snabb-libmoon-compat(40GbE)

Fabian Bonk – Accelerating Snabb Programs 19

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Intel Xeon E5-2630 v4 + Intel XL710/Intel 82599ES

2 4 6 8 10 12 14 16 18 200

10

20

30

10GbE maximum

~XL710 maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)snabb-libmoon-compat(40GbE)

Fabian Bonk – Accelerating Snabb Programs 20

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Intel Xeon E5-2630 v4 + Intel XL710/Intel 82599ES

10 20 30 40 50 60 70 80 90 1000

10

20

30

10GbE maximum

~XL710 maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)snabb-libmoon-compat(40GbE)

Fabian Bonk – Accelerating Snabb Programs 21

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Intel i7-3770K + Intel 82599ES

2 4 6 8 10 12 14 16 18 200

5

10

15

20

10GbE maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)

Fabian Bonk – Accelerating Snabb Programs 22

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Intel i7-3770K + Intel 82599ES

10 20 30 40 50 60 70 80 90 1000

5

10

15

20

10GbE maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)

Fabian Bonk – Accelerating Snabb Programs 23

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

AMD Threadripper 1950X + Intel 82599ES

2 4 6 8 10 12 14 16 18 200

5

10

15

20

10GbE maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)

Fabian Bonk – Accelerating Snabb Programs 24

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

AMD Threadripper 1950X + Intel 82599ES

10 20 30 40 50 60 70 80 90 1000

5

10

15

20

10GbE maximum

Number of forward apps

Pac

ketr

ate

[Mpp

s]

Snabbsnabb-libmoon-compat(10GbE)

Fabian Bonk – Accelerating Snabb Programs 25

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Future Development

• automatic/manual multithreading• expanded API compatibility• UI improvements

Fabian Bonk – Accelerating Snabb Programs 26

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Future Development

• automatic/manual multithreading

• expanded API compatibility• UI improvements

Fabian Bonk – Accelerating Snabb Programs 26

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Future Development

• automatic/manual multithreading• expanded API compatibility

• UI improvements

Fabian Bonk – Accelerating Snabb Programs 26

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Future Development

• automatic/manual multithreading• expanded API compatibility• UI improvements

Fabian Bonk – Accelerating Snabb Programs 26

Chair of Network Architectures and ServicesDepartment of InformaticsTechnical University of Munich

Links

Questions:

• [email protected](PGP 0x04124B4CFA189C26)

• [email protected](PGP 0x578B2D00E529403B)

Code:

• https://github.com/Reperator/snabb-libmoon-compat

Info & Thesis:

• https://fabianbonk.de/snabb-libmoon-compat/

Fabian Bonk – Accelerating Snabb Programs 27