OS*: OSv on *

OS*: OSv on *

@ysmoo

BitVisor

BitVisor

Hardware

OS

App App App

http://www.justis.as-1.co.jp

https://bitbucket.org/bitvisor/bitvisor

OSv

VMM

Hardware

OSvApp

http://medical-care.feed.jp

https://github.com/cloudius-systems/osv

BitVisor

BitVisor OSv

BitVisor OSv

http://www.root.ne.jp/nishide/shs/

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSv on BitVisor

BitVisor

Hardware

OSvApp

OSb

OSb Design

BitVisor

Hardware

OSvApp

Virtio NIC

With a small fix,OSv can handle chipsets

OSv has an AHCI driver

BitVisor provides Virtio NIC

OSb Design

BitVisor

Hardware

OSvApp

Virtio NIC

With a small fix,OSv can handle chipsets

OSv has an AHCI driver

BitVisor provides Virtio NIC

One thing to fix

Skipped the code!(drivers/acpi.cc:early_init())

OSb Design

BitVisor

Hardware

OSvApp

Virtio NIC

With a small fix,OSv can handle chipsets

OSv has an AHCI driver

BitVisor provides Virtio NIC

Para Pass-throughVirtio

BitVisor

Hardware

Virtio Ring

PRO/1000 Ring

OSvApp

MSI-X etc.

Handle minimal operations.

PCI Function

Pass-through!

For more details, see:http://www.slideshare.net/yushiomote/osb-osv-on-bitvisor-2

for (12-billion times) val++;

0.00

6.50

13.00

19.50

26.00

OSb Linux (Host, Bare-metal)

25.51

20.75

Elap

sed

Tim

e (s

ec)

Netperf TCP_STREAM

0.00

237.50

475.00

712.50

950.00

OSb Linux (Host, Bare-metal)

941.39921.83

Thr

ough

put

(Mbp

s)

Ping

0.00

0.04

0.08

0.12

0.16

OSb Linux (Host, Bare-metal)

0.16

0.12

Late

ncy

(mse

c)

Thinner and thinner

Hardware

KVM/Linux

OSv

Hardware

BitVisor

OSv12.8M LOC

0.3M LOC

virtio

sched.

driver

http://cdn-ak.f.st-hatena.com/images/fotolife/j/jovi1kamiya/20140607/20140607114847.jpg

Forultimate thinness…

OSv on Physical Machine

Hardware

KVM/Linux

OSv

Hardware

BitVisor

OSv

Hardware

OSv

OSp

OSp Design

Hardware

OSv

App

Physical NICDriver

ImplementedIntel X540 10GbE Driverin C++11

With a small fix,OSv can handle chipsets

OSv has an AHCI driver

About implementation

• Intel NIC TX/RX Basics

• Device initialization

• TX initialization/operations

• RX initialization/operations

Intel NIC TX/RX Basics

Descriptor 0

Descriptor 2

Descriptor 3

Descriptor 4

Descriptor 5

Descriptor 6

Descriptor 7

Ring Buffer

Descriptor 1

Producer I/O Register Consumer I/O Register

OSv

I/O Registers

Intel NIC TX Basics

Descriptor 0

Descriptor 2

Descriptor 3

Descriptor 4

Descriptor 5

Descriptor 6

Descriptor 7

Ring Buffer

Descriptor 1

Producer I/O Register Consumer I/O Register

OSv

I/O Registers

Intel NIC TX Basics

Descriptor 0

Descriptor 2

Descriptor 3

Descriptor 4

Descriptor 5

Descriptor 6

Descriptor 7

Ring Buffer

Descriptor 1

Producer I/O Register Consumer I/O Register

OSv

I/O Registers

Intel NIC TX Basics

Descriptor 0

Descriptor 2

Descriptor 3

Descriptor 4

Descriptor 5

Descriptor 6

Descriptor 7

Ring Buffer

Descriptor 1

Producer I/O Register Consumer I/O Register

OSv

I/O Registers

Xmit!

Intel NIC RX Basics

Descriptor 0

Descriptor 2

Descriptor 3

Descriptor 4

Descriptor 5

Descriptor 6

Descriptor 7

Ring Buffer

Descriptor 1

Consumer I/O Register Producer I/O Register

OSv

I/O Registers

Recv!

Device Init.

TX Init.

TX Interface(OSv => Driver)

• OSv’s upper layer (TCP/IP) calls:

• int try_xmit_one_locked(void *mbuf_etc)

• Then, driver puts mbuf on ring buffer.

• bool kick_hw()

• Then, driver updates producer reg.

• …

EOP EOP

mbuf

try_xmit_one_locked()to put mbuf on ring buffer

kick_hw()to update producer reg.

RX Init.

RX Interface(Driver => OSv)

• struct ifnet *_ifn = if_alloc (IFT_ETHER);

• On init., driver creates ifnet and…

• ether_ifattach(_ifn, macaddr);

• …passes it to the upper layer. (Then, the upper layer puts an callback function on ifnet.if_input)

• (*_ifn->if_input)(_ifn, mbuf);

• On packet reception, driver calls the callback function (if_input) by passing mbuf as an argument.

Under improvement(A lot of TODOs)

• No interrupts… (bad polling impl.)

• No TSO, LRO, checksum assist…

• No advanced descriptor… (legacy descriptor seems slow)

• …

Netperf TCP_STREAM

0.00

2000.00

4000.00

6000.00

8000.00

OSp OSv

7229.00

4200.00

Thr

ough

put

(Mbp

s)

Future Work

0.00

2000.00

4000.00

6000.00

8000.00

OSp OSv

7229.00

4200.00

Thr

ough

put

(Mbp

s)Revenge of the OSp

One more thing

KVM/Linux

Hardware

OSvApp

./scripts/bindctrl bind 8086 1528 0000:07:00.0

./scripts/run.py -a 07:00.0

Additional scripts & optionsto start OSv withPCI pass-through config.

Output$ sudo ./scripts/run.py -n -v -c 8 -a 07:00.0ixgbe [00:04.0] 8086:1528Address: a0:36:9f:14:5f:84PHY Status: State: Connected Type: MDI-X Speed: 10Gbps Duplex: FullMAC Status: Link: UP Speed: 10Gbpsixgbe [00:04.0] 8086:1528eth0: 192.168.122.15ixgbe0: 192.168.100.2/#/#/#/# exitGoodbye

Summary

OSv　→　OSb　→　OSp

Thank you !

http://www.root.ne.jp/nishide/shs/

References

• The Ugly Patch of Intel X540 10GbE Driver and Additional scripts for PCI Pass-through Configuration

• http://goo.gl/0yRcV4

• git diff -r 5e5c227b2aec23418e5e3fd104cc21e98f31729c

• Usage & True Motivation

• http://www.yushiomote.org/blog/?p=428