OSDC 2014 ONIE by Nat Morris

Open Network Install Environment

NETWAYS OSDC 2014, Berlin

Nat Morris

9th April 2014

•Overview

•What Is ONIE?

•Lessons Learned

•ONIE Development

•Demo

Agenda

IP Fabric Networking Landscape

Netw

ork

H

ard

ware

Netw

ork

O

S

Open Closed

cumulusnetworks.com

The Expanding Landscape

hardware

operating system

appapp

hardware

operating system

app app

Single Vendor Blob

Multi-Vendor Ecosystem

app app

cumulusnetworks.com

Understanding Characteristics of a Leaf Switch

cumulusnetworks.com

10/40 Gigabit spine uplink ports

Serial console port

Ethernet Out-of-Band

Management Port

1/10 Gigabit Ethernetnetwork access ports

* SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options* QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options

Understanding Characteristics of a Spine Switch

cumulusnetworks.com

Serial console port

Ethernet Out-of-Band Management Port

* QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options

40G / 10G Fabric ports

Bare Metal Switch Provisioning

Similar approach to installing OS on server

BIOS + PXE = U-Boot + ONIE (Open Network Install Environment)

Supported hardware (HCL) preloaded with ONIE

ONIE available on GitHub• http://onie.github.io/onie/

bare metal server

operating system

app

app

app

BIOS and PXEbare metal

switch

operating system

app

app

app

U-Boot and ONIE

cumulusnetworks.com

Choice

cumulusnetworks.com

Choice

cumulusnetworks.com

What Is ONIE?

Network OS Install Environment Provides an environment for network OS installer

discovery and execution Like a pre-installed kickstarter Defined by its behaviors Implemented using a modern Linux kernel and BusyBox

An Open Source Project within OCP http://www.onie.org/

Evolving …

cumulusnetworks.com

Hardware Vendors

cumulusnetworks.com

Operating System Vendors

cumulusnetworks.com

ONIE: Anatomy of a Network Switch

( Management Interfaces ) ( Data Plane )

CPUSoC

DRAMBootFlash

MassStorage

SwitchingASIC

SerialConsole

EthernetMgmt Port

10GbPort

40GbPort…10Gb

Port40GbPort

…

PCIe

cumulusnetworks.com

ONIE: Uses Management Interfaces

( Management Interfaces ) ( Data Plane )

CPUSoC

DRAMBootFlash

MassStorage

SwitchingASIC

SerialConsole

EthernetMgmt Port

10GbPort

40GbPort…10Gb

Port40GbPort

…

PCIe

cumulusnetworks.com

ONIE: Bare Metal Install – First Time Boot Up

Boot Loader(HW Vendor Supplied)

ONIE(HW Vendor Supplied)

Installer(OS Vendor)

Boot Loader• Low Level boot loader, configures CPU complex• Loads and boots ONIE

ONIE• Linux Kernel with Busybox• Configures management Ethernet interface• Locates and executes an OS installer• Provides tools and environment for installer

OS Installer• Available from network or USB• Linux executable• Installs vendor OS into mass storage

Network OS(OS Vendor Supplied)

Fetches

Installs

cumulusnetworks.com

ONIE: Subsequent Reboots – Vendor’s OS is Already Installed

Boot Loader(HW Vendor Supplied)

ONIE(HW Vendor Supplied)

Boot Loader• Low Level boot loader, configures CPU complex• Loads and boots OS vendor’s installed OS

Network OS• Configures Switching ASIC• Runs Network Protocols• Provides CLI

Network OS(OS Vendor Supplied)

ONIE• Still exists, but is not used• Available for uninstall / re-install operations

cumulusnetworks.com

ONIE: Network OS Installer Discovery and Install Behavior

Configure Network Interface

Locate Installer

Run Installer

• Uses DHCPv4, DHCPv6• Configures Ethernet interface for IPv4 / IPv6• Configures DNS and hostname

• Determines the location of an installer executable• Examines local file systems, e.g. USB flash drives• Uses DHCP options, DNS Service Discovery, Multicast DNS

and IPv6 Neighbors

• Downloads installer via URL• Passes various environment variables to installer• Launches installer

cumulusnetworks.com

Why Not Use PXE?

• IPv6 and HTTP out of the box• Use existing Linux device drivers. No need to write new ones• Integrated automation

cumulusnetworks.com

ONIE: Other Behaviors

Provides a mechanism for a network OS to invoke the above behaviors.

Reinstall Remove currently installed OS and return to the “out of box” provisioning state

Uninstall Completely wipe out everything, except ONIE

Rescue Reboot box into ONIE for repair, debug and forensics

Update Install a new ONIE version

Diag Run HW Vendor’s diag (optional)

cumulusnetworks.com

Lessons Learned

Hardware Platforms Diverse mix of CPU complex designs, even

within a single CPU family. HW Vendors need freedom to customize ONIE Need mechanism to run HW vendor diag

Recurring Themes TLV based EEPROM format widely adopted Common HW designs reduce development time

cumulusnetworks.com

ONIE Ongoing Development

Support Multiple CPU Architectures PowerPC – Today x86 – VM Prototype available today Thinking about ARM, MIPS Maintain ONIE behaviors across architectures

Testing and Compliance Enhance and extend regression test suites Develop compliance test suites

cumulusnetworks.com

ONIE Ongoing Development

New Features DHCPv6 DNS Service Discovery / Multicast DNS

discovery HW Vendor diagnostic

Releases Quarterly release cadence Ongoing maintenance Enhancements

cumulusnetworks.com

ONIE x86 Strategy

Use existing BIOS from hardware vendors During manufacturing install GRUB2 and

ONIE-x86 on the mass storage block device NOS Installer adds partitions, installs software

and updates GRUB2 configuration.

cumulusnetworks.com

Block Device Partitioning

Partition using GUID Partition Table (GPT) format GPT supported by Linux, GRUB, UEFI Supports dual-booting multiple operating systems

Partition # Name R/W Notes

1 GRUB Boot r/o Used by GRUB and GPT

2 ONIE BOOT r/o ONIE Kernel

3 ONIE CONFIG r/w Configuration

4+ For NOS use

cumulusnetworks.com

GRUB Menu

cumulusnetworks.com

NOS Installer Duties

Create new GPT partition(s) Create file systems on partition(s) Install NOS files into partition(s) Update ONIE-CONFIG using ONIE provided

CLIs onie-boot-entry-add onie-boot-entry-remove onie-boot-entry-show onie-boot-default onie-boot-update

cumulusnetworks.com

Resources

ONIE Websites Main Page: http://www.onie.org/ Source Code: https://github.com/onie/onie/ Documentation: http://onie.github.io/onie/

cumulusnetworks.com

© 2013 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.

Thank You!

Back Up Slides – Cumulus Background

cumulusnetworks.com

Pushing Changes Down

CPU, RAM, Flash, etc. Switch Silicon

Front Panel Ports

lldpd

Routing Tables

ARP Table

Devices

Bridge FDB Filter Tables

Bonds VLANs

Lin

ux

Kern

el

Virtual Kernel Ports

Bridging

mstpd

ACLRouting Suite

Quaggasnmpd

vconfig

iptable

ebtable

ip6tableiproute

2

VXLAN Bridges

Switch HAL

brctl

SwitchDriver

Use

r Space

Quagga daemon, Quagga.conf, and vtysh

CLI and /etc/network/interfaces

switchd

cumulusnetworks.com

Cumulus Linux Hardware Compatibility List

40G Portfolio Part number DescriptionMerchant Silicon

Cumulus Linux Release

Dell S6000 32 x 40G-QSFP+ Trident II 2.1 or later

Edge-Core AS6700-32X 32 x 40G-QSFP+ Trident II 2.0 or later

Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later

Quanta QCT T5032-LY6* 32 x 40G-QSFP+ Trident II 2.0 or later

10G Portfolio Part number DescriptionMerchant Silicon

Cumulus Linux Release

Agema AG-7448CU48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident 1.5.0 or later

Dell S481048 x 10G-SFP+ and 4 x 40G-QSFP+

Trident 2.0.x or later**

Edge-Core AS5600-52X48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident+ 1.5.0 or later

Edge-Core AS5710-54X48 x 10G-SFP+ and 6 x 40G-QSFP+

Trident II 2.1 or later**

Edge-Core AS5710-96X96 x 10G-SFP+ and 8 x 40G-QSFP+

Trident II 2.1 or later**

Penguin Computing Arctica 4804X48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident+ 1.5.1 or later

Quanta QCT T-3048-LY2*48 x 10G-SFP+ and 4 x 40G-QSFP+

Trident+ 1.5.0 or later

Quanta QCT T5048-LY8*48 x 10G-SFP+ and 6 x 40G-QSFP+

Trident II 2.1 or later**

1G Portfolio Part number DescriptionMerchant Silicon

Cumulus Linux Release

Edge-Core AS4600-54T 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later

Penguin Computing Artica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2  1.5.1 or later

Quanta QCT T1048-LB9* 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or latercumulusnetworks.com