OSDC 2014: Nat Morris - Open Network Install Environment

• 1. Open Network Install Environment NETWAYS OSDC 2014, Berlin Nat Morris 9th April 2014

2. cumulusnetworks.com 2 Overview What Is ONIE? Lessons Learned ONIE Development Demo Agenda 3. IP Fabric Networking Landscape cumulusnetworks.com 3 Network Hardware NetworkOS Open Closed 4. The Expanding Landscape hardware operating system appapp hardware operating system app app Single Vendor Blob Multi-Vendor Ecosystem app app cumulusnetworks.com 4 5. Understanding Characteristics of a Leaf Switch 5cumulusnetworks.com 10/40 Gigabit spine uplink ports Serial console port Ethernet Out-of-Band Management Port 1/10 Gigabit Ethernetnetwork access ports * SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options * QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options 6. Understanding Characteristics of a Spine Switch 6cumulusnetworks.com Serial console port Ethernet Out-of-Band Management Port * QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options 40G / 10G Fabric ports 7. Bare Metal Switch Provisioning Similar approach to installing OS on server BIOS + PXE = U-Boot + ONIE (Open Network Install Environment) Supported hardware (HCL) preloaded with ONIE ONIE available on GitHub http://onie.github.io/onie/ cumulusnetworks.com 7 bare metal server operating system app app app BIOS and PXE bare metal switch operating system app app app U-Boot and ONIE 8. Choice cumulusnetworks.com 8 9. Choice cumulusnetworks.com 9 10. What Is ONIE? Network OS Install Environment Provides an environment for network OS installer discovery and execution Like a pre-installed kickstarter Defined by its behaviors Implemented using a modern Linux kernel and BusyBox An Open Source Project within OCP http://www.onie.org/ Evolving cumulusnetworks.com 10 11. Hardware Vendors cumulusnetworks.com 11 12. Operating System Vendors cumulusnetworks.com 12 13. ONIE: Anatomy of a Network Switch cumulusnetworks.com 13 (ManagementInterfaces) (DataPlane) CPU SoC DRAM Boot Flash Mass Storage Switching ASIC Serial Console Ethernet MgmtPort 10Gb Port 40Gb Port 10Gb Port 40Gb Port PCIe 14. ONIE: Uses Management Interfaces cumulusnetworks.com 14 (ManagementInterfaces) (DataPlane) CPU SoC DRAM Boot Flash Mass Storage Switching ASIC Serial Console Ethernet MgmtPort 10Gb Port 40Gb Port 10Gb Port 40Gb Port PCIe 15. ONIE: Bare Metal Install First Time Boot Up cumulusnetworks.com 15 Boot Loader (HW Vendor Supplied) ONIE (HW Vendor Supplied) Installer (OS Vendor) BootLoader LowLevelbootloader,conguresCPUcomplex LoadsandbootsONIE ONIE LinuxKernelwithBusybox ConguresmanagementEthernetinterface LocatesandexecutesanOSinstaller Providestoolsandenvironmentforinstaller OSInstaller AvailablefromnetworkorUSB Linuxexecutable InstallsvendorOSintomassstorage Network OS (OS Vendor Supplied) Fetches Installs 16. ONIE: Subsequent Reboots Vendors OS is Already Installed cumulusnetworks.com 16 Boot Loader (HW Vendor Supplied) ONIE (HW Vendor Supplied) BootLoader LowLevelbootloader,conguresCPUcomplex LoadsandbootsOSvendorsinstalledOS NetworkOS ConguresSwitchingASIC RunsNetworkProtocols ProvidesCLI Network OS (OS Vendor Supplied) ONIE SSllexists,butisnotused Availableforuninstall/re-installoperaSons 17. ONIE: Network OS Installer Discovery and Install Behavior cumulusnetworks.com 17 Configure Network Interface Locate Installer Run Installer UsesDHCPv4,DHCPv6 ConguresEthernetinterfaceforIPv4/IPv6 ConguresDNSandhostname DeterminesthelocaSonofaninstallerexecutable Examineslocallesystems,e.g.USBashdrives UsesDHCPopSons,DNSServiceDiscovery,MulScast DNSandIPv6Neighbors DownloadsinstallerviaURL Passesvariousenvironmentvariablestoinstaller Launchesinstaller 18. WhyNotUsePXE? cumulusnetworks.com 18 IPv6andHTTPoutofthebox UseexisSngLinuxdevicedrivers.Noneedtowritenewones IntegratedautomaSon 19. ONIE: Other Behaviors Provides a mechanism for a network OS to invoke the above behaviors. cumulusnetworks.com 19 Reinstall Remove currently installed OS and return to the out of box provisioning state Uninstall Completely wipe out everything, except ONIE Rescue Reboot box into ONIE for repair, debug and forensics Update Install a new ONIE version Diag Run HW Vendors diag (optional) 20. Lessons Learned Hardware Platforms Diverse mix of CPU complex designs, even within a single CPU family. HW Vendors need freedom to customize ONIE Need mechanism to run HW vendor diag Recurring Themes TLV based EEPROM format widely adopted Common HW designs reduce development time cumulusnetworks.com 20 21. ONIE Ongoing Development Support Multiple CPU Architectures PowerPC Today x86 VM Prototype available today Thinking about ARM, MIPS Maintain ONIE behaviors across architectures Testing and Compliance Enhance and extend regression test suites Develop compliance test suites cumulusnetworks.com 21 22. ONIE Ongoing Development New Features DHCPv6 DNS Service Discovery / Multicast DNS discovery HW Vendor diagnostic Releases Quarterly release cadence Ongoing maintenance Enhancements cumulusnetworks.com 22 23. ONIE x86 Strategy Use existing BIOS from hardware vendors During manufacturing install GRUB2 and ONIE-x86 on the mass storage block device NOS Installer adds partitions, installs software and updates GRUB2 configuration. cumulusnetworks.com 23 24. Block Device Partitioning Partition using GUID Partition Table (GPT) format GPT supported by Linux, GRUB, UEFI Supports dual-booting multiple operating systems cumulusnetworks.com 24 Partition # Name R/W Notes 1 GRUB Boot r/o Used by GRUB and GPT 2 ONIE BOOT r/o ONIE Kernel 3 ONIE CONFIG r/w Configuration 4+ For NOS use 25. GRUB Menu cumulusnetworks.com 25 26. NOS Installer Duties Create new GPT partition(s) Create file systems on partition(s) Install NOS files into partition(s) Update ONIE-CONFIG using ONIE provided CLIs onie-boot-entry-add! onie-boot-entry-remove! onie-boot-entry-show! onie-boot-default! onie-boot-update! cumulusnetworks.com 26 27. Resources ONIE Websites Main Page: http://www.onie.org/ Source Code: https://github.com/onie/onie/ Documentation: http://onie.github.io/onie/ cumulusnetworks.com 27 28. 2013 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis. Thank You! cumulusnetworks.com 28 29. Back Up Slides Cumulus Background cumulusnetworks.com 29 30. Pushing Changes Down cumulusnetworks.com 30 CPU, RAM, Flash, etc. Switch Silicon Front Panel Ports lldpd Routing Tables ARP Table Devices Bridge FDB Filter Tables Bonds VLANs LinuxKernel Virtual Kernel Ports Bridging mstpd ACLRouting Suite Quagga snmpd vconfig iptable ebtable ip6tableiproute2 VXLAN Bridges Switch HAL brctl Switch Driver UserSpace Quagga daemon, Quagga.conf, and vtysh CLI and /etc/network/ interfaces switchd 31. Cumulus Linux Hardware Compatibility List Cumulus Networks and Partner Confidential 40G Portfolio Part number Description Merchant Silicon Cumulus Linux Release Dell S6000 32 x 40G-QSFP+ Trident II 2.1 or later Edge-Core AS6700-32X 32 x 40G-QSFP+ Trident II 2.0 or later Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later Quanta QCT T5032-LY6* 32 x 40G-QSFP+ Trident II 2.0 or later 10G Portfolio Part number Description Merchant Silicon Cumulus Linux Release Agema AG-7448CU 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 1.5.0 or later Dell S4810 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 2.0.x or later** Edge-Core AS5600-52X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later Edge-Core AS5710-54X 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later** Edge-Core AS5710-96X 96 x 10G-SFP+ and 8 x 40G-QSFP+ Trident II 2.1 or later** Penguin Computing Arctica 4804X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.1 or later Quanta QCT T-3048-LY2* 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later Quanta QCT T5048-LY8* 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later** 1G Portfolio Part number Description Merchant Silicon Cumulus Linux Release Edge-Core AS4600-54T 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later Penguin Computing Artica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later Quanta QCT T1048-LB9* 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later 31