filedownloads.cavium.comfiledownloads.cavium.com/Files_bk/96813/UsersGuide_45000_CNA-I… · ii BC0154501-00 T User’s Guide—Converged Network Adapters and Intelligent Ethernet

BC0154501-00 T

User’s GuideConverged Network Adapters and

Intelligent Ethernet Adapters

FastLinQ 45000 Series

ii BC0154501-00 T

User’s Guide—Converged Network Adapters and Intelligent Ethernet AdaptersFastLinQ 45000 Series

Document Revision HistoryRevision A, April 19, 2016

Revision B, April 26, 2016

Revision C, May 11, 2016

Revision D, August 29, 2016

Revision E, August 30, 2016

Revision F, September 30, 2016

Revision G, March 10, 2017

Revision H, June 26, 2017

Revision J, November 21, 2017

Revision K, February 7, 2018

Revision L, May 22, 2018

Revision M, June 18, 2018

Revision N, August 24, 2018

Revision P, January 7, 2019

Revision R, March 5, 2019

Revision T, May 17, 2019

Changes in Revision T Sections Affected

In Step 3, described how to chose the right version of ADK.

“Injecting (Slipstreaming) Adapter Drivers into Win-dows Image Files” on page 115

Added a second paragraph to the NOTE indicating that dracut only activates interfaces that have an fcoe= entry on the kernel command line.

“FCoE Configuration” on page 210

Changes in Revision R Sections Affected

Removed support for ESXi 5.5.x. All

In Table 3-1, in the qedr driver row, changed the last sentence to “Some RDMA user space applica-tions may require that the libqedr user library is installed on the server.”

“Installing Linux Driver Software” on page 11

In Table 3-4, added the qedi native iSCSI-Offload driver.

“VMware Drivers and Driver Packages” on page 24

iii BC0154501-00 T


In Table 3-4, removed the Driver Version column. Added text pointing to the Release Notes for this information.

“VMware Drivers and Driver Packages” on page 24

In Table 7-1:Removed RHEL 6.9, RHEL 6.10, SLES 11 SP4.Consolidated OS versions with the same inboxOFED and OFED 4.8-2 GA protocols.

“Supported Operating Systems and OFED” on page 127

Updated this section to the latest scripts and example.

“Script Example” on page 167

Changed Step 2 to “Use the following command syntax to override the current NVRAM RDMA-Offload protocol configuration…”

“Configuring iWARP and RoCE” on page 173

Removed the Cables and Optical Modules appen-dix. This information is available on marvell.com

—

Changed the first paragraph. “Concurrent FCoE and iSCSI Is Not Supported on the Same Port in NPAR Mode” on page 282

Changed title to: “…on the Same Physical Func-tion”Changed the first paragraph

“Concurrent RoCE and iWARP Is Not Supported on the Same Physical Function” on page 282

Changed the titleChanged the first paragraph.Changed the first bullet.Added the third bullet (only one boot mode is sup-ported per physical port).

“NIC and SAN Boot Is Supported Only on Select PFs” on page 283

Changed the first paragraph. “25G and 40G 45000 Series Adapters Support iWARP” on page 283

http:www.marvell.com

iv BC0154501-00 T

Table of Contents

PrefaceSupported Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxIntended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiWhat Is in This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiRelated Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiDocumentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiiTechnical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv

Downloading Updates and Documentation . . . . . . . . . . . . . . . . . . . . . xxvKnowledgebase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvi

Legal Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviLaser Safety—FDA Notice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviAgency Certification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxvii

EMI and EMC Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviiKCC: Class A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviiiVCCI: Class A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviii

Product Safety Compliance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviii

1 Product OverviewFunctional Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Adapter Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

QLogic Control Suite CLI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4QLogic QConvergeConsole GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4QLogic QConvergeConsole vCenter Plug-in. . . . . . . . . . . . . . . . . . . . 4QConvergeConsole PowerKit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5FastLinQ ESXCLI VMware Plug-in . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Adapter Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Physical Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Standards Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

v BC0154501-00 T


2 Hardware InstallationSystem Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Safety Precautions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Preinstallation Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Installing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Driver InstallationInstalling Linux Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Installing the Linux Drivers Without RDMA . . . . . . . . . . . . . . . . . . . . . 13Removing the Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14Installing Linux Drivers Using the src RPM Package . . . . . . . . . 15Installing Linux Drivers Using the kmp/kmod RPM Package . . . 16Installing Ubuntu Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . 16Installing Linux Drivers Using the TAR File. . . . . . . . . . . . . . . . . 16

Installing the Linux Drivers with RDMA . . . . . . . . . . . . . . . . . . . . . . . . 17Linux Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Linux Driver Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Importing a Public Key for Secure Boot. . . . . . . . . . . . . . . . . . . . . . . . 19

Installing Windows Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20Installing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Removing the Windows Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Managing Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Setting Power Management Options. . . . . . . . . . . . . . . . . . . . . . . . . . 23

Installing VMware Driver Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23VMware Drivers and Driver Packages. . . . . . . . . . . . . . . . . . . . . . . . . 24Installing VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . 27VMware Driver Parameter Defaults. . . . . . . . . . . . . . . . . . . . . . . . . . . 28Removing the VMware Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29FCoE Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29iSCSI Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4 Firmware Upgrade UtilityImage Verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Upgrading Adapter Firmware on Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Upgrading Adapter Firmware on Windows. . . . . . . . . . . . . . . . . . . . . . . . . . 31Converting a 100G Adapter to Four-port 25G Adapter. . . . . . . . . . . . . . . . . 32

vi BC0154501-00 T


5 Adapter Preboot ConfigurationGetting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Displaying Firmware Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Configuring Device-level Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38Configuring Port-level Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Configuring FCoE Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Configuring iSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Configuring Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Changing the Adapter Port Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Converting a 100G Adapter to Four-port 25G Adapter . . . . . . . . . . . . 53Converting a 25G Adapter to Single-port 100G Adapter . . . . . . . . . . . 54Converting a 40G Adapter to a Four-port 10G Adapter . . . . . . . . . . . . 55Converting a 10G Adapter to a Four-port 40G Adapter . . . . . . . . . . . . 56

6 Boot from SAN ConfigurationiSCSI Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

iSCSI Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 59iSCSI Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

Supported Operating Systems for iSCSI Boot . . . . . . . . . . . . . . 60Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 61Selecting the iSCSI UEFI Boot Protocol . . . . . . . . . . . . . . . . . . . 63Configuring the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Configuring iSCSI Boot Parameters . . . . . . . . . . . . . . . . . . . . . . 65Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 65Configuring iSCSI Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . 67Configuring the DHCP Server to Support iSCSI Boot . . . . . . . . 77

Configuring iSCSI Boot from SAN on Windows . . . . . . . . . . . . . . . . . 81Before You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81Selecting the Preferred iSCSI Boot Mode . . . . . . . . . . . . . . . . . 82Configuring iSCSI General Parameters . . . . . . . . . . . . . . . . . . . 82Configuring the iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . 82Configuring the iSCSI Targets . . . . . . . . . . . . . . . . . . . . . . . . . . 83Detecting the iSCSI LUN and Injecting the QLogic Drivers . . . . 84

Configuring iSCSI Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 86Configuring iSCSI Boot from SAN for RHEL 7.4 . . . . . . . . . . . . 86Configuring iSCSI Boot from SAN for RHEL 7.5 and Later . . . . 90Configuring iSCSI Boot from SAN for SLES 12 SP3 and Later . 91Configuring iSCSI Boot from SAN for Other Linux Distributions. 92

Configuring iSCSI Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 106

vii BC0154501-00 T


FCoE Boot from SAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106FCoE Out-of-Box and Inbox Support. . . . . . . . . . . . . . . . . . . . . . . . . . 107FCoE Preboot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Specifying the BIOS Boot Protocol . . . . . . . . . . . . . . . . . . . . . . . 108Configuring Adapter UEFI Boot Mode . . . . . . . . . . . . . . . . . . . . 108

Configuring FCoE Boot from SAN on Windows . . . . . . . . . . . . . . . . . 113Windows Server 2012 R2 and 2016 FCoE Boot Installation . . . 113Configuring FCoE on Windows . . . . . . . . . . . . . . . . . . . . . . . . . 114FCoE Crash Dump on Windows. . . . . . . . . . . . . . . . . . . . . . . . . 114Injecting (Slipstreaming) Adapter Drivers into Windows

Image Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115Configuring FCoE Boot from SAN on Linux . . . . . . . . . . . . . . . . . . . . 116

Prerequisites for Linux FCoE Boot from SAN. . . . . . . . . . . . . . . 116Configuring Linux FCoE Boot from SAN . . . . . . . . . . . . . . . . . . 117

Configuring FCoE Boot from SAN on VMware . . . . . . . . . . . . . . . . . . 122Injecting (Slipstreaming) ESXi Adapter Drivers into Image Files 122Installing the Customized ESXi ISO . . . . . . . . . . . . . . . . . . . . . . 123

7 RoCE ConfigurationSupported Operating Systems and OFED . . . . . . . . . . . . . . . . . . . . . . . . . . 127Planning for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127Preparing the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128Preparing the Ethernet Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

Configuring the Cisco Nexus 6000 Ethernet Switch . . . . . . . . . . . . . . 129Configuring the Dell Z9100 Ethernet Switch . . . . . . . . . . . . . . . . . . . . 130Configuring the Arista 7060X Ethernet Switch. . . . . . . . . . . . . . . . . . . 133

Configuring RoCE on the Adapter for Windows Server . . . . . . . . . . . . . . . . 134Viewing RDMA Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Configuring RoCE on the Adapter for Linux . . . . . . . . . . . . . . . . . . . . . . . . . 143RoCE Configuration for RHEL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144RoCE Configuration for SLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144RoCE Configuration for Ubuntu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145Verifying the RoCE Configuration on Linux . . . . . . . . . . . . . . . . . . . . . 147vLAN Interfaces and GID Index Values . . . . . . . . . . . . . . . . . . . . . . . . 150RoCE v2 Configuration for Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Identifying the RoCE v2 GID Index or Address . . . . . . . . . . . . . 151Verifying the RoCE v1 or RoCE v2 GID Index and Address

from sys and class Parameters . . . . . . . . . . . . . . . . . . . . . . . . 151Verifying the RoCE v1 or RoCE v2 Function Through

perftest Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

viii BC0154501-00 T


Configuring RoCE on the Adapter for VMware ESX . . . . . . . . . . . . . . . . . . 156Configuring RDMA Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156Configuring MTU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158RoCE Mode and Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158Configuring a Paravirtual RDMA Device (PVRDMA). . . . . . . . . . . . . . 159

Configuring DCQCN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162DCQCN Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163DCQCN Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163DCB-related Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164Global Settings on RDMA Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

Setting vLAN Priority on RDMA Traffic . . . . . . . . . . . . . . . . . . . . 164Setting ECN on RDMA Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . 165Setting DSCP on RDMA Traffic . . . . . . . . . . . . . . . . . . . . . . . . . 165

Configuring DSCP-PFC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165Enabling DCQCN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165Configuring CNP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166MAC Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166Script Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168

8 iWARP ConfigurationPreparing the Adapter for iWARP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169Configuring iWARP on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170Configuring iWARP on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

Installing the Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173Configuring iWARP and RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173Detecting the Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174Supported iWARP Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175Running Perftest for iWARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175Configuring NFS-RDMA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176iWARP RDMA-Core Support on SLES 12 SP3 and RHEL 7.4 . . . . . . 178

9 iSER ConfigurationBefore You Begin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180Configuring iSER for RHEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181Configuring iSER for SLES 12 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184Configuring iSER for Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Configuring LIO as a Target. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185Configuring the Initiator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

ix BC0154501-00 T


Optimizing Linux Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192Configuring CPUs to Maximum Performance Mode . . . . . . . . . . . . . . 192Configuring Kernel sysctl Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . 192Configuring IRQ Affinity Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Configuring Block Device Staging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

Configuring iSER on ESXi 6.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Before You Begin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Configuring iSER for ESXi 6.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

10 iSCSI ConfigurationiSCSI Boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197iSCSI Offload in Windows Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

Installing Cavium QLogic Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198Installing the Microsoft iSCSI Initiator . . . . . . . . . . . . . . . . . . . . . . . . . 198Configuring Microsoft Initiator to Use Cavium’s iSCSI Offload . . . . . . 198iSCSI Offload FAQs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204Windows Server 2012 R2, 2016, and 2019 iSCSI Boot Installation . . 205iSCSI Crash Dump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

iSCSI Offload in Linux Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206Differences from bnx2i. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207Configuring qedi.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207Verifying iSCSI Interfaces in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

11 FCoE ConfigurationConfiguring Linux FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

Differences Between qedf and bnx2fc. . . . . . . . . . . . . . . . . . . . . . . . . 211Configuring qedf.ko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211Verifying FCoE Devices in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

12 SR-IOV ConfigurationConfiguring SR-IOV on Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214Configuring SR-IOV on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221Configuring SR-IOV on VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

13 NVMe-oF Configuration with RDMAInstalling Device Drivers on Both Servers . . . . . . . . . . . . . . . . . . . . . . . . . . 232Configuring the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233Configuring the Initiator Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235Preconditioning the Target Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236Testing the NVMe-oF Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

x BC0154501-00 T


Optimizing Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238.IRQ Affinity (multi_rss-affin.sh) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239CPU Frequency (cpufreq.sh). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

14 Windows Server 2016Configuring RoCE Interfaces with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . 242

Creating a Hyper-V Virtual Switch with an RDMA NIC . . . . . . . . . . . . 243Adding a vLAN ID to Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . 244Verifying If RoCE is Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245Adding Host Virtual NICs (Virtual Ports) . . . . . . . . . . . . . . . . . . . . . . . 246Mapping the SMB Drive and Running RoCE Traffic . . . . . . . . . . . . . . 246

RoCE over Switch Embedded Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248Creating a Hyper-V Virtual Switch with SET and RDMA

Virtual NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248Enabling RDMA on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248Assigning a vLAN ID on SET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249Running RDMA Traffic on SET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

Configuring QoS for RoCE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249Configuring QoS by Disabling DCBX on the Adapter . . . . . . . . . . . . . 250Configuring QoS by Enabling DCBX on the Adapter. . . . . . . . . . . . . . 254

Configuring VMMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258Enabling VMMQ on the Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259Creating a Virtual Machine Switch with or Without SR-IOV . . . . . . . . 259Enabling VMMQ on the Virtual Machine Switch . . . . . . . . . . . . . . . . . 260Getting the Virtual Machine Switch Capability. . . . . . . . . . . . . . . . . . . 261Creating a VM and Enabling VMMQ on VMNetworkAdapters

in the VM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261Default and Maximum VMMQ Virtual NIC . . . . . . . . . . . . . . . . . . . . . . 263Enabling and Disabling VMMQ on a Management NIC . . . . . . . . . . . 263Monitoring Traffic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

Configuring VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263Enabling VXLAN Offload on the Adapter. . . . . . . . . . . . . . . . . . . . . . . 264Deploying a Software Defined Network. . . . . . . . . . . . . . . . . . . . . . . . 264

Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265Configuring the Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265Deploying a Hyper-Converged System . . . . . . . . . . . . . . . . . . . . . . . . 266

Deploying the Operating System . . . . . . . . . . . . . . . . . . . . . . . . 266Configuring the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266Configuring Storage Spaces Direct . . . . . . . . . . . . . . . . . . . . . . 268

xi BC0154501-00 T


15 Traffic Control OffloadRequirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272Current Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272Ingress Packet Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

Ingress Packet Redirection for SR-IOV VFs . . . . . . . . . . . . . . . . . . . . 273Ingress Packet Redirection with MAC-vLAN Offloaded

Devices and SR-IOV VFs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273TC Drop Action Support (Ingress Drop) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274TC MQPRIO Offload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274

16 TroubleshootingTroubleshooting Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275Verifying that Current Drivers Are Loaded . . . . . . . . . . . . . . . . . . . . . . . . . . 276

Verifying Drivers in Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276Verifying Drivers in Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276Verifying Drivers in VMware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

Testing Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277Testing Network Connectivity for Windows . . . . . . . . . . . . . . . . . . . . . 277Testing Network Connectivity for Linux . . . . . . . . . . . . . . . . . . . . . . . . 278

Microsoft Virtualization with Hyper-V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278Linux-specific Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278Miscellaneous Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278Troubleshooting Windows FCoE and iSCSI Boot from SAN . . . . . . . . . . . . 279Collecting Debug Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280

A Adapter LEDS

B Feature Constraints

Glossary

Index

xii BC0154501-00 T


List of FiguresFigure Page3-1 Setting Advanced Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223-2 Power Management Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235-1 System Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345-2 System Setup: Device Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345-3 Main Configuration Page, Setting Default Partitioning Mode . . . . . . . . . . . . . . . . . . 355-4 Main Configuration Page, Setting NPAR Partitioning Mode. . . . . . . . . . . . . . . . . . . 365-5 Firmware Information Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385-6 Device Level Configuration Page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395-7 Port Level Configuration Page: Setting Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . 425-8 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445-9 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445-10 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465-11 iSCSI Initiator Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465-12 iSCSI First Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475-13 iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475-14 Partitions Configuration Page (No FCoE Offload or iSCSI Offload). . . . . . . . . . . . . 485-15 Partitions Configuration Page (with FCoE Offload and iSCSI Offload) . . . . . . . . . . 485-16 Global Bandwidth Allocation Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495-17 Partition 1 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505-18 Partition 2 Configuration: FCoE Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515-19 Partition 3 Configuration: iSCSI Offload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515-20 Partition 4 Configuration: Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525-21 Changing the Port Mode to 4x25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535-22 Changing the Port Mode to 1x100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545-23 Changing the Port Mode to 4x10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555-24 Changing the Port Mode to 1x40 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 566-1 Systems Utilities at Boot Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616-2 Configuration Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626-3 Selecting Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626-4 Port Level Configuration, Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636-5 Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646-6 Systems Utilities at Boot Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 656-7 Selecting Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 666-8 Port Level Configuration, Boot Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676-9 Selecting iSCSI Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686-10 Selecting General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696-11 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696-12 Selecting iSCSI Initiator Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 706-13 iSCSI Initiator Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 726-14 iSCSI First Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 726-15 iSCSI First Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736-16 iSCSI Second Target Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736-17 Saving iSCSI Changes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

xiii BC0154501-00 T


6-18 iSCSI General Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766-19 iSCSI Initiator Configuration, VLAN ID. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816-20 Detecting the iSCSI LUN Using UEFI Shell (Version 2). . . . . . . . . . . . . . . . . . . . . . 846-21 Windows Setup: Selecting Installation Destination . . . . . . . . . . . . . . . . . . . . . . . . . 856-22 Windows Setup: Selecting Driver to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 856-23 Prompt for Out-of-Box Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 876-24 Red Hat Enterprise Linux 7.4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 886-25 System Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1086-26 System Configuration, Port Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1096-27 Port Level Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1096-28 Boot Mode in Port Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1106-29 FCoE Offload Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1106-30 Selecting FCoE Boot Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1116-31 Selecting General Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1116-32 FCoE General Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1126-33 FCoE Target Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1136-34 Selecting the Driver Disk Image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1186-35 Selecting the Drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1196-36 Prompt for Out-of-Box Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1206-37 Red Hat Enterprise Linux 7.4 Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1216-38 ESXi-Customizer Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1236-39 Select a VMware Disk to Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1246-40 VMware Generic USB Boot Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1257-1 Configuring RoCE Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1367-2 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1387-3 Performance Monitor: Cavium FastLinQ Counters . . . . . . . . . . . . . . . . . . . . . . . . . 1407-4 Switch Settings, Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1547-5 Switch Settings, Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1557-6 Configuring RDMA_CM Applications: Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1557-7 Configuring RDMA_CM Applications: Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1567-8 Configuring a New Distributed Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1607-9 Assigning a vmknic for PVRDMA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1617-10 Setting the Firewall Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1618-1 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 1708-2 Windows PowerShell Command: Get-NetOffloadGlobalSetting . . . . . . . . . . . . . . . 1718-3 Perfmon: Add Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1718-4 Perfmon: Verifying iWARP Traffic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1729-1 RDMA Ping Successful . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1829-2 iSER Portal Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1829-3 Iface Transport Confirmed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1839-4 Checking for New iSCSI Device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18410-1 iSCSI Initiator Properties, Configuration Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19910-2 iSCSI Initiator Node Name Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19910-3 iSCSI Initiator—Discover Target Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20010-4 Target Portal IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

xiv BC0154501-00 T


10-5 Selecting the Initiator IP Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20210-6 Connecting to the iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20310-7 Connect To Target Dialog Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20412-1 Device Level Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21512-2 Adapter Properties, Advanced: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . 21612-3 Virtual Switch Manager: Enabling SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21712-4 Settings for VM: Enabling SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21912-5 Device Manager: VM with QLogic Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22012-6 Windows PowerShell Command: Get-NetadapterSriovVf . . . . . . . . . . . . . . . . . . . . 22012-7 Integrated Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22112-8 Editing the grub.conf File for SR-IOV. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22212-9 Command Output for sriov_numvfs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22312-10 Command Output for ip link show Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22312-11 RHEL68 Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22412-12 Add New Virtual Hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22512-13 VMware Host Edit Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22813-1 NVMe-oF Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23113-2 Subsystem NQN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23513-3 Confirm NVMe-oF Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23613-4 FIO Utility Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23714-1 Enabling RDMA in Host Virtual NIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24314-2 Hyper-V Virtual Ethernet Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24414-3 Windows PowerShell Command: Get-VMNetworkAdapter . . . . . . . . . . . . . . . . . . . 24514-4 Windows PowerShell Command: Get-NetAdapterRdma. . . . . . . . . . . . . . . . . . . . . 24514-5 Add Counters Dialog Box. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24714-6 Performance Monitor Shows RoCE Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24714-7 Windows PowerShell Command: New-VMSwitch . . . . . . . . . . . . . . . . . . . . . . . . . . 24814-8 Windows PowerShell Command: Get-NetAdapter. . . . . . . . . . . . . . . . . . . . . . . . . . 24914-9 Advanced Properties: Enable QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25114-10 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25214-11 Advanced Properties: Enabling QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25614-12 Advanced Properties: Setting VLAN ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25714-13 Advanced Properties: Enabling Virtual Switch RSS. . . . . . . . . . . . . . . . . . . . . . . . . 25914-14 Virtual Switch Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26014-15 Windows PowerShell Command: Get-VMSwitch. . . . . . . . . . . . . . . . . . . . . . . . . . . 26114-16 Advanced Properties: Enabling VXLAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26414-17 Example Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26516-1 Windows Setup Error Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

xv BC0154501-00 T


List of TablesTable Page2-1 Host Hardware Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72-2 Minimum Host Operating System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 83-1 Cavium QLogic 45000 Series Adapters Linux Drivers . . . . . . . . . . . . . . . . . . . . . . . 123-2 qede Driver Optional Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183-3 Linux Driver Operation Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183-4 VMware Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243-5 ESXi Driver Packages by Release. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243-6 VMware NIC Driver Optional Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273-7 VMware Driver Parameter Defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285-1 Adapter Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376-1 iSCSI Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 596-2 iSCSI General Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 706-3 DHCP Option 17 Parameter Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776-4 DHCP Option 43 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786-5 DHCP Option 17 Sub-option Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 806-6 FCoE Out-of-Box and Inbox Boot from SAN Support . . . . . . . . . . . . . . . . . . . . . . . 1077-1 OS Support for RoCE v1, RoCE v2, iWARP, iSER, and OFED . . . . . . . . . . . . . . . . 1277-2 Advanced Properties for RoCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1347-3 Cavium FastLinQ RDMA Error Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1407-4 DCQCN Algorithm Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16613-1 Target Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23316-1 Collecting Debug Data Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280A-1 Adapter Port Link and Activity LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

xvi BC0154501-00 T

Preface

This preface lists the supported products, specifies the intended audience, explains the typographic conventions used in this guide, lists related documents, provides technical support information, and describes legal notices.

Supported ProductsThis user’s guide describes the following Cavium FastLinQ products:

25Gb Intelligent Ethernet Adapters:

QL45211HLCU-BK/SP/CK QL45212HLCU-BK/SP/CK


QL45411HLCU-BK/SP/CK QL45412HLCU-BK/SP/CK

40Gb Converged Network Adapter1:

QL45462HLCU-BK/SP/CK


QL45611HLCU-BK/SP/CK QL45631HOCU-BK/SP/CK

NOTEQConvergeConsole® (QCC) GUI is the only GUI management tool across all Cavium® QLogic® adapters. QLogic Control Suite™ (QCS) GUI is no longer supported for the FastLinQ® 45000 Series Adapters and adapters based on 57xx/57xxx controllers, and has been replaced by the QCC GUI management tool. The QCC GUI provides single-pane-of-glass GUI management for all Cavium adapters.In Windows® environments, when you run QCS CLI and Management Agents Installer, it will uninstall the QCS GUI (if installed on the system) and any related components from your system. To obtain the new GUI, download QCC GUI for your adapter from the Marvell Web site (see “Downloading Updates and Documentation” on page xxi).

1 Windows and Linux only. VMware support for FCoE and iSCSI offload is coming soon.

PrefaceIntended Audience

xvii BC0154501-00 T

Intended AudienceThis guide is intended for system administrators and other technical staff members responsible for configuring and managing adapters installed on servers in Windows, Linux®, or VMware® environments.

What Is in This GuideFollowing this preface, the remainder of this guide is organized into the following chapters and appendices:

Chapter 1 Product Overview provides a product functional description, a list of features, adapter management tool descriptions, and the adapter specifications.

Chapter 2 Hardware Installation describes how to install the adapter, including the list of system requirements and a preinstallation checklist.

Chapter 3 Driver Installation describes the installation of the adapter drivers on Windows, Linux, and VMware.

Chapter 4 Firmware Upgrade Utility describes the use of the utility to upgrade adapter firmware and boot code.

Chapter 5 Adapter Preboot Configuration describes the preboot adapter configuration tasks using the Human Infrastructure Interface (HII) application.

Chapter 6 Boot from SAN Configuration covers boot from SAN configuration for both iSCSI and FCoE.

Chapter 7 RoCE Configuration describes how to configure the adapter, the Ethernet switch, and the host to use RDMA over converged Ethernet (RoCE).

Chapter 8 iWARP Configuration provides procedures for configuring Internet wide area RDMA protocol (iWARP) on Windows, Linux, and VMware ESXi 6.7 systems.

Chapter 9 iSER Configuration describes how to configure iSCSI Extensions for RDMA (iSER) for Linux RHEL, SLES, Ubuntu, and ESXi 6.7.

Chapter 10 iSCSI Configuration describes iSCSI boot and iSCSI offload for Windows and Linux.

Chapter 11 FCoE Configuration covers configuring Linux FCoE offload.

Chapter 12 SR-IOV Configuration provides procedures for configuring single root input/output virtualization (SR-IOV) on Windows, Linux, and VMware systems.

PrefaceRelated Materials

xviii BC0154501-00 T

Chapter 13 NVMe-oF Configuration with RDMA demonstrates how to configure NVMe-oF on a simple network for 45000 Series Adapters.

Chapter 14 Windows Server 2016 describes the Windows Server 2016 features.

Chapter 15 Traffic Control Offload describes the traffic control (TC) offload feature.

Chapter 16 Troubleshooting describes a variety of troubleshooting methods and resources.

Appendix A Adapter LEDS lists the adapter LEDs and their significance.

Appendix B Feature Constraints provides information about feature constraints implemented in the current release.

At the end of this guide are a glossary of terms and an index to help you quickly locate the information you need.

Related MaterialsFor additional information, refer to the following documents that are available on the Marvell Web Site:

Installation Guide—QConvergeConsole GUI (part number SN0051105-00) contains detailed information on how to install and use the QConvergeConsole GUI management tool.

User's Guide—QLogic Control Suite CLI (part number BC0054511-00) contains detailed information on how to install, start, and use QLogic Control Suite CLI.

User's Guide—PowerShell (part number BC0054518-00) contains detailed information on how to install QConvergeConsole FastLinQ PowerKit to manage the FastLinQ adapters on the system.

User’s Guide—QConvergeConsole Plug-ins for vSphere (part number SN0054677-00) provides details for using the two QCC plug-ins to extend the capabilities of VMware vCenter Server and the vSphere Web Client.

User’s Guide—FastLinQ ESXCLI VMware Plug-in (part number BC0151101-00) describes the plug-in that extends the capabilities of the ESX® CLI to manage Cavium QLogic 3400, 8400, 41000, and 45000 Series Adapters installed in VMware ESX/ESXi hosts.

For information about downloading documentation from the Marvell Web site, see “Downloading Updates and Documentation” on page xxi.

In addition the QConvergeConsole GUI help system provides topics available while using the QCC GUI.

PrefaceDocumentation Conventions

xix BC0154501-00 T

Documentation ConventionsThis guide uses the following documentation conventions:

provides additional information.

without an alert symbol indicates the presence of a hazard that could cause damage to equipment or loss of data.

with an alert symbol indicates the presence of a hazard that could cause minor or moderate injury.

indicates the presence of a hazard that could cause serious injury or death.

Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this guide, and links to Web sites are shown in underlined blue. For example:

Table 9-2 lists problems related to the user interface and remote agent.

See “Installation Checklist” on page 6.

For more information, visit www.marvell.com.

Text in bold font indicates user interface elements such as a menu items, buttons, check boxes, or column headings. For example:

Click the Start button, point to Programs, point to Accessories, and then click Command Prompt.

Under Notification Options, select the Warning Alarms check box.

Text in Courier font indicates a file name, directory path, or command line text. For example:

To return to the root directory from anywhere in the file structure:Type cd/ root and press ENTER.

Issue the following command: sh ./install.bin. Key names and key strokes are indicated with UPPERCASE:

Press CTRL+P.

Press the UP ARROW key.

NOTE

CAUTION

CAUTION!

! WARNING

PrefaceDocumentation Conventions

xx BC0154501-00 T

Text in italics indicates terms, emphasis, variables, or document titles. For example:

What are shortcut keys?

To enter the date type mm/dd/yyyy (where mm is the month, dd is the day, and yyyy is the year).

Topic titles between quotation marks identify related topics either within this manual or in the online help, which is also referred to as the help system throughout this document.

Command line interface (CLI) command syntax conventions include the following:

Plain text indicates items that you must type as shown. For example:

qaucli -pr nic -ei < > (angle brackets) indicate a variable whose value you must

specify. For example:

[ ] (square brackets) indicate an optional parameter. For example: [] means specify a file name, or omit it to select

the default file name.

| (vertical bar) indicates mutually exclusive options; select one option only. For example: on|off 1|2|3|4

... (ellipsis) indicates that the preceding item may be repeated. For example:

x... means one or more instances of x. [x...] means zero or more instances of x.

⋮ (vertical ellipsis) within command example output indicate where portions of repetitious output data have been intentionally omitted.

NOTEFor CLI commands only, variable names are always indicated using angle brackets instead of italics.

PrefaceTechnical Support

xxi BC0154501-00 T

( ) (parentheses) and { } (braces) are used to avoid logical ambiguity. For example:

a|b c is ambiguous{(a|b) c} means a or b, followed by c{a|(b c)} means either a, or b c

Technical SupportCustomers should contact their authorized maintenance provider for technical support of their Cavium QLogic products. Technical support for QLogic-direct products under warranty is available with a Cavium support account.

To set up a support account (if needed) and submit a case:1. Go to www.marvell.com.

2. Point to Support. Under Marvell Extranets, click LOGIN NOW.3. In the Marvell QLogic Extranet box, click SELECT.4. Do one of the following:

If you do not have an account, click Sign Up. You are directed to a page to create a support account.

If you have an account, enter your User Name and Password, and then click Sign In.

5. From the links at the top of the page, click Submit a Case.

Downloading Updates and DocumentationThe Marvell Web site provides periodic updates to product firmware, software, and documentation.

To download QLogic firmware, software, and documentation:1. Go to www.marvell.com.

2. Point to Support, and then under Driver Downloads, click Marvell QLogic/FastLinQ Drivers.

3. On the Cavium Downloads and Documentation page, click Adapters.4. Click the corresponding button to search by Model or by Operating

System.5. To define a search, click an item in each selection column, and then click

Go.6. Locate the firmware, software, or document you need, and then click the

item’s name or icon to download or open the item.

www.marvell.comwww.marvell.com

PrefaceLegal Notices

xxii BC0154501-00 T

KnowledgebaseThe Cavium QLogic knowledgebase is an extensive collection of product information that you can search for specific solutions. Cavium is constantly adding to the collection of information in the database to provide answers to your most urgent questions.

To access the knowledgebase:1. Go to www.marvell.com.

2. Point to Support, and then under Support by Product, click GET SUPPORT.

3. On the Support page, select the product line for your adapter:

QLogic Fibre Channel Adapters and Controllers FastLinQ® | LiquidIO® Ethernet Adapters and Controllers

4. Under Related Links, click Knowledgebase.

Legal NoticesLegal notices covered in this section include laser safety (FDA notice), agency certification, and product safety compliance.

Laser Safety—FDA NoticeThis product complies with DHHS Rules 21CFR Chapter I, Subchapter J. This product has been designed and manufactured according to IEC60825-1 on the safety label of laser product.

Class 1 Laser Product

Caution—Class 1 laser radiation when openDo not view directly with optical instruments

Appareil laser de classe 1

Attention—Radiation laser de classe 1Ne pas regarder directement avec des instruments optiques

Produkt der Laser Klasse 1

Vorsicht—Laserstrahlung der Klasse 1 bei geöffneter Abdeckung Direktes Ansehen mit optischen Instrumenten vermeiden

Luokan 1 Laserlaite Varoitus—Luokan 1 lasersäteilyä, kun laite on aukiÄlä katso suoraan laitteeseen käyttämällä optisia instrumenttej

CLASS I LASER

www.marvell.com


xxiii BC0154501-00 T

Agency CertificationThe following sections summarize the EMC and EMI test specifications performed on the 45000 Series Adapters.

EMI and EMC RequirementsFCC Part 15 compliance: Class AFCC compliance information statement: This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation.

ICES-003 Compliance: Class AThis Class A digital apparatus complies with Canadian ICES-003. Cet appareil numériqué de la classe A est conformé à la norme NMB-003 du Canada.

CE Mark 2014/30/EU, 2014/35/EU EMC Directive Compliance:EN55032:2012/ CISPR 32:2015 Class A

EN55024: 2010EN61000-3-2: Harmonic Current EmissionEN61000-3-3: Voltage Fluctuation and Flicker

Immunity StandardsEN61000-4-2: ESDEN61000-4-3: RF Electro Magnetic FieldEN61000-4-4: Fast Transient/BurstEN61000-4-5: Fast Surge Common/ DifferentialEN61000-4-6: RF Conducted SusceptibilityEN61000-4-8: Power Frequency Magnetic FieldEN61000-4-11: Voltage Dips and Interrupt

VCCI: 2015-04; Class A AS/NZS; CISPR 32: 2015 Class ACNS 13438: 2006 Class A


xxiv BC0154501-00 T

KCC: Class AKorea RRA Class A Certified

Korean Language Format—Class A

VCCI: Class AThis is a Class A product based on the standard of the Voluntary Control Council for Interference (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.

Product Safety ComplianceUL, cUL product safety:UL 60950-1 (2nd Edition) A1 + A2 2014-10-14CSA C22.2 No.60950-1-07 (2nd Edition) A1 +A2 2014-10

Use only with listed ITE or equivalent.

Complies with 21 CFR 1040.10 and 1040.11, 2014/30/EU, 2014/35/EU.

Product Name/Model: Converged Network Adapters and Intelligent Ethernet AdaptersCertification holder: QLogic CorporationManufactured date: Refer to date code listed on productManufacturer/Country of origin: QLogic Corporation/USA

A class equipment(Business purpose info/telecommunications equipment)

As this equipment has undergone EMC registration for busi-ness purpose, the seller and/or the buyer is asked to beware of this point and in case a wrongful sale or purchase has been made, it is asked that a change to household use be made.


xxv BC0154501-00 T

2006/95/EC low voltage directive:TUV EN60950-1:2006+A11+A1+A12+A2 2nd EditionTUV IEC 60950-1: 2005 2nd Edition Am1: 2009 + Am2: 2013 CB

CB Certified to IEC 60950-1 2nd Edition

1 BC0154501-00 T

1 Product OverviewThis chapter provides the following information for the 45000 Series Adapters:

Functional Description

Features

“Adapter Management” on page 3

“Adapter Specifications” on page 5

Functional DescriptionThe Cavium FastLinQ 45000 Series Adapters include 10, 25, 40, and 100Gb Converged Network Adapters and Intelligent Ethernet Adapters that are designed to perform accelerated data networking for server systems. The 45000 Series Adapters include a 10/25/40/50/100Gb Ethernet MAC with full-duplex capability.

FeaturesThe 45000 Series Adapters provide the following features. Some features may not be available on all adapters:

NIC partitioning (NPAR)

Universal RDMA: Remote Direct Memory Access over converged Ethernet, versions 1 and 2 (RoCE v1/v2) and Internet wide area Protocol (iWARP):

iSCSI Extensions for RDMA (iSER)

NVMe over Fabrics (NVMe-oF) (For details, download the Deployment Guide: Non-volatile Memory Express over Fabrics with Universal Remote Direct Memory Access, part number BC0054519-00)

NFS over RDMA (NFSoRDMA)

25G and 40G 45000 Series Adapters support iWARP

iSCSI and FCoE hardware offloads (QL45462 adapter only)

Data center bridging (DCB):

Enhanced transmission selection (ETS; IEEE 802.1Qaz)

Priority-based flow control (PFC; IEEE 802.1Qbb)

1–Product OverviewFeatures

2 BC0154501-00 T

Data center bridging eXchange protocol (DCBX; CEE version 1.01, IEEE)

Single-chip solution:

25/40/100Gb MAC

SerDes interface for direct attach copper (DAC) transceiver connection

PCI Express® (PCIe®) 3.0 x8

Zero copy capable hardware

Performance features:

TCP, IP, UDP checksum offloads

TCP segmentation offload (TSO)

Large segment offload (LSO)

Generic segment offload (GSO)

HW based Generic Receive Offload (HW-GRO)

Large receive offload (LRO)

Receive segment coalescing (RSC)

Microsoft® dynamic virtual machine queue (VMQ), and Linux Multiqueue

Adaptive interrupts:

Transmit/receive side scaling (TSS/RSS)

Accelerated Receive Flow Steering (ARFS)

Stateless offloads for Network Virtualization using Generic Routing Encapsulation (NVGRE), Generic Network Virtualization Encapsulation (GENEVE), and virtual LAN (VXLAN) L2/L3 GRE tunneled traffic1

Manageability:

System management bus (SMB) controller

Advanced Configuration and Power Interface (ACPI) 1.1a compliant (multiple power modes)

Network controller-sideband interface (NC-SI) support

The management applications listed in Adapter Management

1 This feature requires OS or Hypervisor support to use the offloads.

1–Product OverviewAdapter Management

3 BC0154501-00 T

Advanced network features:

Jumbo frames (up to 9,600 bytes). The OS and the link partner must support jumbo frames.

Virtual LANs (vLANs)

Flow control (IEEE Std 802.3x)

Virtualization (SR-IOV Hypervisor)

VMware NetQueue

Windows Hyper-V® Virtual Machine Queue (VMQ)

Linux Multiqueue

Windows, Linux, and VMware switch-independent NPAR

Windows Hyper-V2, Linux Kernel-based Virtual Machine (KVM), and VMware ESXi™ SR-IOV

Logical link control (IEEE Std 802.2)

High-speed on-chip reduced instruction set computer (RISC) processor

Integrated 96KB frame buffer memory (not applicable to all models)

1,024 classification filters (not applicable to all models)

Support for multicast addresses through 128-bit hashing hardware function

Serial flash NVRAM memory

PCI Power Management Interface (v1.1)

64-bit base address register (BAR) support

EM64T processor support

iSCSI and FCoE boot support3

Adapter ManagementThe following applications are available to manage 45000 Series Adapters:

QLogic Control Suite CLI

QLogic QConvergeConsole GUI

QLogic QConvergeConsole vCenter Plug-in

QConvergeConsole PowerKit

FastLinQ ESXCLI VMware Plug-in2 The QL456x1 (1×100G) adapter does not currently support SR-IOV on Windows Hyper-V.3 Hardware support limit of SR-IOV VFs varies. The limit may be lower in some OS environments;refer to the appropriate section for your OS.

1–Product OverviewAdapter Management

4 BC0154501-00 T

For document part numbers and download instructions, see “Related Materials” on page xviii.

QLogic Control Suite CLIQLogic Control Suite (QCS) CLI is a console application that you can run from a Windows command prompt or a Linux terminal console. Use QCS CLI to manage Cavium QLogic FastLinQ 3400/8400/41000/45000 Series Adapters and any QLogic adapter based on 57xx/57xxx controllers on both local and remote computer systems. For information about installing and using QCS CLI, see the User’s Guide—QLogic Control Suite CLI.

QLogic QConvergeConsole GUIQConvergeConsole (QCC) GUI is a Web-based management tool for configuring and managing Cavium QLogic Fibre Channel Adapters, Converged Network Adapters, and Intelligent Ethernet Adapters. You can use QCC GUI on Windows and Linux platforms to manage Cavium QLogic adapters on both local and remote computer systems. QCC GUI is dependent upon additional software (a management agent) for the adapter. The management agent is installed separately from QCC GUI. QCC GUI cannot communicate with the hardware until the management agent has been installed.

For information about installing QCC GUI, see the Installation Guide—QConvergeConsole GUI. For information about using the QCC GUI, see the online help.

QLogic QConvergeConsole vCenter Plug-inQConvergeConsole vCenter® Plug-in is a Web-based management tool that is integrated into the VMware vCenter Server for configuring and managing Cavium QLogic Fibre Channel Adapters, Converged Network Adapters, and Intelligent Ethernet Adapters in a virtual environment. You can use the vCenter Plug-in installed in the VMware vCenter Server to manage Cavium QLogic adapters. For information about installing and using the vCenter Plug-in, see the User’s Guide—QConvergeConsole Plug-ins for vSphere.

NOTEQCS CLI, QCC GUI, and QCC PowerKit also require installation of their specific RPC agent.

1–Product OverviewAdapter Specifications

5 BC0154501-00 T

QConvergeConsole PowerKitQConvergeConsole PowerKit lets you manage Cavium QLogic FastLinQ 3400/8400/41000/45000 Series Adapters on the system using Cavium cmdlets in the Windows PowerShell® application. Windows PowerShell is a Microsoft-developed scriptable language for performing task automation and configuration management both locally and remotely. Windows PowerShell is based on the .NET framework and includes a command-line shell and a GUI-integrated scripting environment (ISE) that allows you to create scripts without having to type all the commands in the command line. This feature allows you to streamline and automate repetitive and monotonous Windows and Linux server jobs through scripts by linking multiple instructions together. In addition to being a powerful scripting tool, QCC PowerKit comes with a selection of preconfigured cmdlets (scripts that perform a single function) to monitor and manage your Cavium QLogic FastLinQ Adapters. For information about installing and using QConvergeConsole PowerKit, refer to the User's Guide—PowerShell.

FastLinQ ESXCLI VMware Plug-inFastLinQ ESXCLI VMware Plug-in extends the capabilities of the ESX® CLI to manage Cavium QLogic FastLinQ 3400/8400/41000/45000 Series Adapters installed in VMware ESX/ESXi hosts. For information about installing and using the ESXCLI plug-in, see the User’s Guide—FastLinQ ESXCLI VMware Plug-in.

Adapter SpecificationsThe 45000 Series Adapter specifications include the adapter’s physical characteristics and standards-compliance references.

Physical CharacteristicsThe 45000 Series Adapters are standard PCIe cards and ship with either a full-height or a low-profile bracket for use in a standard PCIe slot.

Standards SpecificationsSupported standards specifications include:

PCI Express Base Specification, rev. 3.1

PCI Express Card Electromechanical Specification, rev. 3.0

PCI Bus Power Management Interface Specification, rev. 1.2

IEEE Specifications:

802.1ad (QinQ) 802.1AX (Link Aggregation) 802.1p (Priority Encoding) 802.1q (VLAN)

1–Product OverviewAdapter Specifications

6 BC0154501-00 T

802.3-2015 IEEE Standard for Ethernet (flow control) 802.3-2015 Clause 78 Energy Efficient Ethernet (EEE) 1588-2002 PTPv1 (Precision Time Protocol) 1588-2008 PTPv2

IPv4 (RFQ 791)

IPv6 (RFC 2460)

7 BC0154501-00 T

2 Hardware InstallationThis chapter provides the following hardware installation information:

System Requirements

“Safety Precautions” on page 8

“Preinstallation Checklist” on page 9

“Installing the Adapter” on page 9

System RequirementsBefore you install a Cavium 45000 Series Adapter, verify that your system meets the hardware and operating system requirements shown in Table 2-1 and Table 2-2. For a complete list of supported operating systems, visit the Marvell Web site.

Table 2-1. Host Hardware Requirements

Hardware Requirement

Architecture IA-32 or EMT64 that meets operating system requirements

PCIe PCIe Gen 2 x8 (2x25G NIC)/x16 (2x40G/1x100G NIC)PCIe Gen 3 x8 (2x25G NIC)/x16 (2x40G/1x100G NIC)Full dual-port 40Gb and single port 100Gb bandwidth is sup-ported on PCIe Gen3 x16 or faster slots. Full dual-port 25Gb bandwidth is supported on PCIe Gen 3 x8 or faster slots.

Memory 8GB RAM (minimum)

Cables and Optical Modules

The 45000 Series Adapters have been tested for interoperability with a variety of 10G, 25G, 40G, and 100G cables and optical modules. See “Tested Cables and Optical Modules” on page 290.

2–Hardware InstallationSafety Precautions

8 BC0154501-00 T

Safety Precautions

Table 2-2. Minimum Host Operating System Requirements

Operating System Requirement

Windows Server 2012, 2012 R2, 2016

Linux

RHEL 6.6, 7.0, and higherSLES 11 SP4, SLES 12, and higher

CentOS® 7.0 and higher

Ubuntu® 14.04 LTS, 16.04 LTS

VMwareESXi 6.0 U2 and later for 25G adaptersESXi 6.0.x and later for 40G adaptersESXi 6.5 and later for 100G adapters

XenServer 7.0, 7.1

NOTETable 2-2 denotes minimum host OS requirements. For a complete list of supported operating systems, visit the Marvell Web site.

! WARNINGThe adapter is being installed in a system that operates with voltages that can be lethal. Before you open the case of your system, observe the following precautions to protect yourself and to prevent damage to the system components. Remove any metallic objects or jewelry from your hands and wrists. Make sure to use only insulated or nonconducting tools. Verify that the system is powered OFF and is unplugged before you

touch internal components. Install or remove adapters in a static-free environment. The use of a

properly grounded wrist strap or other personal antistatic devices and an antistatic mat is strongly recommended.

2–Hardware InstallationPreinstallation Checklist

9 BC0154501-00 T

Preinstallation ChecklistBefore installing the adapter, complete the following:

1. Verify that the system meets the hardware and software requirements listed under “System Requirements” on page 7.

2. Verify that the system is using the latest BIOS.

3. If the system is active, shut it down.

4. When system shutdown is complete, turn off the power and unplug the power cord.

5. Remove the adapter from its shipping package and place it on an anti-static surface.

6. Check the adapter for visible signs of damage, particularly on the edge connector. Never attempt to install a damaged adapter.

Installing the AdapterThe following instructions apply to installing the Cavium 45000 Series Adapters in most systems. For details about performing these tasks, refer to the manuals that were supplied with the system.

To install the adapter:1. Review “Safety Precautions” on page 8 and “Preinstallation Checklist” on

page 9. Before you install the adapter, ensure that the system power is OFF, the power cord is unplugged from the power outlet, and that you are following proper electrical grounding procedures.

2. Open the system case, and select the slot that matches the adapter size, which can be PCIe Gen 2 x8 or PCIe Gen 3 x8. A lesser-width adapter can be seated into a greater-width slot (x8 in an x16), but a greater-width adapter cannot be seated into a lesser-width slot (x8 in an x4). If you do not know how to identify a PCIe slot, refer to your system documentation.

3. Remove the blank cover-plate from the slot that you selected.

4. Align the adapter connector edge with the PCIe connector slot in the system.

NOTEIf you acquired the adapter software from the Marvell Web site, verify the path to the adapter driver files.

2–Hardware InstallationInstalling the Adapter

10 BC0154501-00 T

5. Applying even pressure at both corners of the card, push the adapter card into the slot until it is firmly seated. When the adapter is properly seated, the adapter port connectors are aligned with the slot opening, and the adapter faceplate is flush against the system chassis.

6. Secure the adapter with the adapter clip or screw as recommended by the system vendor.

7. Close the system case and disconnect any personal anti-static devices.

8. Connect to the Ethernet network using your preferred and supported cabling option (see “Tested Cables and Optical Modules” on page 290).

CAUTIONDo not use excessive force when seating the card, because this may damage the system or the adapter. If you have difficulty seating the adapter, remove it, realign it, and try again.

11 BC0154501-00 T

3 Driver Installation

This chapter provides the following information about driver installation:

Installing Linux Driver Software

“Installing Windows Driver Software” on page 20

“Installing VMware Driver Software” on page 23

Installing Linux Driver SoftwareThis section describes how to install Linux drivers with or without remote direct memory access (RDMA). It also describes the Linux driver optional parameters, default values, messages, statistics, and public key for Secure Boot.

Installing the Linux Drivers Without RDMA

Installing the Linux Drivers with RDMA

Linux Driver Optional Parameters

Linux Driver Operation Defaults

Linux Driver Messages

Statistics

Importing a Public Key for Secure Boot

NOTECavium now supports QConvergeConsole (QCC) GUI as the only GUI management tool across all Cavium QLogic adapters. QLogic Control Suite (QCS) GUI is no longer supported for the 45000 Series Adapters and adapters based on 57xx/57xxx controllers, and has been replaced by the QCC GUI management tool. QCC GUI provides single-pane-of-glass GUI management for all QLogic adapters from Cavium.In Windows environments, when you run the QCS CLI and Management Agents Installer, it will uninstall the QCS GUI (if installed on the system) and any related components from your system. To obtain the new GUI, download QCC GUI for your adapter from the Marvell Web site (see “Downloading Updates and Documentation” on page xxi).

3–Driver InstallationInstalling Linux Driver Software

12 BC0154501-00 T

The 45000 Series Adapter Linux drivers and supporting documentation are available on the Marvell Web site. See “Downloading Updates and Documentation” on page xxi.

Table 3-1 describes the 45000 Series Adapter Linux drivers.

Install the Linux drivers using either a source Red Hat® Package Manager (RPM) package or a kmod RPM package. The RHEL RPM packages are as follows: qlgc-fastlinq-..src.rpm qlgc-fastlinq-kmp-default-..rpm

The SLES source and kmp RPM packages are as follows: qlgc-fastlinq-..src.rpm qlgc-fastlinq-kmp-default-...rpm

The Ubuntu Debian package is as follows: qlgc-fastlinq-dkms-_all.deb

Table 3-1. Cavium QLogic 45000 Series Adapters Linux Drivers

Linux Driver Description

qed The qed core driver module directly controls the firmware, handles interrupts, and pro-vides the low-level API for the protocol specific driver set. The qed interfaces with the qede, qedr, qedi, and qedf drivers. The Linux core module manages all PCI device resources (registers, host interface queues, and so on). The qed core module requires Linux kernel version 2.6.32 or later. Testing was concentrated on the x86_64 architecture.

qede Linux Ethernet driver for the 45000 Series Adapter. This driver directly controls the hard-ware and is responsible for sending and receiving Ethernet packets on behalf of the Linux host networking stack. This driver also receives and processes device interrupts on behalf of itself (for L2 networking). The qede driver requires Linux kernel version 2.6.32 or later. Testing was concentrated on the x86_64 architecture.

qedr Linux RoCE driver that works in the OpenFabrics Enterprise Distribution (OFED™) envi-ronment in conjunction with the qed core module and the qede Ethernet driver. Some RDMA user space applications may require that the libqedr user library is installed on the server. R.

qedi Linux iSCSI-Offload driver for the 45000 Series Adapters. This driver works with the Open iSCSI library.

qedf Linux FCoE-Offload driver for the 45000 Series Adapters. This driver works with Open FCoE library.


13 BC0154501-00 T

The following kernel module (kmod) RPM installs Linux drivers on SLES hosts running the Xen Hypervisor: qlgc-fastlinq-kmp-xen-...rpmThe following source RPM installs the RDMA library code on RHEL and SLES hosts: qlgc-libqedr-...src.rpmThe following source code TAR BZip2 (BZ2) compressed file installs Linux drivers on RHEL and SLES hosts: fastlinq-.tar.bz2

Installing the Linux Drivers Without RDMA

To install the Linux drivers without RDMA:1. Download the 45000 Series Adapter Linux drivers from the Marvell Web site.

2. Remove the existing Linux drivers, as described in “Removing the Linux Drivers” on page 14.

3. Install the new Linux drivers using one of the following methods:

Installing Linux Drivers Using the src RPM Package

Installing Linux Drivers Using the kmp/kmod RPM Package

Installing Linux Drivers Using the TAR File

NOTEFor network installations through NFS, FTP, or HTTP (using a network boot disk), you may require a driver disk that contains the qede driver. Compile the Linux boot drivers by modifying the makefile and the make environment.

NOTEWhen using the following procedures in a CentOS environment, follow the instructions for RHEL. When performing these tasks in an Ubuntu environment, see the Install.txt document in the Ubuntu package that is available for your adapter on the Marvell Web site.


14 BC0154501-00 T

Removing the Linux DriversThere are two procedures for removing Linux drivers: one for a non-RDMA environment and another for an RDMA environment. Choose the procedure that matches your environment.

To remove Linux drivers in a non-RDMA environment, unload and remove the drivers:Follow the procedure that relates to the original installation method and the OS.

If the Linux drivers were installed using an RPM package, issue the following commands:rmmod qedermmod qeddepmod -arpm -e qlgc-fastlinq-kmp-default-.

If the Linux drivers were installed using a TAR file, issue the following commands:rmmod qedermmod qeddepmod -a For RHEL and CentOS:

cd /lib/modules//extra/qlgc-fastlinqrm -rf qed.ko qede.ko qedr.ko

For SLES:cd /lib/modules//updates/qlgc-fastlinqrm -rf qed.ko qede.ko qedr.ko

For Ubuntu, issue the following command:dpkg -r fastlinq-dkms

To remove Linux drivers in an RDMA environment:1. Unload and remove the Linux drivers.

modprobe -r qedrmodprobe -r qedemodprobe -r qeddepmod -a


15 BC0154501-00 T

2. Remove the driver module files:

If the drivers were installed using an RPM package, issue the following command:rpm -e qlgc-fastlinq-kmp-default-.

If the drivers were installed using a TAR file, issue the following commands for your operating system:

For RHEL and CentOS:cd /lib/modules//extra/qlgc-fastlinqrm -rf qed.ko qede.ko qedr.koFor SLES:cd /lib/modules//updates/qlgc-fastlinqrm -rf qed.ko qede.ko qedr.ko

Installing Linux Drivers Using the src RPM PackageTo install Linux drivers using the src RPM package:1. Issue the following at a command prompt:

rpm -ivh qlgc-fastlinq-.src.rpm2. Change the directory to the RPM path and build the binary RPM for the

kernel.

For RHEL and CentOS:cd /root/rpmbuildrpmbuild -bb SPECS/fastlinq-.specFor SLES:cd /usr/src/packagesrpmbuild -bb SPECS/fastlinq-.spec

3. Install the newly compiled RPM:rpm -ivh RPMS//qlgc-fastlinq-..rpm

NOTEThe --force option may be needed on some Linux distributions if conflicts are reported.


16 BC0154501-00 T

The drivers will be installed in the following paths.

For SLES:/lib/modules//updates/qlgc-fastlinqFor RHEL and CentOS:/lib/modules//extra/qlgc-fastlinq

4. Turn on all ethX interfaces as follows:ifconfig up

5. For SLES, use YaST to configure the Ethernet interfaces to automatically start at boot by setting a static IP address or enabling DHCP on the interface.

Installing Linux Drivers Using the kmp/kmod RPM PackageTo install kmod RPM package:1. Issue the following command at a command prompt:

rpm -ivh qlgc-fastlinq-..rpm 2. Reload the driver:

modprobe -r qedemodprobe qede

Installing Ubuntu Linux DriversTo install Ubuntu Linux drivers, issue the following command:dpkg -i fastlinq-dkms__all.deb

Installing Linux Drivers Using the TAR FileTo install Linux drivers using the TAR file:1. Create a directory and extract the TAR files to the directory:

tar xjvf fastlinq-.tar.bz22. Change to the recently created directory, and then install the drivers:

cd fastlinq-make clean; make installThe qed and qede drivers will be installed in the following paths.

For SLES:/lib/modules//updates/qlgc-fastlinq


17 BC0154501-00 T

For RHEL and CentOS:/lib/modules//extra/qlgc-fastlinq

3. Test the drivers by loading them (unload the existing drivers first, if necessary):rmmod qedermmod qed modprobe qedmodprobe qede

Installing the Linux Drivers with RDMA

To install Linux drivers in an inbox OFED environment:1. Download the 45000 Series Adapter Linux drivers from the Marvell Web site.

2. Configure RoCE on the adapter, as described in “Configuring RoCE on the Adapter for Linux” on page 143.

3. Remove existing Linux drivers, as described in “Removing the Linux Drivers” on page 14.

4. Install the new Linux drivers using one of the following methods:

Installing Linux Drivers Using the kmp/kmod RPM Package

Installing Linux Drivers Using the TAR File

5. Install libqedr libraries to work with RDMA user space applications. The libqedr RPM is available only for inbox OFED. Issue the following command:rpm –ivh qlgc-libqedr-..rpm

6. To build and install the libqedr user space library, issue the following command:'make libqedr_install'

NOTEWhen using the following procedures in a CentOS environment, follow the instructions for RHEL. When performing these tasks in an Ubuntu environment, see the Install.txt document in the Ubuntu package that is available for your adapter on the Marvell Web site.


18 BC0154501-00 T

7. Test the drivers by loading them as follows:modprobe qedmodprobe qedemodprobe qedr

Linux Driver Optional ParametersTable 3-2 describes the optional parameters for the qede driver.

Linux Driver Operation DefaultsTable 3-3 lists the qed and qede Linux driver operation defaults.

Table 3-2. qede Driver Optional Parameters

Parameter Description

debug Controls driver verbosity level similar to ethtool -s msglvl.int_mode Controls interrupt mode other than MSI-X.gro_enable Enables or disables the hardware generic receive offload (GRO) feature.

This feature is similar to the kernel's software GRO, but is only performed by the device hardware.

err_flags_override A bitmap for disabling or forcing the actions taken in case of a hardware error: bit #31 – An enable bit for this bitmask bit #0 – Prevent hardware attentions from being reasserted bit #1 – Collect debug data bit #2 – Trigger a recovery process bit #3 – Call WARN to get a call trace of the flow that led to the error

Table 3-3. Linux Driver Operation Defaults

Operation qed Driver Default qede Driver Default

Speed Auto-negotiation with speed advertised

Auto-negotiation with speed advertised

MSI/MSI-X Enabled EnabledFlow Control — Auto-negotiation with RX and

TX advertised


19 BC0154501-00 T

Linux Driver MessagesTo set the Linux driver message detail level, issue one of the following commands:

ethtool -s msglvl modprobe qede debug=

Where represents bits 0–15, which are standard Linux networking values, and bits 16 and greater are driver-specific.

StatisticsTo view detailed statistics and configuration information, use the ethtool utility. See the ethtool man page for more information.

Importing a Public Key for Secure BootLinux drivers require that you import and enroll the QLogic public key to load the drivers in a Secure Boot environment. Before you begin, ensure that your server supports Secure Boot. This section provides two methods for importing and enrolling the public key.

To import and enroll the QLogic public key:1. Download the public key from the following Web page:

http://ldriver.qlogic.com/Module-public-key/

2. To install the public key, issue the following command:# mokutil --root-pw --import cert.derWhere the --root-pw option enables direct use of the root user.

3. Reboot the system.

MTU — 1500 (range is 46–9600)Rx Ring Size — 1000Tx Ring Size — 4078 (range is 128–8191)Coalesce Rx Microseconds — 24 (range is 0–255)Coalesce Tx Microseconds — 48TSO — Enabled

Table 3-3. Linux Driver Operation Defaults (Continued)

Operation qed Driver Default qede Driver Default

http://ldriver.qlogic.com/Module-publi

filedownloads.cavium.comfiledownloads.cavium.com/Files_bk/96813/UsersGuide_45000_CNA-I… · ii BC0154501-00 T User’s Guide—Converged Network Adapters and Intelligent Ethernet

Documents