Cisco SFS InfiniBand Redundancy Configuration Guide · iii Cisco SFS InfiniBand Redundancy Configuration Guide 78-12957-02 CONTENTS Preface vii Audience vii Organization vii Conventions

Cisco SFS InfiniBand Redundancy Configuration GuideRelease 2.10December 2007

Americas HeadquartersCisco Systems, Inc.170 West Tasman DriveSan Jose, CA 95134-1706 USAhttp://www.cisco.comTel: 408 526-4000

800 553-NETS (6387)Fax: 408 527-0883

Text Part Number: OL-12957-02

http://www.cisco.com

F INAL REV IEW—CISCO CONF IDENT IAL

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.

Cisco SFS InfiniBand Redundancy Configuration Guide© 2007 Cisco Systems, Inc. All rights reserved.

CCVP, the Cisco logo, and Welcome to the Human Network are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of Cisco Systems, Inc.; and Access Registrar, Aironet, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, LightStream, Linksys, MeetingPlace, MGX, Networkers, Networking Academy, Network Registrar, PIX, ProConnect, ScriptShare, SMARTnet, StackWise, The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0711R)

78-12957-02

C O N T E N T S

Preface vii

Audience vii

Organization vii

Conventions viii

Related Documentation ix

x

C H A P T E R 1 Overview 1-1

C H A P T E R 2 Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy 2-1

Cisco SFS 7008P Server Switch Redundancy 2-2

Software Redundancy 2-2

Power Supply Module Redundancy 2-4

Fan Tray Redundancy 2-4

Management Interface Module Redundancy 2-5

Fabric Controller Redundancy 2-6

Line Interface Module Redundancy 2-6

IB Fabric Redundancy 2-8

Cisco SFS 7000P and SFS 7000D Server Switch Redundancy 2-9

Power Supply Redundancy 2-9

Port Redundancy 2-10

IB Fabric Redundancy 2-10

C H A P T E R 3 InfiniBand Server Switch Module Redundancy for the IBM BladeCenter 3-1

C H A P T E R 4 Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy 4-1

Cisco SFS 3504 Server Switch Redundancy 4-2

Software Redundancy 4-2

IB Switch Module Redundancy 4-2

Fabric Redundancy 4-2

AC Power-Fan Module Redundancy 4-3

Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy 4-4

iiiCisco SFS InfiniBand Redundancy Configuration Guide

Contents

Cisco SFS 3012R Server Switch Redundancy 4-5

Power Supply Redundancy 4-5

Blower Redundancy 4-6

Controller Module Redundancy 4-6

InfiniBand Switch Module Redundancy 4-7

Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy 4-7


Cisco SFS 3001 Server Switch Redundancy 4-9

Power Supply Module Redundancy 4-9

Fan Redundancy 4-9


C H A P T E R 5 Subnet Manager Redundancy 5-1

Embedded Subnet Manager 5-3

High-Performance Subnet Manager 5-4

Setting up Master and Standby Subnet Managers 5-4

Setting up Master and Standby Subnet Managers Using Embedded Subnet Managers 5-4

Setting up Master and Standby Subnet Managers with High-Performance Subnet Managers 5-6

Setting Up Database Synchronization 5-8

Setting up Database Synchronization for Embedded Subnet Managers 5-9

Setting up Database Synchronization for High-Performance Subnet Managers 5-10

C H A P T E R 6 Host Redundancy, and IPoIB and SRP Redundancies 6-1

HCA Redundancy 6-1

Single HCA Redundancy 6-1

Multiple HCA Redundancy 6-3

Two HCAs with the IBM BladeCenter 6-3

IPoIB High Availability 6-4

Cisco SFS IPoIB High Availability 6-4

Merging Physical Ports 6-4

Unmerging Physical Ports 6-5

OFED IPoIB High Availability 6-6

Configuring IPoIB High Availability 6-6

Verifying IPoIB High Availability 6-8

OFED SRP High Availability 6-8

C H A P T E R 7 Ethernet Gateway and IPoIB Redundancies 7-1

Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch 7-3

ivCisco SFS InfiniBand Redundancy Configuration Guide

78-12957-02

Contents

Verifying Redundancy Configuration for Cisco SFS 3504 Server Switches 7-6

Verifying Bridge Group Configuration for Cisco SFS 3504 Server Switches 7-8

Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch 7-10

Configuring Ethernet Gateway Redundancy Using a Single Cisco SFS 3012R Server Switch 7-10

Verifying Redundancy Group Configuration for a Single Cisco SFS 3012R Server Switch 7-13

Verifying Bridge Group Configuration for a Single Cisco SFS 3012R Server Switch 7-14

Configuring Ethernet Gateway Redundancy Using Dual Cisco SFS 3012R Server Switches 7-15

Verifying Redundancy Group Configuration for Dual Cisco SFS 3012R Server Switches 7-19

Verifying Bridge Group Configuration for Dual Cisco SFS 3012R Server Switches 7-20

Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch 7-23

Verifying Redundancy Group Configuration for Cisco SFS 3001 Server Switches 7-27

Verifying Bridge Group Configuration for Cisco SFS 3001 Server Switches 7-28

C H A P T E R 8 Fibre Channel Gateway and SRP Redundancies 8-1

Dynamic Load Balancing 8-2

Dynamic Gateway Failover 8-2

Path Affinity 8-2

Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch 8-3

Verifying Configured Initiator 8-5

Verifying IT 8-6

Verifying LU 8-7

Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch 8-8

Configuring Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012 Server Switch 8-8

Verifying Configuration for a Single Cisco SFS 3012R Server Switch 8-10

Configuring Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R Server Switches 8-12

Verifying Configuration for Two Cisco SFS 3012R Server Switches 8-13

Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch 8-16

Configuring Two Cisco SFS 3001 Server Switches 8-16

Verifying Redundancy Configuration for Cisco SFS 3001 Server Switches 8-17

C H A P T E R 9 Typical Redundancy Use Case 9-1

A P P E N D I X A Acronyms and Abbreviations A-1

I N D E X

vCisco SFS InfiniBand Redundancy Configuration Guide

78-12957-02

Contents

viCisco SFS InfiniBand Redundancy Configuration Guide

78-12957-02

Preface

This preface describes who should read the Cisco SFS InfiniBand Redundancy Configuration Guide, how it is organized and its document conventions. It contains the following sections:

• Audience, page vii

• Organization, page vii

• Conventions, page viii

• Related Documentation, page ix

• Obtaining Documentation and Submitting a Service Request, page x

AudienceThe intended audience for this document is the administrator responsible for the Enterprise data center who configures redundancy in a Server Fabric Switch environment. This administrator should have experience configuring and managing equipment such as server switches, the Subnet Manager, Ethernet gateways, Host Channel Adapters, and software drivers.

Organization This guide is organized as follows:

Chapter Title Description

Chapter 1 Overview This chapter provides an overview about the guide.

Chapter 2 Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy

This chapter includes redundancy information for the Cisco SFS 7008P and Cisco SFS 7000 Series Server Switches.

Chapter 3 InfiniBand Server Switch Module Redundancy for the IBM BladeCenter

This chapter includes redundancy information for the InfiniBand Server Switch for the IBM BladeCenter Redundancy.

viiCisco SFS InfiniBand Redundancy Configuration Guide

OL-12957-02

PrefaceConventions

ConventionsThis document uses the following conventions:

Chapter 4 Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy

This chapter includes redundancy information for the Cisco SFS 3504 and the Cisco SFS 3000 Series Server Switches.

Chapter 5 Subnet Manager Redundancy This chapter includes the Subnet Manager redundancy information.

Chapter 6 Host Redundancy, and IPoIB and SRP Redundancies

This chapter includes information about HCA, IPoIB, and SRP redundancies.

Chapter 7 Ethernet Gateway and IPoIB Redundancies

This chapter includes information about Ethernet gateway redundancy.

Chapter 8 Fibre Channel Gateway and SRP Redundancies

This chapter includes information about Fibre Channel gateway redundancy.

Chapter 9 Typical Redundancy Use Case This chapter describes a typical redundancy use case.

Appendix A Acronyms and Abbreviations This chapter defines the acronyms and abbreviations that are used in this publication.

Chapter Title Description

Convention Description

boldface font Commands, command options, and keywords are in boldface. Bold text indicates Chassis Manager elements or text that you must enter as-is.

italic font Arguments in commands for which you supply values are in italics. Italics not used in commands indicate emphasis.

Menu1 > Menu2 > Item…

Series indicate a pop-up menu sequence to open a form or execute a desired function.

[ ] Elements in square brackets are optional.

{ x | y | z } Alternative keywords are grouped in braces and separated by vertical bars. Braces can also be used to group keywords and/or arguments; for example, {interface interface type}.

[ x | y | z ] Optional alternative keywords are grouped in brackets and separated by vertical bars.

string A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

screen font Terminal sessions and information the system displays are in screen font.

boldface screen font

Information you must enter is in boldface screen font.

viiiCisco SFS InfiniBand Redundancy Configuration Guide

OL-12957-02

PrefaceRelated Documentation

Notes use the following convention:

Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

Cautions use the following convention:

Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

Related DocumentationFor additional information related to Cisco SFS InfiniBand Redundancy Configuration Guide, see the following documents:

• Cisco SFS InfiniBand Software Configuration Guide

• Cisco SFS 3504 Multifabric Server Switch Hardware Installation Guide

• Cisco SFS 3012R Multifabric Server Switch Hardware Installation Guide

• Cisco SFS 3001 Multifabric Server Switch Hardware Guide

• Cisco SFS 7008P InfiniBand Server Switch Hardware Installation Guide

• Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide

• Cisco High-Performance Subnet Manager for InfiniBand Server Switches

• Cisco SFS Product Family Chassis Manager User Guide

• Cisco SFS Product Family Element Manager User Guide

• Cisco SFS Product Family Command Reference

• Cisco SFS InfiniBand Fibre Channel Gateway User Guide

• Cisco SFS InfiniBand Ethernet Gateway User Guide

• Cisco InfiniBand Host Channel Adapter Hardware Installation Guide

italic screen font Arguments for which you supply values are in italic screen font.

^ The symbol ^ represents the key labeled Control—for example, the key combination ^D in a screen display means hold down the Control key while you press the D key.

< > Nonprinting characters, such as passwords are in angle brackets.

!, # An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

Convention Description

ixCisco SFS InfiniBand Redundancy Configuration Guide

OL-12957-02

Preface

Obtaining Documentation and Submitting a Service RequestFor information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What’s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Subscribe to the What’s New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.

xCisco SFS InfiniBand Redundancy Configuration Guide

OL-12957-02

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

CisOL-12957-02

C H A P T E R 1
Overview
The Cisco SFS InfiniBand Redundancy Configuration Guide contains redundancy configuration information for use with Enterprise solutions that contain Cisco InfiniBand (IB) Server Fabric Switches (SFS). It describes hardware and software redundancies and redundant configurations that are available in the Cisco SFS IB environment.

Note For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and Abbreviations.”

This guide describes the various types of hardware and software redundancies available with the different products that are available to build an SFS IB environment. It also describes the various typical redundant configurations that a user may build in this environment and includes a typical use case that has no single point of failure.

The redundancies described in this guide include the following topics:

• Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy

• InfiniBand Server Switch Module Redundancy for the IBM BladeCenter

• Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy

• Subnet Manager Redundancy

• Host Redundancy, and IPoIB and SRP Redundancies

• Ethernet Gateway and IPoIB Redundancies

• Fibre Channel Gateway and SRP Redundancies

• Typical Redundancy Use Case

1-1co SFS InfiniBand Redundancy Configuration Guide

Chapter 1 Overview

1-2Cisco SFS InfiniBand Redundancy Configuration Guide

OL-12957-02

CisOL-12957-02

C H A P T E R 2
CiscoSFS 7008P and SFS 7000 Series Server Switch Redundancy
This chapter describes the Cisco SFS 7008P and SFS 7000 Series Server Switch redundancy and includes the following sections:

• Cisco SFS 7008P Server Switch Redundancy, page 2-2

• Cisco SFS 7000P and SFS 7000D Server Switch Redundancy, page 2-9

For information related to Subnet Manager Redundancy, see Chapter 5, “Subnet Manager Redundancy”.



Chapter 2 Cisco SFS 7008P and SFS 7000 Series Server Switch RedundancyCisco SFS 7008P Server Switch Redundancy

Cisco SFS 7008P Server Switch RedundancyThis section describes redundancies in the Cisco SFS 7008P Server Switch and includes the following topics:

• Software Redundancy, page 2-2

• Power Supply Module Redundancy, page 2-4

• Fan Tray Redundancy, page 2-4

• Management Interface Module Redundancy, page 2-5

• Fabric Controller Redundancy, page 2-6

• Line Interface Module Redundancy, page 2-6

• IB Fabric Redundancy, page 2-8

For more details about the Cisco SFS 7008P Server Switch, see the Cisco SFS 7008P InfiniBand Server Switch Hardware Installation Guide.

Software RedundancyThis section describes redundancy in the Cisco SFS 7008P Server Switch software.

The Cisco SFS 7008P Server Switch supports the hot-standby feature. When the primary controller fails, a standby controller assumes management of the server switch without having to reboot or reset other cards in the chassis.

When two controllers are installed in a Cisco SFS 7008P Server Switch, one controller acts as the primary controller, and the other acts as the standby controller. The primary controller is responsible for managing the chassis. The standby controller waits to take-over if the primary controller fails or is rebooted.

Verify the primary switch controller by entering the show card command in the CLI. The oper-code of the primary controller card is normal and for the standby controller is standby. An asterix marks the controller card that services this CLI session. So, from a console CLI session, your console port is on the card marked with an asterix.

The following is sample output from the show card command and verifies the status of each controller card:

SFS-7008P# show card ========================================================================= Card Information ========================================================================= admin oper admin oper oper slot type type status status code ------------------------------------------------------------------------- 11* controllerFabric12x controllerFabric12x up up normal 12 controllerFabric12x controllerFabric12x up up standby


OL-12957-02


The following is sample output from the show card command from the console of slot 12 of the server switch and verifies the status of each controller card:

SFS-7008P# show card ========================================================================= Card Information ========================================================================= admin oper admin oper oper slot type type status status code ------------------------------------------------------------------------- 11 controllerFabric12x controllerFabric12x up up normal 12* controllerFabric12x controllerFabric12x up up standby

When a Cisco SFS 7008P Server Switch is powered on, the fabric card in slot 11 assumes the primary card status, and the fabric card in slot 12 is the standby card. A fabric card is only eligible to be a controller card if the fabric card is installed in either slot 11 or 12 and a corresponding management interface module is available.

If a fabric card is operating in the recovery mode (such as, when it is executing the OS Recovery Image software), it is not eligible to be a primary or standby controller. The master and standby controllers automatically synchronize the state and configuration information. When a standby controller assumes managing a chassis, the service of other cards in the chassis (such as the Line Interface Modules, fabric controllers, and management interface modules) are not impacted. The other cards are not rebooted, reset, or interrupted. The standby controller is accessible through the serial console port. A user cannot access the standby controller using Telnet, SSH, SNMP, or HTTP.

The OS CLI is available on the standby controller (through the serial console only). The CLI on a standby controller is limited to read-only operations. A user can enter show commands but cannot enter config commands.

A card is only placed in-service if that card is running the same software as the primary controller. When the standby controller card has a different version than the primary controller card, the standby controller card shows a wrong image for its card opercode. In this event, no synchronization occurs between the two cards. The sys-sync-state for both the controller cards stay at not started.

When a hot-standby controller card takes over in the event of a primary controller card failure, the hot-standby controller card behaves according to its sys-sync-state.

When the sys-snyc-state is complete, the hot-standby controller card continues management without disturbing the services. No additional configuration file is executed, and there is no reboot to the node cards.

When the sys-sync-state is not started, the hot-standby controller card executes its startup-config, if it is present and continues management. There is no reboot to the node cards.

When the sys-sync-state is in progress, the primary controller card and the hot-standby controller card are partially synchronized. The hot-standby controller card reboots itself to avoid unpredictable results.

The synchronization begins only after the operStatus of the hot-standby controller card changes to up.


OL-12957-02


Power Supply Module RedundancyThis section describes the power supply module redundancy in the Cisco SFS 7008P Server Switch.

The Cisco SFS 7008P Server Switch has two AC-DC bulk power supply modules (see Figure 2-1). Each power supply has self-contained fans for cooling. Only one power supply, in either of the two slots is required to power the system. The second power supply acts as a redundant power supply. Each power supply has its own AC inlet and runs on an independent AC circuit. If the active power supply fails, the second power supply automatically takes over the full load of the server switch. There is no user intervention required in case of a failover. The current is shared in an active-active mode.

Figure 2-1 Cisco SFS 7008P Server Switch Front View

If a power supply module fails, it must remain within the chassis until a replacement is available. If it is removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P Server Switch is in operation, the power supply module bay can remain empty for no more than three minutes.

Fan Tray RedundancyThis section describes fan tray redundancy in the Cisco SFS 7008P Server Switch.

The Cisco SFS 7008P Server Switch has two fan trays. Only one fan tray in either of the two slots is required to cool the system. The fan trays are hot swappable. The fan trays operate in an active-active mode.

If a fan tray fails, it must remain within the chassis until a replacement is available. If it is removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P Server Switch is in operation, the fan tray bay can remain empty for no more than three minutes.

1919

86

Power supplies

Fan FanNode slot 1 Slot 9

Slot 10

Slot 11

Slot 12Slot 13

Slot 14

Node slot 2

Node slot 3

Node slot 4

Core slot 1

Core slot 2


OL-12957-02


Management Interface Module RedundancyThis section describes management interface module redundancy in the Cisco SFS 7008P Server Switch.

The Cisco SFS 7008P Server Switch supports redundant, hot-swappable management interface modules. Each management interface module is paired to one of the fabric controller core modules. There are two core slots in the Cisco SFS 7008P Server Switch and two management interface modules (see Figure 2-1 and Figure 2-2). The controller in each of the core slots uses a management interface module to communicate with the outside network. Each management interface module provides its own serial and Ethernet port. Both sets of ports must be connected. The failover of the management interface module is paired with the failover of the fabric controller cards (see the “Fabric Controller Redundancy” section on page 2-6).

If a management interface module fails, it must remain within the chassis until a replacement is available. If it is removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P Server Switch is in operation, the management interface module bay can remain empty for no more than three minutes.

Figure 2-2 Cisco SFS 7008P Rear View - Management Interface Modules and Line Interface

Modules

1833

86

Management I.Omodules

Line InterfaceModules


OL-12957-02


Fabric Controller RedundancyThis section describes the fabric controller redundancy in the Cisco SFS 7008P Server Switch.

The behavior and responsibility of each fabric controller is determined by the type of slot into which it is inserted. A fabric controller can be installed in either a node slot or a core slot. When the software on a fabric controller module detects that a module is inserted in a core slot, it arbitrates for system mastership and runs. When you power on the Cisco SFS 7008P Server Switch, by default, the card in slot 11 becomes the active card and the card in slot 12 becomes the standby card (see Figure 2-1). The master and standby controller cards automatically synchronize state and configuration information. Thus, the switch does not have to be rebooted and none of the cards in the chassis require resetting if a switch failover occurs.

Fabric controller cards in the core slots are paired to the management interface modules of the Cisco SFS 7008P Server Switch (see the “Management Interface Module Redundancy” section on page 2-5). For redundancy, the pairing of both fabric controller cards and the management interface modules should be installed and operational. The two core cards operate in an active-active mode and are required for 100% throughput. If one core card fails, 50% of bandwidth is available to the system. If the failing card is the active master, the standby master assumes control of the chassis.

Removing the fabric controller in the core slot that currently acts as master, causes a failover to the standby pair of the fabric controller and management interface module pair. Before removing a fabric controller from one of the core slots, make sure that the redundant core fabric controller is functional.

The fabric controllers in the node slots act as slaves and do not operate in an active-standby configuration. Each node card is paired with two Line Interface Modules (see the “Line Interface Module Redundancy” section on page 2-6). If a node card fails, only ports connected to it and its corresponding Line Interface Modules are affected.

If a core or node card fails, it can be left within the chassis until a replacement is available. If it is removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P Server Switch is in operation, the card bay can be left empty for no more than three minutes.

Line Interface Module RedundancyThis section describes the Line Interface Module redundancy in the Cisco SFS 7008P Server Switch.

Line Interface Modules support redundant connection from the HCAs (see Figure 2-2). Line Interface Modules are hot-swappable and redundant components. Each Line Interface Module is paired with a fabric controller node card (see the “Fabric Controller Redundancy” section on page 2-6).

If a Line Interface Module fails, it can be left within the chassis until a replacement is available. If it is removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7008P Server Switch is in operation, the Line Interface Module bay can be left empty for no more than three minutes.

Figure 2-3 is an illustration that shows the core cards and Line Interface Module redundancy within a Cisco SFS 7008P Server Switch.


OL-12957-02


Figure 2-3 Redundancy within a Cisco SFS 7008P Server Switch

1833

40

SFS 7008P

InfiniBand Fabric

InfiniBandHost

InfiniBandHost

Line interface modules pair 1,2




SFS 7000D-1 SFS 7000D-2


OL-12957-02


IB Fabric RedundancyThis section describes the IB fabric redundancy using Cisco SFS 7008P Server Switches.

For redundancy at the fabric level, IB HCAs can be dual-connected to a redundant pair of Cisco SFS 7008P Server Switches (see Figure 2-4). The IB links are active. Traffic over redundant IB links varies and is based on upper-level protocols and applications.

Figure 2-4 Redundancy with Dual Cisco SFS 7008P Server Switches

1833

39

SFS 7008P-1 SFS 7008P-2

InfiniBandHost-1

InfiniBandHost-2

InfiniBandHost-3

InfiniBandHost-4

SFS 7000D-1 SFS 7000D-2 SFS 7000D-3 SFS 7000D-4


OL-12957-02

Chapter 2 Cisco SFS 7008P and SFS 7000 Series Server Switch RedundancyCisco SFS 7000P and SFS 7000D Server Switch Redundancy

Cisco SFS 7000P and SFS 7000D Server Switch RedundancyThis section describes the Cisco SFS 7000P and SFS 7000D Server Switch redundancy and includes the following topics:

• Power Supply Redundancy, page 2-9

• Port Redundancy, page 2-10

• IB Fabric Redundancy, page 2-10

Redundancy in the Cisco SFS 7000P and SFS 7000D Server Switches is supported at the hardware, port, and fabric levels. For more details about the Cisco SFS 7000P and SFS 7000D Server Switches, see the Cisco SFS 7000P and SFS 7000D InfiniBand Server Switches Hardware Installation Guide.

Power Supply RedundancyThis section describes power supply redundancy in the Cisco SFS 7000P and SFS 7000D Server Switches.

The Cisco SFS 7000P and SFS 7000D Server Switches power supply module is an integrated power supply and fan unit. A server switch can have up to two power supplies installed (see Figure 2-5). The switch requires only one power supply to function. The second power supply acts as a redundant power supply. The power supply modules are hot swappable. The replacement of any one power supply module does not disrupt the operation of the device and can be successfully completed without removing the device from a rack or disconnecting any cables.

Each power supply has its own AC inlet and runs on an independent AC circuit. The server switch automatically has the power supplies operating in active-active or active-standby mode. If one power supply were to fail, the second power supply automatically takes over the full load of the server switch. There is no user intervention required in case of a failover.

If a power supply module fails, it can be left within the chassis until a replacement is available. If it is removed, a blanking panel must be installed instead. During replacement, when the Cisco SFS 7000P or the SFS 7000D Server Switch is in operation, the power supply module bay can be left empty for no more than three minutes.

Figure 2-5 Cisco SFS 7000P and SFS 7000D Server Switches

1919

95

Power supply modules


OL-12957-02

Chapter 2 Cisco SFS 7008P and SFS 7000 Series Server Switch RedundancyCisco SFS 7000P and SFS 7000D Server Switch Redundancy

Port RedundancyThis section describes port redundancy using the Cisco SFS 7000P and SFS 7000D Server Switches.

The Cisco SFS 7000P and SFS 7000D Server Switches each have 24 IB ports. Redundancy at the port level is such that if any single IB port fails, none of other ports have interrupted service.

In addition, to achieve port redundancy, users can employ two server switches. If the IB ports on one server switch fails, the second server switch automatically takes over the load of the first server switch.

IB Fabric RedundancyThis section describes fabric redundancy in the Cisco SFS 7000P and SFS 7000D Server Switches.

For redundancy at the fabric level, IB HCAs can be dual-connected to a redundant pair of Cisco SFS 7000 Series Server Switches. The Cisco SFS 7000 Series Server Switch redundant configuration is active-active. No hardware configuration is required. The IB links are active-active. But applications and upper-level protocol use of redundant IB links varies and could be active-active or active-standby, depending on the application.

In this typical configuration, a dual-port HCA is connected to a pair of Cisco SFS 7000 Series Server Switches (see Figure 2-6). This configuration provides server redundancy.

Figure 2-6 One Dual-Port HCA Connected to a Redundant Pair of Cisco SFS 7000 Series Server

Switches

For greater redundancy, connect two single-port HCAs to a redundant pair of server switches. Such a configuration provides host and IB fabric redundancy.

Note Figure 2-6 shows the Cisco SFS 7000D Server Switches in a redundant configuration. The Cisco SFS 7000P Server Switches can also be connected in similar redundant configurations.

SFS 7000D-1 SFS 7000D-2

InfiniBand Host

1828

55

InfiniBand Fabric 1 InfiniBand Fabric 2


OL-12957-02

CisOL-12957-02

C H A P T E R 3
InfiniBand Server Switch Module Redundancy for the IBM BladeCenter
This chapter contains redundancy information about the InfiniBand Server Switch for the IBM BladeCenter.

For more information on the InfiniBand Server Switch for the IBM BladeCenter, see the Cisco 4x InfiniBand Switch Module for IBM BladeCenter User Guide.


The user can create a redundant dual-switch topology in an IBM BladeCenter. To enable IB redundancy for the IBM BladeCenter chassis, you must install one server switch module in each available slot. HCA expansion cards do not support redundant links to a single server switch module slot. When you add a second server switch module to the BladeCenter chassis, each port of each HCA expansion card connects to a server switch module. Figure 3-1 shows an example of a configuration in which there are 14 dual-port HCAs in the IBM BladeCenter. Each HCA, in this redundant configuration, is connected to the two server switch modules in the BladeCenter.


Chapter 3 InfiniBand Server Switch Module Redundancy for the IBM BladeCenter

Figure 3-1 InfiniBand Switch Redundancy for the IBM BladeCenter

Note Connect the server switch modules to a redundant outside fabric.

The server switch modules do not connect to each other within the BladeCenter chassis. To connect the modules to enable the Subnet Manager failover, connect the modules with an IB cable through the external connectors. Before you connect the two server switch modules in your chassis, configure the priority of the Subnet Managers on the modules. For more details about Subnet Manager redundancy, see Chapter 5, “Subnet Manager Redundancy.” For more details about the High-Performance Subnet Manager, see the Cisco High-Performance Subnet Manager for InfiniBand Server Switches.

After you configure your subnet manager priority, connect your server switch module to the external IB fabric. By default, the external interfaces on your server switch module auto-negotiate speed with the fabric.

Port-1

1833

38

HCA-1

RedundantInfiniBand Fabric

IBM BladeCenter

ServerSwitch

Module-1

ServerSwitchModule-2

Port-2

HCA-14

InfiniBand HostBlade 1

InfiniBand HostBlade 14

Port-1 Port-2


OL-12957-02

CisOL-12957-02

C H A P T E R 4
Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy
This chapter describes redundancy information about the Cisco SFS 3504 and Cisco SFS 3000 Series Server Switches and includes the following sections:

• Cisco SFS 3504 Server Switch Redundancy, page 4-2

• Cisco SFS 3012R Server Switch Redundancy, page 4-5

• Cisco SFS 3001 Server Switch Redundancy, page 4-9

The Cisco SFS 3000 Series includes the Cisco SFS 3001, Cisco SFS 3012R Server Switches. For Ethernet gateway and Fibre Channel gateway redundancy related to the Cisco SFS 3504 and SFS 3000 Series Server Switches, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies” and Chapter 8, “Fibre Channel Gateway and SRP Redundancies.”



Chapter 4 Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch RedundancyCisco SFS 3504 Server Switch Redundancy

Cisco SFS 3504 Server Switch Redundancy This section describes the redundancies supported by the Cisco SFS 3504 Server Switch and includes the following topics:

• Software Redundancy, page 4-2

• IB Switch Module Redundancy, page 4-2

• Fabric Redundancy, page 4-2

• AC Power-Fan Module Redundancy, page 4-3

• Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy, page 4-4

For details about the Cisco SFS 3504 Server Switch, see the Cisco SFS 3504 Multifabric Server Switch Hardware Installation Guide.

Software RedundancyThis section describes software redundancy in the Cisco SFS 3504 Server Switch.

With the Cisco SFS 3504 Server Switch, you can store two operating systems, one as active and one as dormant. Use the A/B partitioning feature in one of the following two ways to achieve redundancy:

• Both A and B partitions arrive from the factory preloaded with the same operating system software on the ‘active’ and ‘dormant’ partitions. You may reload an existing operating system software or install a new generation operating system on the ‘active’ operating system partition, thus leaving the ‘dormant’ partition with the factory-installed operating system.

• Alternate operating system upgrades between the ‘active’ and ‘dormant’ partitions (first partition A, then partition B, and then back to partition A), thereby enabling a rollback to the previous operating system without having to reload or install software image files. This method optimizes the ease and speed of switching operating systems and is similar to a dual boot scheme.

The advantage to using A/B partitions is that a recovery image is readily available. There is no need to fix an image on a bad partition. Instead you use the CLI to boot the new or desired partition.

IB Switch Module RedundancyThis section describes IB switch module redundancy in the Cisco SFS 3504 Server Switch.

The IB switch card is not redundant. However, redundant IB links from a host can be set up by connecting to a switch and using each alternate port, such as ports 1, 3, 5 and such. This spreads connections across the multi-IC switches within the card.

Fabric RedundancyThis section describes fabric redundancy in the Cisco SFS 3504 Server Switch.

The user can configure redundancy with two Cisco SFS 3504 Server Switches. A server with a two-port HCA can be attached to two IB switch modules on two separate Cisco SFS 3504 Server Switches (see Figure 4-1).


OL-12957-02


Figure 4-1 Redundancy with Two Cisco SFS 3504 Server Switches

AC Power-Fan Module RedundancyThis section describes AC power-fan module redundancy in the Cisco SFS 3504 Server Switch.

The Cisco SFS 3504 Server Switch provides dual AC power-fan module redundancy (see Figure 4-2). If one AC power-fan module fails, the other AC power-fan module assumes control immediately. The redundant AC power-fan modules also support hot swaps. When the Cisco SFS 3504 Server Switch includes only one AC power-fan module, you may add a second AC power-fan module while the chassis is in operation. If you have two AC power-fan modules installed, you may remove either one of them without removing power from the chassis. The AC power-fan modules are active-active.

2503

11

InfiniBand Host

ENGW-2

IB Switch Card-1 IB Switch Card-2

SFS 3504-1

FCGW-1 FCGW-2

SAN Fabric

FCGW - Fibre Channel GatewayENGW - Ethernet Gateway

ENGW-1

SFS 3504-2

Ethernet Fabric

IB Fabric


OL-12957-02


Figure 4-2 Cisco SFS 3504 Dual Redundant Power-Fan Modules

Ethernet Gateway Redundancy and Fibre Channel Gateway RedundancyThis section describes the Ethernet and Fibre Channel gateway redundancies in the Cisco SFS 3504 Server Switch.

Ethernet and Fibre Channel gateways connect the Cisco SFS 3504 Server Switch to IP and Fibre Channel networks. There are four gateway slots in the chassis, and gateways can be installed in any of the slots in any combination desired. All gateways have an IB connection to the IB switch. The gateway modules can be configured to provide multiple-gateway redundancy if two or more Ethernet gateways or Fibre Channel gateways are installed in one chassis. The gateways are hot pluggable, so you can add or swap gateways while the chassis is in operation.

For Ethernet gateway and Fibre Channel gateway redundancy related to the Cisco SFS 3504 Server Switch, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies” and Chapter 8, “Fibre Channel Gateway and SRP Redundancies.”

2501

53

1

2

1 - 2 AC-DC power-fan modules


OL-12957-02

Chapter 4 Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch RedundancyCisco SFS 3012R Server Switch Redundancy

Cisco SFS 3012R Server Switch RedundancyThis section describes the Cisco SFS 3012R Server Switch redundancies and includes the following topics:

• Power Supply Redundancy, page 4-5

• Blower Redundancy, page 4-6

• Controller Module Redundancy, page 4-6

• InfiniBand Switch Module Redundancy, page 4-7

• Ethernet Gateway Redundancy and Fibre Channel Gateway Redundancy, page 4-7


For details about the Cisco SFS 3012R Server Switch, see the Cisco SFS 3012R Multifabric Server Switch Hardware Installation Guide.

Power Supply RedundancyThis section describes power supply redundancy in the Cisco SFS 3012R Server Switch.

The Cisco SFS 3012R Server Switch provides dual power supply redundancy (see Figure 4-3). If one power supply fails, the other power supply assumes control immediately. The redundant power supplies also support hot swaps. When the Cisco SFS 3012R Server Switch includes only one power supply, you can add a second power supply while the chassis is in operation. If you have two power supplies installed, you can remove either one of them without removing power from the chassis. The power supplies are active-active.

Figure 4-3 Cisco SFS 3012R Server Switch Power Supply and Blower Modules

1919

85

Power supply modules

Blower Modules


OL-12957-02


Blower RedundancyThis section describes blower modules redundancy in the Cisco SFS 3012R Server Switch.

The two, hot-swappable blower modules of the Cisco SFS 3012R Server Switch maintain the internal temperature in the chassis. Each module contains two blowers, so a fully functional system has four operational blowers (see Figure 4-3). At least three blowers must be functional for continuous operation. You do not need to turn off power to the Cisco SFS 3012R Server Switch to replace a blower module. During replacement, when the Cisco SFS 3012R Server Switch is in operation, the blower can remain empty for no more than three minutes.

Controller Module RedundancyThis section describes controller module redundancy in the Cisco SFS 3012R Server Switch.

Controller modules manage the Cisco SFS 3012R Server Switch and provide Ethernet and serial console port access to the Cisco SFS 3012R Server Switch. Each Cisco SFS 3012R Server Switch contains two controller modules. Figure 4-4 displays the Cisco SFS 3012R Server Switch controller modules with its slot numbers.

Figure 4-4 Cisco SFS 3012R Server Switch Controller Module Side

The Cisco SFS 3012R Server Switch contains an active controller and one active standby controller for redundancy. Upon power-up, the controller in slot 1 becomes the master or active controller. The controller in slot 14 becomes the standby controller. If the master controller fails or reboots, the standby

1807

40

1 3 5 7 9 11 13

2 4 6 8 10 12 14

15

16

1 Slot 1—Default master controller module 14 Slot 14—Default standby controller module

2-13 Slots 2-13—Gateway slots 15-16 Slots 15, 16—IB switches


OL-12957-02


controller automatically takes over as the new master. All I/O modules such as IB switch modules, Fibre Channel gateways, and Ethernet gateways reboot when a failover occurs. During the software upgrade and the installation process, both the master controller and the standby controller get upgraded at the same time. A Subnet Manager runs on each controller module. The Subnet Manager can manage a single Cisco SFS 3012R Server Switch or redundant Cisco SFS 3012R Server Switches. For more details on Subnet Manager, see Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy.”

InfiniBand Switch Module RedundancyThis section describes IB switch module redundancy in the Cisco SFS 3012R Server Switch.

IB switch modules connect the Cisco SFS 3012R Server Switch to IB-attached hosts and other switches in the IB network. The Cisco SFS 3012R Server Switch supports one or two IB switch modules in slots 15 and 16.

Note The current software release requires an IB switch card with an operational CPU to remain installed in slot 16 (see Figure 4-4).

The IB switch cards are not redundant. However, redundant IB links from a host can be set up by connecting to a switch and using each alternate port, such as ports 1, 3, 5 and such. This spreads connections across the multi-IC switches within the card.

Ethernet Gateway Redundancy and Fibre Channel Gateway RedundancyThis section describes Ethernet and Fibre Channel gateway redundancies in the Cisco SFS 3012R Server Switch.

Ethernet and Fibre Channel gateways connect the Cisco SFS 3012R Server Switch to IP and Fibre Channel networks. All gateways have an IB connection to each of the IB switches.

For Ethernet gateway and Fibre Channel gateway redundancy related to the Cisco SFS 3012R Server Switch, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies” and Chapter 8, “Fibre Channel Gateway and SRP Redundancies.”

Fabric RedundancyThis section describes fabric redundancy in the Cisco SFS 3012R Server Switch.

The Cisco SFS 3012R Server Switch is designed for various types of redundancies within a single chassis. Every component in the data path can be dual-connected. A server with a two-port HCA can be dual attached to the two IB switch modules on the Cisco SFS 3012R Server Switch. Each gateway is also dual attached to the dual switch module. In turn, the gateway modules are dual attached to Ethernet or Fibre Channel networks in pairs. With the addition of centralized load balancing and port aggregation, Ethernet and Fibre Channel uplink connections can be trunked and load balanced, allowing the Cisco SFS 3012R Server Switch to reroute around failures. Each independently removable component is hot-swappable.

Figure 4-5 shows the internal Cisco SFS 3012R Server Switch architecture. Components within the dotted line reside within the Cisco SFS 3012R Server Switch chassis.


OL-12957-02


Figure 4-5 Cisco SFS 3012R Server Switch Redundant System Architecture

InfiniBandHost

2500

94

ENGW-1 ENGW-2

Ethernet Fabric

IB Fabric

IB Switch Card-1 IB Switch Card-2

SFS 3012R

FCGW-1 FCGW-2

SAN Fabric



OL-12957-02


Cisco SFS 3001 Server Switch RedundancyThis section describes the Cisco SFS 3001 Server Switch redundancies and includes the following topics:

• Power Supply Module Redundancy, page 4-9

• Fan Redundancy, page 4-9


For more details about the Cisco SFS 3001 Server Switch, see the Cisco SFS 3001 Multifabric Server Switch Hardware Guide.

Power Supply Module RedundancyThis section describes power supply module redundancy in the Cisco SFS 3001 Server Switch.

The Cisco SFS 3001 Server Switch provides two power supplies for 1:1 redundancy (see Figure 4-6). If one power supply module fails, the other immediately assumes control. A power supply module can be hot swapped without any disruption in power. The power supplies are active-active.

Figure 4-6 Cisco SFS 3001 Server Switch -- Front with Bezel Removed

Fan RedundancyThis section describes fan redundancy in the Cisco SFS 3001 Server Switch.

A single Cisco SFS 3001 Server Switch provides fan redundancy. The Cisco SFS 3001 Server Switch has a fan tray module with three individual fans (see Figure 4-6). The fans in the fan tray module provide 1:N redundancy. Two of the three fans in the fan module are required to cool the system and to keep it operational.

Note If more than one fan fails, or you want to hot swap the fan tray, it must be replaced within three minutes after it is removed if the system is in operation.

1919

87

Fan traymodule

Power supplymodules


OL-12957-02


Fabric RedundancyThis section describes fabric redundancy in the Cisco SFS 3001 Server Switch.

Cisco SFS 3001 Server Switches can be arranged in a dual chassis redundant configuration (see Figure 4-7).

Figure 4-7 Cisco SFS 3001 Server Switches in a Dual-Chassis Redundant Configuration

For redundancy, IB HCAs can be dual connected to a redundant pair of Cisco SFS 3001 Server Switches. In an IB fabric that includes more than one Cisco SFS 3001 Server Switch, the Subnet Manager manages the task of assigning a new master in the event of a switch failure. For more about Subnet Managers, see Chapter 5, “Subnet Manager Redundancy.”

1919

94

SFS 3001-1 SFS 3001-2

Ethernet Fabricor SAN Fabric

IB Fabric

InfiniBandHost


OL-12957-02

CisOL-12957-02

C H A P T E R 5
Subnet Manager Redundancy
This chapter describes Subnet Manager redundancy and includes the following sections:

• Embedded Subnet Manager

• High-Performance Subnet Manager

• Setting up Master and Standby Subnet Managers

• Setting Up Database Synchronization


Cisco Subnet Managers support redundancy as described in the IB specifications. There is a master Subnet Manager and there are one or more standby Subnet Managers. In the event that something happens to the master Subnet Manager, the next-in-line standby Subnet Manager assumes control of the IB fabric.

There are two types of Cisco Subnet Managers. They are as follows:

• Embedded Subnet Manager

• High-Performance Subnet Manager

The Embedded Subnet Manager runs on a chassis, and the High-Performance Subnet Manager runs on hosts. Both types of Subnet Managers support the IB standard master/standby failover between each other.

The Cisco Subnet Managers have a proprietary database synchronization protocol that synchronizes important data between the master Subnet Manager and one or more standby Subnet Managers. This provides high-availability redundancy, enabling a database synchronized standby Subnet Manager to assume control as master without disrupting the IB fabric.

Note For redundancy, we recommend that you either have two Embedded Subnet Managers operating together or two High-Performance Subnet Managers operating together. Database-synchronization is not supported between the Embedded Subnet Manager and the High-Performance Subnet Manager, so configuring an Embedded Subnet Manager and a High-Performance Subnet Manager could cause disruption in data traffic in the case of a failure. For more information about database synchronization, see the “Setting Up Database Synchronization” section on page 5-8.


Chapter 5 Subnet Manager Redundancy

Figure 5-1 shows the IB fabric with the Embedded Subnet Manager and Figure 5-2 shows the IB fabric with the High-Performance Subnet Manager. Figure 5-1 and Figure 5-2 show typical Subnet Manager configurations. The user can set up different configurations as required for use.

Figure 5-1 InfiniBand Fabric and Embedded Subnet Manager

1828

53


Master EmbeddedSubnet Manager

Standby EmbeddedSubnet Manager

SFS 3504-1

EthernetFabric

SAN Fabric

SFS 3504-2

SFS 7008P-1 SFS 7008P-2

InfiniBandHost-1

InfiniBandHost-2

InfiniBandHost-3

InfiniBandHost-4


OL-12957-02

Chapter 5 Subnet Manager RedundancyEmbedded Subnet Manager

Figure 5-2 InfiniBand Fabric and High-Performance Subnet Manager

For more information about High-Performance Subnet Managers, see the Cisco High-Performance Subnet Manager for InfiniBand Server Switches.

Embedded Subnet ManagerThis section describes the Embedded Subnet Manager.

The Embedded Subnet Manager operates on the Cisco SFS 3504, the Cisco SFS 3000 series, the Cisco SFS 7000 series, and the Cisco SFS 7008P Server Switches. When deployed in pairs, the Embedded Subnet Manager prevents single points of failure at the system level. The Embedded Subnet Manager is recommended for use in subnets of up to 1,000 nodes, when available. With the Embedded Subnet Manager, a user can detect changes in large subnets within a short duration.

Note The Cisco SFS 3012R Server Switch and the Cisco SFS 7008P Server Switch have a Subnet Manager running on each controller card. There are two Subnet Managers in every chassis.

EthernetFabric

1828

54

MasterHigh-

PerformanceSubnet

Manager


SFS 7008P-1 SFS 7008P-2

StandbyHigh-

PerformanceSubnet

Manager

SAN Fabric

SFS 3504-1 SFS 3504-2

InfiniBandHost-4

InfiniBandHost-1

InfiniBandHost-2

InfiniBandHost-3


OL-12957-02

Chapter 5 Subnet Manager RedundancyHigh-Performance Subnet Manager

High-Performance Subnet ManagerThis section describes the High-Performance Subnet Manager.

The High-Performance Subnet Manager is a standalone software package that centrally manages and controls an IB subnet. It provides high availability when configured in an N+1 configuration and provides high performance and scalability to large clusters as well.

For fabrics containing over 1,000 nodes, we recommend that you use the High-Performance Subnet Manager. Although the Embedded Subnet Manager operates on the Cisco SFS 3504, the Cisco SFS 3000 series, the Cisco SFS 7000 series, and the Cisco SFS 7008P Server Switches, due to larger memory capacity and faster CPU performance the High-Performance Subnet Manager scales more effectively in larger fabrics.

The Cisco High-Performance Subnet Manager complements the Embedded Subnet Manager by off loading the Subnet Manager function from the embedded processors on the IB switches.

The High-Performance Subnet Manager is also required in networking configurations of IB fabrics where chassis types do not contain Embedded Subnet Managers.

Note The Subnet Manager maintains optimal routing decisions. When a switch fails, the Subnet Manager is notified through an in-band trap mechanism, and it resets by re-running the routing calculation for subnets and reprogramming the routes.

Setting up Master and Standby Subnet ManagersThis section describes how to set up the master and standby Subnet Managers and includes the following topics:

• Setting up Master and Standby Subnet Managers Using Embedded Subnet Managers

• Setting up Master and Standby Subnet Managers with High-Performance Subnet Managers

Note In the following sections, values are provided as examples only. We do not recommend that you use non-default values.

Setting up Master and Standby Subnet Managers Using Embedded Subnet Managers

This section describes how to set up master and standby Subnet Managers using the Embedded Subnet Manager.

Typically Embedded Subnet Managers are set up with two chassis in the IB fabric and should be disabled on all other chassis. The priority number of the Subnet Manager determines which is the master. We recommend that you keep the priority of all Subnet Managers in a network equal, to ensure that when a new Subnet Manager is added to the network, it does not take over mastership of the network.


OL-12957-02

Chapter 5 Subnet Manager RedundancySetting up Master and Standby Subnet Managers

To configure and verify the priority between Subnet Managers using CLI commands, perform the following steps:

Step 1 Configure the priority between Subnet Managers to determine which is the master Subnet Manager and which is the standby Subnet Manager.

Note The priority range is between 0 and 15. The Subnet Manager that is the master, is the one that is assigned the highest number in this range. The default priority number is 10. The High-Performance Subnet Manager is assigned a higher priority than the Embedded Subnet Manager to ensure that the High-Performance Subnet Manager takes preference over the Embedded Subnet Manager if they are both present in the IB fabric at the same time.

The following example shows how to configure priority between Embedded Subnet Managers in two Cisco SFS 3504 Server Switches:

SFS-3504# configSFS-3504(config)# ib sm subnet-prefix fe:80:00:00:00:00:00:00 priority 12SFS-3504(config)# exit

Step 2 Configure the master-poll-interval to set the time interval at which the master Subnet Manager is polled to see whether it is active.

The following example shows how to configure the master-poll-interval to 5:

SFS-3504(config)# ib sm subnet-prefix fe:80:00:00:00:00:00:00 master-poll-intval 5

Note Decrease the master-poll-interval value to hasten the standby Subnet Manager takeover and increase the value to slow it.

Step 3 Configure the master-poll-retries to set the number of times it polls the master.

The following example shows how to configure the master-poll-retries to 0:

SFS-3504(config)# ib sm subnet-prefix fe:80:00:00:00:00:00:00 master-poll-retries 0

Note Decrease the master-poll-retries value to hasten the standby Subnet Manager takeover and increase the value to slow it.

Step 4 Verify the configuration.

The following example shows how to verify the Subnet Manager configuration, the master-poll-interval value, and the master-poll-retries value:

SFS-3504# show ib sm configuration subnet-prefix all================================================================================ Subnet Manager Information================================================================================ subnet-prefix : fe:80:00:00:00:00:00:00 guid : 00:05:ad:00:00:01:0c:19 priority : 12 sm-key : 00:00:00:00:00:00:00:00 oper-status : master act-count : 12938 sweep-interval(sec) : 10 response-timeout(msec) : 200 master-poll-intval(sec) : 5


OL-12957-02


master-poll-retries : 0 max-active-sms : 0 LID-mask-control : 0 switch-life-time : 18 switch-hoq-life-time : 18 host-hoq-life-time : 18 max-hops : 64 mad-retries : 5 node-timeout(sec) : 10 wait-report-response : false sa-mad-queue-depth : 256 qos-admin-state : disabled max-operational-v1 : auto-link min-vl-cap-detected : vl0-vl7

The output in this instance verifies that the operational status of this Subnet Manager is that of master, the master-poll-interval is 5 seconds, and the master-poll-retries is 0.

Step 5 Verify that the Subnet Managers are present in the IB fabric.

The following is sample output from the show ib sm sm-info subnet-prefix command and shows how to verify that the Subnet managers are present in the IB fabric:

SFS-3504# show ib sm sm-info subnet-prefix fe:80:00:00:00:00:00:00

================================================================================ Discovered Subnet Managers in Fabric================================================================================ subnet-prefix : fe:80:00:00:00:00:00:00 port-guid : 00:05:ad:00:00:01:1d:20 priority : 0 sm-state : standby sm-key : 00:00:00:00:00:00:00:00 act-count : 219

The sm-state in this instance shows that this standby Subnet Manager is discovered in the fabric.

Setting up Master and Standby Subnet Managers with High-Performance Subnet Managers

This section describes how to set up Master and Standby Subnet Managers using the High-Performance Subnet Managers.

Typically High-Performance Subnet Managers are set up with two hosts in the IB fabric and should be disabled on all other chassis. The priority number of the High-Performance Subnet Manager determines which is the master. We recommend that you keep the priority of all Subnet Managers in a network equal, to ensure that when a new Subnet Manager is added to the network, it does not take over mastership of the network.

To configure and verify the priority between Subnet Managers using CLI commands, perform the following steps:

Step 1 Configure the priority between the High-Performance Subnet Managers to determine which is the master Subnet Manager, and which is the standby Subnet Manager.


OL-12957-02


Note The priority range is between 0 and 15. The Subnet Manager that is the master, is the one that is assigned the highest number in this range. The default priority number is 11. The High-Performance Subnet Manager is assigned a higher priority than the Embedded Subnet Manager to ensure that the High-Performance Subnet Manager takes preference over the Embedded Subnet Manager if they are both present in the IB fabric at the same time.

The following example shows how to configure priority between High-Performance Subnet Managers in two hosts:

ib_sm> config priority 12

Step 2 Configure the master-poll-interval to set the time interval at which the master Subnet Manager is polled to see whether it is active.

The following example shows how to configure the master-poll-interval to 5:

ib_sm> config master-poll-interval 5

Step 3 Configure the master-poll-retries to set the number of times it polls the master.

The following example shows how to configure the master-poll-retries to 0:

ib_sm> config master-poll-retries 0

Step 4 Verify the configuration.

The following is sample output from the show config command and shows how to verify the Subnet Manager configuration, the master-poll-interval value, and the master-poll-retries value:

ib_sm> show config

================================================================================ Subnet Manager Configuration================================================================================ subnet-prefix : fe:80:00:00:00:00:00:00 guid : 00:05:ad:00:00:01:0c:19 priority : 12 sm-key : 00:00:00:00:00:00:00:00 oper-status : master act-count : 2923 sweep-interval(sec) : 10 response-timeout(msec) : 200 mad-retries : 5 node-timeout : 10 master-poll-interval(sec) : 5 master-poll-retries : 0 max-active-sms : 0 LID-mask-control : 0 switch-life-time : 18 sw-link-hoqlife : 18 ca-link-hoqlife : 18 max-hops : 64 wait-report-response : false sa-mad-queue-depth : 256 local-node-retries : 10 qos-admin-state : disabled max-operational-vl : default min-vl-cap-detected : vl0-vl7ib_sm>

The output in this instance verifies that the operational status of this Subnet Manager is that of master, the master-poll-interval is 5 seconds, and the master-poll-retries is 0.


OL-12957-02

Chapter 5 Subnet Manager RedundancySetting Up Database Synchronization

Step 5 Verify the High-Performance Subnet Managers on the IB fabric.

The following is sample output from the show other-sm command and shows how to verify the Subnet Managers on the IB fabric:

ib_sm> show other-sm

================================================================================ Subnet Managers in the subnet================================================================================ subnet-prefix : fe:80:00:00:00:00:00:00 port-guid : 00:05:ad:00:00:01:1d:20 sm-key : 00:00:00:00:00:00:00:00 priority : 0 sm-state : standby act-count : 1133

ib_sm>

This instance shows that one standby Subnet Manager is discovered in the fabric.

Setting Up Database SynchronizationThis section describes how to set up database synchronization and includes the following topics:

• Setting up Database Synchronization for Embedded Subnet Managers, page 5-9

• Setting up Database Synchronization for High-Performance Subnet Managers, page 5-10

Cisco Subnet Managers have a proprietary database synchronization protocol that synchronizes important data between the master Subnet Manager and one or more standby Subnet Managers. This provides high-availability redundancy, enabling a database synchronized standby Subnet Manager to take over as master without disrupting the IB fabric. It is critical in large clusters with enterprise-class IB fabrics and where MTBF is required to be minimal.

Note We recommend that you keep the priority of all Subnet Managers in a network equal. Thus, when a new Subnet Manager is added to the network, it enters as a standby and synchronizes itself to the master.


OL-12957-02


Setting up Database Synchronization for Embedded Subnet ManagersTo set up database synchronization configurations for the Embedded Subnet Manager, perform the following steps:

Step 1 Verify that database synchronization is enabled (enable : true), and view the current configurations.

Note By default the database synchronization feature is enabled.

The following is sample output from the show ib db-sync subnet-prefix command:

SFS-3504# show ib sm db-sync subnet-prefix fe:80:00:00:00:00:00:00

================================================================================ Subnet Manager Database Synchronization Information================================================================================ subnet-prefix : fe:80:00:00:00:00:00:00 enable : true max-dbsync-sms : 1 session-timeout(sec) : 10 poll-interval(sec) : 3 cold-sync-timeout(sec) : 10 cold-sync-limit : 2 cold-sync-period(sec) : 900 new-session-delay(sec) : 120 resync-interval(sec) : 3600 state : in-syncSFS-3504#

Step 2 (Optional) Configure max-dbsync-sms to set the maximum number of standby Subnet Managers with which the master Subnet Manager database can synchronize.

The following example shows how to set the max-dbsync-sms to 2:

SFS-3504(config)# ib sm db-sync subnet-prefix fe:80:00:00:00:00:00:00 max-dbsync-sms 2

Other values displayed in Step 1 under the Subnet Manager Database Synchronization Information can similarly be configured by the user.


OL-12957-02


Step 3 Verify that the Subnet Managers are synchronized.

The following is sample output from the show ib sm db-sync subnet-prefix command and shows how to view the standby Subnet Managers:

SFS-3504# show ib sm db-sync subnet-prefix fe:80:00:00:00:00:00:00 sm-list

================================================================================ DB Synchronizing SMs================================================================================ subnet-prefix : fe:80:00:00:00:00:00:00 port-guid : 00:05:ad:00:00:01:1d:20 entry-state : active session-state : active session-timeout-current(sec) : 8 poll-interval-current(sec) : 1new-session-delay-current(sec) : 120 resync-interval-current(sec) : 3589 state : in-sync

SFS-3504#

The display verifies that there is one standby Subnet Manager as listed.

Setting up Database Synchronization for High-Performance Subnet ManagersTo set up database synchronization configurations for the High-Performance Subnet Manager, perform the following steps:

Step 1 Verify that database synchronization is enabled (admin-state : enabled), and see the current configurations.

Note By default the database synchronization feature is enabled.

The following is sample output from the show db-sync command:

ib_sm> show db-sync

================================================================================ DB Sync Configuration and Status================================================================================ protocol-version : 10 admin-state : enabled max-dbsync-sms : 1 session-timeout(sec) : 10 poll-interval(sec) : 3 cold-sync-timeout(sec) : 10 cold-sync-limit : 2 cold-sync-period(sec) : 900 new-session-delay(sec) : 120 resync-interval(sec) : 3600 state : in-syncib_sm>


OL-12957-02


Step 2 (Optional) Configure max-dbsync-sms to set the maximum number of standby Subnet Managers with which the master Subnet Manager database can synchronize.

The following example shows how to set the max-dbsync-sms to 2:

ib_sm> config db-sync max-dbsync-sms 2

Other values displayed in Step 1 under the Database Synchronization Configuration and Status display can similarly be configured by the user.

Step 3 Verify that the Subnet Managers are synchronized.

The following is sample output from the show db-sync sm-list command and shows how to list the standby Subnet Managers:

ib_sm> show db-sync sm-list================================================================================ DB Synchronizing SMs================================================================================ port-guid : 00:05:ad:00:00:01:1d:20 entry-state : active session-state : active session-timeout-current(sec) : 8 poll-interval-current(sec) : 1new-session-delay-current(sec) : 120 resync-interval-current(sec) : 3373 state : in-sync

ib_sm>

The display verifies that there is one standby Subnet Manager as listed.


OL-12957-02



OL-12957-02

CisOL-12957-02

C H A P T E R 6
Host Redundancy, and IPoIB and SRP Redundancies
This chapter describes host redundancy, IPoIB redundancy, and SRP redundancy and includes the following sections:

• HCA Redundancy, page 6-1

• IPoIB High Availability, page 6-4

• OFED SRP High Availability, page 6-8

IPoIB and SRP are drivers that currently support redundancy.


HCA RedundancyThis section describes HCA redundancy and includes the following topics:

• Single HCA Redundancy, page 6-1

• Multiple HCA Redundancy, page 6-3

• Two HCAs with the IBM BladeCenter, page 6-3

Single HCA RedundancyThis section describes how a single HCA may be configured to provide redundancy.

Single HCAs can each have two ports for redundancy within a single unit. Because such HCAs contain two ports, port-to-port redundancy can be achieved with a single, dual-port HCA (see Figure 6-1). In such cases, the HCA hardware and software drivers handle failovers between ports on the same HCA.


Chapter 6 Host Redundancy, and IPoIB and SRP RedundanciesHCA Redundancy

See the Cisco InfiniBand Host Channel Adapter Hardware Installation Guide for further details about your HCA hardware installation.

Figure 6-1 Single HCA Redundancy with Dual Ports

SFS 7000D-1 SFS 7000D-2

InfiniBand Host

1828

55



OL-12957-02

Chapter 6 Host Redundancy, and IPoIB and SRP RedundanciesHCA Redundancy

Multiple HCA RedundancyThis section describes how multiple HCAs may be configured to provide redundancy.

Multiple HCAs can be installed in a single host. This enables network traffic to failover from one HCA to another HCA. For example, redundancy can be provided with two HCAs serving one host. Installing two HCAs in one host is the minimum recommended configuration for a redundant IB fabric (see Figure 6-2). Such a configuration provides an extra level of redundancy at the host level.

Figure 6-2 Two HCAs in a Single Host for Redundancy

Two HCAs with the IBM BladeCenter For a description about this redundant configuration, see Chapter 3, “InfiniBand Server Switch Module Redundancy for the IBM BladeCenter”.

SFS 7000D-1 SFS 7000D-2

Port 1 Port 2

InfiniBand Host

1832

67


HCA-1 HCA-2


OL-12957-02

Chapter 6 Host Redundancy, and IPoIB and SRP RedundanciesIPoIB High Availability

IPoIB High AvailabilityThis section describes IPoIB high availability and includes the following topics:

• Cisco SFS IPoIB High Availability, page 6-4

• OFED IPoIB High Availability, page 6-6

Note Every host that complies with the RFC-4391 IPoIB specification can use Ethernet gateway redundancies. For more information about Ethernet gateway redundancies, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies.”

Cisco SFS IPoIB High AvailabilityThis section describes IPoIB high availability and includes the following topics:

• Merging Physical Ports

• Unmerging Physical Ports

IPoIB supports active/passive port failover high availability between two or more ports. When you enable the high availability feature, the ports on the HCA (for example, ib0 and ib1) merge into one virtual port. If you configure high availability between the ports on the HCA(s), only one of the physical ports passes traffic. The other ports are used as standby in the event of a failure.

For more details about the Cisco SFS host drivers, see the Cisco SFS InfiniBand Host Drivers User Guide for Linux.

Merging Physical Ports

To configure IPoIB high availability on HCA ports in a Linux host, perform the following steps:

Step 1 Log in to your Linux host.

Step 2 Display the available interfaces by entering the ipoibcfg list command. The following example shows how to configure IPoIB high availability between two ports on one HCA.

The following example shows how to display the available interfaces:

host1# /usr/local/topspin/sbin/ipoibcfg listib0 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/1, Active: InfiniHost0/1)ib1 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/2, Active: InfiniHost0/2)

Step 3 Take the interfaces offline.

Note You cannot merge interfaces until you take them offline.

The following example shows how to take the interfaces offline:

host1# ifconfig ib0 downhost1# ifconfig ib1 down


OL-12957-02


Step 4 Merge the two ports into one virtual IPoIB high availability port by entering the ipoibcfg merge command with the IB identifiers of the first and the second IB ports on the HCA.

The following example shows how to merge the two ports into one virtual IPoIB high availability port:

host1# /usr/local/topspin/sbin/ipoibcfg merge ib0 ib1

Step 5 Display the available interfaces by entering the ipoibcfg list command.


host1# /usr/local/topspin/sbin/ipoibcfg listib0 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/1, Active: InfiniHost0/1)

Note The ib1 interface no longer appears, as it is merged with ib0.

Step 6 Enable the interface by entering the ifconfig command with the appropriate port identifier ib# argument and the up keyword.

The following example shows how to enable the interface with the ifconfig command:

host1# ifconfig ib0 up

Step 7 Assign an IP address to the merged port just as you would assign an IP address to a standard interface.

Unmerging Physical Ports

To unmerge physical ports and disable active-passive IPoIB high availability, perform the following steps:

Step 1 Disable the IPoIB high availability interface that you want to unmerge by entering the ifconfig command with the appropriate IB interface argument and the down argument.

The following example shows how to unmerge by disabling the IPoIB high availability interface:

host1# ifconfig ib0 down

Step 2 Unmerge the port by entering the ipoibcfg unmerge command with the identifier of the port that you want to unmerge.

The following example shows how to unmerge the port:

host1# /usr/local/topspin/sbin/ipoibcfg unmerge ib0 ib1

Note After unmerging the port, ib1 no longer has an IP address and must be configured.

Step 3 Display the available interfaces by entering the ipoibcfg list command.


host1# /usr/local/topspin/sbin/ipoibcfg listib0 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/1, Active: InfiniHost0/1)ib1 (P_Key 0xffff) (SL:255) (Ports: InfiniHost0/2, Active: InfiniHost0/2)


OL-12957-02


Step 4 Enable the interfaces by entering the ifconfig command with the appropriate IB interface argument and the up argument.

The following example shows how to enable the interfaces:

host1# ifconfig ib0 up

OFED IPoIB High AvailabilityThis section describes the OFED IPoIB high availability and includes the following topics:

• Configuring IPoIB High Availability

• Verifying IPoIB High Availability

This section describes IPoIB high availability. IPoIB supports active/passive port failover high availability between two or more ports. When you enable the high availability feature, the ports on the HCA(s) (such as ib0 and ib1) bond into one virtual port. If you configure high availability between the ports on the HCA(s), only one of the physical ports passes traffic. The other ports are used as standby in the event of a failure.

IPoIB high availability is implemented through the IPoIB bonding driver. This driver is based on the Linux Ethernet bonding driver and has been changed to work with IPoIB. The ib-bonding package contains the bonding driver and a utility named ib-bond to manage and control the driver operation.

For more details about OFED host drivers, see the Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux.

Configuring IPoIB High Availability

To configure IPoIB high availability, perform the following steps:

Step 1 Remove the existing IP addresses from the interfaces.

The IP address from ib0 will be reassigned to the bonding interface.

The following example shows how to remove the existing IP addresses:

host1# ifconfig ib0 0.0.0.0host1# ifconfig ib1 0.0.0.0

Step 2 Bond the two ports into one virtual IPoIB high availability port by using the ib-bond command.

The following example shows how to bond two ports into one virtual IPoIB high availability port in verbose mode:

host1# ib-bond --bond-ip 192.168.0.1/24 --slaves ib0,ib1 -venslaving ib0enslaving ib1bonding is up: 192.168.0.1bond0: 80:00:04:04:fe:80:00:00:00:00:00:00:00:05:ad:02:00:23:f0:d0 192.168.0.1/24slave0: ib0 *slave1: ib1

In the preceding output, ib0 * indicates that ib0 is the active interface, and ib1 is the passive interface. Partition interfaces such as ib0.8002 can also be used with IPoIB high availability. In addition, /24 is the subnet mask.


OL-12957-02


Step 3 (Optional) Enter the ifconfig command.

The following example shows how to enter the ifconfig command:

host1# ifconfig bond0bond0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::205:ad00:20:849/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1 RX packets:33523452 errors:0 dropped:0 overruns:0 frame:0 TX packets:165408699 errors:2 dropped:3 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:175570845580 (163.5 GiB) TX bytes:619840713192 (577.2 GiB)

The IPoIB high availability status information can be printed at any time with the ib-bond --status-all command.

The following example shows how to print the IPoIB high availability status information:

host1# ib-bond --status-allbond0: 80:00:04:04:fe:80:00:00:00:00:00:00:00:05:ad:02:00:23:f0:d0 192.168.0.1/24slave0: ib0 *slave1: ib1

The IPoIB high availability configuration can be removed with the ib-bond --stop-all command.

The following example shows how to remove the IPoIB high availability configuration:

host1# ib-bond --stop-all

IPoIB high availability interfaces that are configured manually are not persistent across reboots. You must use the configuration file /etc/infiniband/openib.conf to configure IPoIB high availability when the host boots. You must also remove any existing IPoIB boot-time configuration files such as ifcfg-ib0.

The following example shows the portion of openib.conf that must be edited to configure IPoIB high availability at boot time:

# Enable the bonding driver on startupIPOIBBOND_ENABLE=yes# Set bond interface namesIPOIB_BONDS=bond0# Set specific bond params; address and slavesbond0_IP=192.168.0.1/24bond0_SLAVES=ib0,ib1

The drivers can be restarted for the change to take effect without rebooting.

The following example shows how the drivers can be restarted:

host1# /etc/init.d/openibd restartUnloading HCA driver: [ OK ]Loading HCA driver and Access Layer: [ OK ]Setting up InfiniBand network interfaces:No configuration found for ib0No configuration found for ib1Setting up service network . . . [ done ]Setting up bonding interfaces:Bringing up interface bond0 [ OK ]


OL-12957-02

Chapter 6 Host Redundancy, and IPoIB and SRP RedundanciesOFED SRP High Availability

Verifying IPoIB High Availability

To force an IPoIB high availability failover while IPoIB traffic is running, perform the following steps:

Step 1 Start ping or Netperf between two IPoIB hosts.

For more details about how to start ping or Netperf between two IPoIB hosts, see the Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux.

Step 2 Remove the cable connected to ib0, either manually or by using the OS CLI or GUI. Print the ib-bond --status-all command to verify the IPoIB high availability status.

The following example shows how to print the IPoIB high availability status:

host1# ib-bond --status-allbond0: 80:00:04:04:fe:80:00:00:00:00:00:00:00:05:ad:02:00:23:f0:d0 192.168.0.1/24slave0: ib0slave1: ib1 *

The ib-bond --status-all command prints an asterix next to the primary interface. The primary interface has now switched to ib1 as shown in the preceding example.

A kernel sys log message is also printed every time there is a failover.

host1# dmesgbonding: bond0: link status definitely down for interface ib0, disabling itbonding: bond0: making interface ib1 the new active one.bonding: send gratuitous arp: bond bond0 slave ib1

Step 3 Verify that the ping or Netperf continues with little or no interruption.

Note The Element Manager GUI can also be used to display port statistics, which is useful for watching a port failover. For more information about the Element Manager GUI, see the Cisco SFS Product Family Element Manager User Guide.

OFED SRP High AvailabilityThis section describes how to configure SRP for use with Device Mapper Multipath, which is included with both RHEL and SLES.

Device Mapper Multipath supports both active/active (load balancing and failover) and active/passive (failover only) high availability, depending on the capability of the storage device. SRP should always be used with multipathing software for high availability, to prevent data corruption and data loss. Other third-party multipathing software can also be used with SRP, for configuration information. Consult the relevant documentation for that software.

Device Mapper Multipath allows hosts to route I/O over the multiple paths available to an end storage unit. A path refers to the connection from a host IB port to a storage controller port. When an active path through which I/O happens fails, Device Mapper Multipath reroutes the I/O over other available paths. In a Linux host, when there are multiple paths to a storage controller, each path appears as a separate block device and hence results in multiple block devices for a single LUN. Device Mapper Multipath creates a new multipath block device for those devices having the same LUN WWN. For example, a host with two IB ports attached to a Cisco SFS 3012R Server Switch with two Fibre Channel port(s) attached


OL-12957-02


to a storage controller, sees two block devices: /dev/sda and /dev/sdb, for example. Device Mapper Multipath creates a single block device, /dev/mapper/360003ba27cf53000429f82b300016652, that reroutes I/O through those two underlying block devices.

Device Mapper Multipath includes the following software components:

• dm-multipath kernel module — routes I/O and does failover to paths

• multipath configuration tool — provides commands to configure, list, and flush multipath devices

• multipathd daemon — monitors paths to check if paths have failed or been fixed

Independent of storage device high availability capability, the Device Mapper Multipath provides active/active high availability on host IB ports. The Cisco SFS Fibre Channel gateway similarly provides active/active high availability between the SFS chassis and the Fibre Channel fabric.

To configure SRP high availability with Device Mapper Multipath, perform the following steps:

Step 1 Edit the file /etc/infiniband/openib.conf. Change SRPHA_ENABLE=no to SRPHA_ENABLE=yes. This starts the srp_daemon program at boot time to create block devices for all paths to the SRP storage.

The srp_daemon program also handles dynamic storage reconfiguration, such as new storage being added after the host is booted.

Note Both SRP_LOAD and SRPHA_ENABLE must be set to yes for SRP high availability to function correctly.

Step 2 (Optional) Edit the file /etc/srp_daemon.conf to restrict SRP host driver access to a subset of available SRP targets. By default, srp_daemon configures block devices for all SRP targets. The default /etc/srp_daemon.conf file contains this information.

Note For more details about the srp_daemon.conf file, see the Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux.

Step 3 Edit the file /etc/multipath.conf. On RHEL4, the devnode_blacklist section (blacklist on RHEL5) should be removed, commented out, or modified.

The following example shows the section of the file to be edited:

devnode_blacklist { devnode "*"}

(Optional) Change user_friendly_names yes to user_friendly_names no in /etc/multipath.conf on RHEL as well. The friendly names are not consistent between different hosts and operating systems.

Note On SLES, /etc/multipath.conf does not exist by default, and devnode_blacklist is not in effect.

On both RHEL and SLES, additional storage-specific configuration information may be required in /etc/multipath.conf. Consult your storage device documentation for more details. For more information on multipath.conf, consult the device-mapper-multipath package (RHEL) or multipath-tools (SLES). Both packages have well-documented sample multipath.conf example files in /usr/share/doc.


OL-12957-02


Step 4 Configure Device Mapper Multipath to start at boot time.

The following example shows the command to enter on RHEL:

host1# chkconfig multipathd on

The following example shows the commands to enter on SLES:

host1# chkconfig boot.multipath onhost1# chkconfig multipathd on

Step 5 Reboot the Linux host.

After the reboot, multipath SRP devices should be accessible in /dev/mapper. Depending on the configuration, it may take a few minutes after reboot for /dev/mapper to be fully populated.

The following example shows the output for the Fibre Channel gateway configuration when the IB host has one HCA with both IB ports connected to the Server Fabric Switch:

host1# ls /dev/mapper3600c0ff00000000007a6d11b6f245e003600c0ff00000000007a6d11b6f245e00p13600c0ff00000000007a6d11b6f245e00p23600c0ff00000000007a6d11b6f245e013600c0ff00000000007a6d11b6f245e023600c0ff00000000007a6d11b6f245e02p13600c0ff00000000007a6d11b6f245e02p23600c0ff00000000007a6d11b6f245e02p33600c0ff00000000007a6d11b6f245e033600c0ff00000000007a6d11b6f245e03p13600c0ff00000000007a6d11b6f245e03p23600c0ff00000000007a6d11b6f245e043600c0ff00000000007a6d11b6f245e053600c0ff00000000007a6d11b6f245e063600c0ff00000000007a6d11b6f245e06p1control

Step 6 View the SCSI devices.

The following example shows how to view the SCSI devices:

host1# lsscsi[0:0:0:0] disk IBM-ESXS MAY2036RC T107 /dev/sda[1:0:0:0] disk SUN StorEdge 3510 327P /dev/sdb[1:0:0:1] disk SUN StorEdge 3510 327P /dev/sdd[1:0:0:2] disk SUN StorEdge 3510 327P /dev/sdk[1:0:0:3] disk SUN StorEdge 3510 327P /dev/sdl[1:0:0:4] disk SUN StorEdge 3510 327P /dev/sdm[1:0:0:5] disk SUN StorEdge 3510 327P /dev/sdn[1:0:0:6] disk SUN StorEdge 3510 327P /dev/sdo[2:0:0:0] disk SUN StorEdge 3510 327P /dev/sdc[2:0:0:1] disk SUN StorEdge 3510 327P /dev/sde[2:0:0:2] disk SUN StorEdge 3510 327P /dev/sdf[2:0:0:3] disk SUN StorEdge 3510 327P /dev/sdg[2:0:0:4] disk SUN StorEdge 3510 327P /dev/sdh[2:0:0:5] disk SUN StorEdge 3510 327P /dev/sdi[2:0:0:6] disk SUN StorEdge 3510 327P /dev/sdj

Note The lsscsi command is supported by RHEL5 and SLES10 only. The lsscsi command is not supported by RHEL4.


OL-12957-02


Step 7 List the relationship between the SRP block devices and multipath devices by using the multipath -l command.

The following example shows how to use the multipath -l command:

host1# multipath -l3600c0ff00000000007a6d11b6f245e06dm-13 SUN,StorEdge 3510[size=13G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active] \_ 2:0:0:6 sdj 8:144 [active][undef] \_ 1:0:0:6 sdo 8:224 [active][undef]3600c0ff00000000007a6d11b6f245e05dm-12 SUN,StorEdge 3510[size=15G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active] \_ 2:0:0:5 sdi 8:128 [active][undef] \_ 1:0:0:5 sdn 8:208 [active][undef]3600c0ff00000000007a6d11b6f245e04dm-11 SUN,StorEdge 3510[size=15G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active] \_ 2:0:0:4 sdh 8:112 [active][undef] \_ 1:0:0:4 sdm 8:192 [active][undef]3600c0ff00000000007a6d11b6f245e03dm-3 SUN,StorEdge 3510[size=15G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active] \_ 2:0:0:3 sdg 8:96 [active][undef] \_ 1:0:0:3 sdl 8:176 [active][undef]3600c0ff00000000007a6d11b6f245e02dm-2 SUN,StorEdge 3510[size=15G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active] \_ 2:0:0:2 sdf 8:80 [active][undef] \_ 1:0:0:2 sdk 8:160 [active][undef]3600c0ff00000000007a6d11b6f245e01dm-1 SUN,StorEdge 3510[size=15G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active] \_ 2:0:0:1 sde 8:64 [active][undef] \_ 1:0:0:1 sdd 8:48 [active][undef]3600c0ff00000000007a6d11b6f245e00dm-0 SUN,StorEdge 3510[size=15G][features=0][hwhandler=0]\_ round-robin 0 [prio=0][active] \_ 2:0:0:0 sdc 8:32 [active][undef] \_ 1:0:0:0 sdb 8:16 [active][undef]

In the preceding output, each multipath device corresponds to two SRP block devices, one on each of the two attached host IB ports.


OL-12957-02



OL-12957-02

CisOL-12957-02

C H A P T E R 7
Ethernet Gateway and IPoIB Redundancies
This chapter describes the Ethernet gateway redundancies and includes the following sections:

• Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch, page 7-3

• Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch, page 7-10

• Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch, page 7-23


Ethernet gateway redundancy is based on the concept of redundancy group. Redundancy group is a logical entity bridging an Ethernet VLAN to an IB partition, just like a bridge group, but in redundant fashion. A redundancy group can contain one or more bridge groups located on the same or different gateways. Gateways with bridge group members of the same redundancy group can be in the same or different chassis.

A redundancy group can operate in two modes. The first one, which is the default mode, is active-passive. In active-passive mode, only one bridge group is active and all others are in hot-standby state. In case the active bridge group fails, another bridge group from the same redundancy group is selected and activated. In active-active mode, all bridge groups are active and load balancing is enabled. Load balancing allows the user to distribute the load of IB nodes among all bridge groups of the redundancy group. In case of a bridge group failure, the load of the IB nodes is redistributed among the remaining bridge groups.

Note Even in an active-active mode, only one bridge group forwards the broadcast and multicast traffic. If the bridge group forwarding broadcast and multicast fails, a new bridge group member of the same redundancy group is selected to forward broadcast and multicast. The selection mechanism is identical to the one used in active-passive mode to select the active bridge group.

A logical diagram of gateways in a redundant configuration is shown in Figure 7-1. Two or more Ethernet gateways are configured to bridge between the Ethernet fabric and the IB fabric. The gateways can be in the same or different chassis. The redundancy manager monitors the health of the gateways and in case of failure elects a new primary bridge group. The redundancy manager is part of the SFS OS and runs on the controller card of each chassis.


Chapter 7 Ethernet Gateway and IPoIB Redundancies

Figure 7-1 Logical Diagram of Ethernet Gateways in a Redundant Configuration

Different topologies work in similar ways but provide different levels of redundancy. For example, a redundancy group with two bridge groups in the same chassis but different gateways protects against gateway failure but does not provide chassis redundancy. Alternatively, a redundancy group with two bridge groups on gateways in different chassis provides the same level of gateway redundancy but also provides chassis redundancy. Examples are provided later in this chapter.

When a bridge group becomes a member of a redundancy group, some parameters are overwritten with the corresponding parameters from the redundancy group. These parameters are restored when the bridge group is removed from the redundancy group. Some examples of such parameters are broadcast and multicast forwarding.

The examples in the sections that follow show the most popular deployment of different types of I/O chassis. To simplify configuration, only one IP subnet is bridged (Data IP subnet). The Ethernet ports are not VLAN tagged. The configuration on the Ethernet switch connected to the gateways determines which VLAN is bridged. This VLAN is mapped to the default IB partition using the Ethernet gateway. One IP subnet is allocated for in-band IB management (Management IP subnet). The in-band IB management interface must be configured in order for the redundancy to work. This is true even in a single chassis configuration.

Each bridge group must have an IP address assigned from the data subnet for the redundancy with load-balancing (active-active mode) to work. The following addresses are assigned for the purpose of the examples in this chapter:

Data Subnet: 10.0.0.0/8

Default Gateway (on the Ethernet switch): 10.0.0.1

IB Management Subnet: 192.168.0.0/8

Note Every host that complies with the RFC-4391 IPoIB specification, can use Ethernet gateway redundancies. For more information on IPoIB redundancies, see Chapter 6, “Host Redundancy, and IPoIB and SRP Redundancies.”Ethernet gateway uses gratuitous ARPs to redirect traffic from one gateway to another during failover, fail-back, new member join/leave events and such. Because gratuitous ARPs are not guaranteed to reach all network nodes, some ARP entries may become out of sync for a period of time up to the ARP cache timeout. That is why ARP cache timeout on all network nodes must be set to the maximum acceptable outage time

Ethernet Gateway Ethernet Gateway

Ethernet Fabric

InfiniBand Fabric

1826

41

Additional Ethernetgateways areadded as required


OL-12957-02

Chapter 7 Ethernet Gateway and IPoIB RedundanciesConfiguring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch

Configuring Ethernet Gateway Redundancy with the Cisco SFS 3504 Server Switch

This section describes how to configure redundant Ethernet gateways with the Cisco SFS 3504 Server Switch to provide high availability redundancy and includes the following topics:

• Verifying Redundancy Configuration for Cisco SFS 3504 Server Switches, page 7-6

• Verifying Bridge Group Configuration for Cisco SFS 3504 Server Switches, page 7-8

Two or more Cisco SFS 3504 Server Switches must be used to provide high availability redundancy. A very typical deployment would consist of two Cisco SFS 3504 Server Switches with two Ethernet gateways in each chassis (see Figure 7-2). The switches are configured with one bridge group for every gateway and each bridge group having six Ethernet ports aggregated in single link aggregation groups (trunk). All four bridge groups are in a single redundancy group.

Data subnet: 10.0.0.0/8

Management subnet: 192.168.0.0/24

Figure 7-2 Ethernet Gateway Redundancy with Dual Cisco SFS 3504 Server Switches

Note For the purpose of the examples in the following sections, the Ethernet gateways are in slots numbers 1 and 2 for both Cisco SFS 3504 chassis.

To configure the first Cisco SFS 3504 Server Switch, perform the following steps:

Step 1 Enter configuration mode.

The following example shows how to enter configuration mode:

SFS-3504-1> enableSFS-3504-1# configure terminal

Ethernet Gateway

InfiniBand Switch

Ethernet Gateway Ethernet Gateway

InfiniBand Switch

Ethernet Gateway

SFS 3504-1 SFS 3504-2

Ethernet Fabric

InfiniBand Fabric18

2642


OL-12957-02


Step 2 Configure and connect an IB in-band management interface.

The IP address must be unique on each chassis.

Note If an out-of-band Ethernet interface on the controller card is also configured, it must be on a different IP subnet.

The following example shows how to configure an IB management interface:

SFS-3504-1(config)# interface mgmt-ib SFS-3504-1(config)# ipaddress 192.168.0.1 255.255.255.0SFS-3504-1(config-if-mgmt-ib)# no shutdownSFS-3504-1(config-if-mgmt-ib)# exitSFS-3504-1(config)#

Step 3 Create and configure two link aggregation groups (trunks) by assigning Ethernet ports to the link aggregation groups.

The following example shows how to create and configure link aggregation groups by assigning Ethernet ports:

SFS-3504-1(config)# interface trunk 1SFS-3504-1(config-if-trunk)# enableSFS-3504-1(config-if-trunk)# distribution-type src-dst-ipSFS-3504-1(config-if-trunk)# interface ethernet 1/1-1/6SFS-3504-1(config-if-ether-1/1-1/6)# trunk-group 1SFS-3504-1(config-if-ether-1/1-1/6)# exitSFS-3504-1(config)# interface trunk 2SFS-3504-1(config-if-trunk)# enableSFS-3504-1(config-if-trunk)# distribution-type src-dst-ipSFS-3504-1(config-if-trunk)# interface ethernet 2/1-2/6SFS-3504-1(config-if-ether-2/1-2/6)# trunk-group 2SFS-3504-1(config-if-ether-2/1-2/6)# exitSFS-3504-1(config)#

Step 4 Configure two bridge groups and assign ports to them.

Note IP addresses must be from the data IP subnet and must be unique for each bridge group.

The following example shows how to configure bridge groups and assign ports to them:

SFS-3504-1(config)# bridge-group 1 subnet-prefix 10.0.0.0 8SFS-3504-1(config)# bridge-group 1 ip-addr 10.0.0.101SFS-3504-1(config)# bridge-group 1 ib-next-hop 10.0.0.1SFS-3504-1(config)# interface trunk 1SFS-3504-1(config-if-trunk)# bridge-group 1SFS-3504-1(config-if-trunk)# interface gateway 1SFS-3504-1(config-if-gw-1/2)# bridge-group 1SFS-3504-1(config-if-gw-1/2)# exitSFS-3504-1(config)# bridge-group 2 subnet-prefix 10.0.0.0 8SFS-3504-1(config)# bridge-group 2 ip-addr 10.0.0.102SFS-3504-1(config)# bridge-group 2 ib-next-hop 10.0.0.1SFS-3504-1(config)# interface trunk 2SFS-3504-1(config-if-trunk)# bridge-group 2SFS-3504-1(config-if-trunk)# interface gateway 2SFS-3504-1(config-if-gw-2/2)# bridge-group 2SFS-3504-1(config-if-gw-2/2)# exitSFS-3504-1(config)#


OL-12957-02


Step 5 Configure a redundancy group and assign both bridge groups to it.

Note The redundancy group ID must be the same in both chassis.

The following example shows how to configure a redundancy group and assign both bridge groups to it:

SFS-3504-1(config)# redundancy-group 1SFS-3504-1(config)# bridge-group 1 redundancy-group 1SFS-3504-1(config)# bridge-group 2 redundancy-group 1

Step 6 (Optional) Enable load balancing between bridge groups.

The following example shows how to enable load balancing:

SFS-3504-1(config)# redundancy-group 1 load-balancingSFS-3504-1(config)# exit

To configure the second Cisco SFS 3504 Server Switch, perform the following steps:



SFS-3504-2> enableSFS-3504-2# configure terminal

Step 2 Configure and connect an IB in-band management interface




SFS-3504-2(config)# interface mgmt-ibSFS-3504-2(config-if-mgmt-ib)# ip address 192.168.0.2 255.255.255.0SFS-3504-2(config-if-mgmt-ib)# no shutdownSFS-3504-2(config-if-mgmt-ib)# exitSFS-3504-2(config)#

Step 3 Create and configure two link aggregation groups (trunks) by assigning Ethernet ports to the link aggregation groups.

The following example shows how to create and configure link aggregation groups:

SFS-3504-2(config)# interface trunk 3SFS-3504-2(config-if-trunk)# enableSFS-3504-2(config-if-trunk)# distribution-type src-dst-ipSFS-3504-2(config-if-trunk)# interface ethernet 1/1-1/6SFS-3504-2(config-if-ether-1/1-1/6)# trunk-group 3SFS-3504-2(config-if-ether-1/1-1/6)# interface trunk 4SFS-3504-2(config-if-trunk)# enableSFS-3504-2(config-if-trunk)# distribution-type src-dst-ipSFS-3504-2(config-if-trunk)# interface ethernet 2/1-2/6SFS-3504-2(config-if-ether-2/1-2/6)# trunk-group 4SFS-3504-2(config-if-ether-2/1-2/6)# exitSFS-3504-2(config)#


OL-12957-02



Note IP addresses must be from the data IP subnet and must be unique for each bridge group.


SFS-3504-2(config)# bridge-group 3 subnet-prefix 10.0.0.0 8SFS-3504-2(config)# bridge-group 3 ip-addr 10.0.0.103SFS-3504-2(config)# bridge-group 3 ib-next-hop 10.0.0.1SFS-3504-2(config)# interface trunk 3SFS-3504-2(config-if-trunk)# bridge-group 3SFS-3504-2(config-if-trunk)# interface gateway 1SFS-3504-2(config-if-gw-1/2)# bridge-group 3SFS-3504-2(config-if-gw-1/2)# exitSFS-3504-2(config)# bridge-group 4 subnet-prefix 10.0.0.0 8SFS-3504-2(config)# bridge-group 4 ip-addr 10.0.0.104SFS-3504-2(config)# bridge-group 4 ib-next-hop 10.0.0.1SFS-3504-2(config)# interface trunk 4SFS-3504-2(config-if-trunk)# bridge-group 4SFS-3504-2(config-if-trunk)# interface gateway 2SFS-3504-2(config-if-gw-2/2)# bridge-group 4SFS-3504-2(config-if-gw-2/2)# exitSFS-3504-2(config)#




SFS-3504-2(config)# redundancy-group 1SFS-3504-2(config)# bridge-group 3 redundancy-group 1SFS-3504-2(config)# bridge-group 4 redundancy-group 1SFS-3504-2(config)#


The following example shows how to enable load balancing:

SFS-3504-2(config)# redundancy-group 1 load-balancingSFS-3504-2(config)# exit

Verifying Redundancy Configuration for Cisco SFS 3504 Server SwitchesTo verify the redundancy group configuration and status, use the show redundancy-group CLI command. This command shows the redundancy group properties and all its members. It is important to make sure the properties match the configuration and all members are reported. Redundancy groups must be checked on both chassis.


OL-12957-02


The following is sample output from show redundancy-group command for the first switch:

SFS-3504-1# show redundancy-group================================================================================ Redundancy Groups================================================================================ rlb-id : 1 name : group-p_key : ff:ff load-balancing : enabled broadcast-forwarding : false directed-broadcast : false multicast : false gratuitous-igmp : false igmp-version : v2 num-members : 4 new-member-force-reelection : false

================================================================================= Redundancy Group Members================================================================================bridge-group src-addr last-receive--------------------------------------------------------------------------------1 192.168.0.1 Sun Jan 4 00:27:31 19702 192.168.0.1 Sun Jan 4 00:27:31 19703 192.168.0.2 Sun Jan 4 00:27:31 19704 192.168.0.2 Sun Jan 4 00:27:31 1970

The following example shows how to use the show redundancy command for the second switch:

SFS-3504-2# show redundancy-group

================================================================================ Redundancy Groups================================================================================ rlb-id : 1 name : group-p_key : ff:ff load-balancing : enabled broadcast-forwarding : false directed-broadcast : false multicast : false gratuitous-igmp : false igmp-version : v2 num-members : 4 new-member-force-reelection : false

================================================================================Redundancy Group Members================================================================================bridge-group src-addr last-receive--------------------------------------------------------------------------------1 192.168.0.1 Sun Jan 4 00:27:12 19702 192.168.0.1 Sun Jan 4 00:27:23 19703 192.168.0.2 Sun Jan 4 00:25:51 19704 192.168.0.2 Sun Jan 4 00:25:54 1970

SFS-3504-2#


OL-12957-02


Verifying Bridge Group Configuration for Cisco SFS 3504 Server SwitchesTo check the bridge group configuration and status, use the show bridge-group CLI command. Check both the Cisco SFS 3504 Server Switches, and make sure all the bridge groups are members of the same redundancy group.

When a bridge group is a member of a redundancy group, most of the properties are inherited from the redundancy group. Also make sure only one bridge group, across all chassis, is primary and the rest are secondary. This is true even when load balancing is enabled (active-active mode). In active-passive mode, only the primary bridge group is forwarding. All others are in hot standby state. In active-active mode, all bridge groups are forwarding unicast traffic and only the primary is forwarding broadcast and multicast if enabled.

The following is sample output from the show bridge-group command and shows how to verify the bridge group configuration for the first switch:

SFS-3504-1# show bridge-group================================================================================ Bridge Groups================================================================================ bridge-group-id : 1 bridge-group-name : ip-addr : 10.0.0.101 eth-bridge-port : trunk 1 (not tagged) ib-bridge-port : 1/2(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group directed-broadcast : false directed-broadcast-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : primary

bridge-group-id : 2 bridge-group-name : ip-addr : 10.0.0.102 eth-bridge-port : trunk 2 (not tagged) ib-bridge-port : 2/2(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group directed-broadcast : false directed-broadcast-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : secondary


OL-12957-02


The following is sample output from the show bridge-group command that shows how to verify the bridge group configuration for the second switch:

SFS-3504-2# show bridge-group

================================================================================ Bridge Groups================================================================================ bridge-group-id: 3 bridge-group-name: I-addr : 10.0.0.103 eth-bridge-port : trunk 3 (not tagged) ib-bridge-port : 1/2(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group directed-broadcast : false directed-broadcast-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : secondary

bridge-group-id : 4 bridge-group-name : ip-addr : 10.0.0.104 eth-bridge-port : trunk 4 (not tagged) ib-bridge-port : 2/2(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group directed-broadcast : false directed-broadcast-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : secondary

SFS-3504#


OL-12957-02

Chapter 7 Ethernet Gateway and IPoIB RedundanciesConfiguring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch

Configuring Ethernet Gateway Redundancy Using the Cisco SFS 3012R Server Switch

This section describes how to configure the Ethernet gateway redundancy using the Cisco SFS 3012R Server Switch and includes the following topics:

• Configuring Ethernet Gateway Redundancy Using a Single Cisco SFS 3012R Server Switch, page 7-10

• Configuring Ethernet Gateway Redundancy Using Dual Cisco SFS 3012R Server Switches, page 7-15

The Cisco SFS 3012R Server Switch is a fully redundant chassis. It has twelve gateway slots, two controller modules, and two switch cards. Each gateway is connected to both switch cards through the backplane. A single Cisco SFS 3012R Server Switch is therefore capable of being configured for high availability.

Configuring Ethernet Gateway Redundancy Using a Single Cisco SFS 3012R Server Switch

This section describes how to configure Ethernet gateway redundancy using a single Cisco SFS 3012R Server Switch and includes the following topics:

• Verifying Redundancy Group Configuration for Cisco SFS 3001 Server Switches, page 7-27


The example in this section shows a typical single-switch configuration with four Ethernet gateways and two switch cards. Two of the gateways, slots 2 and 4, are configured to use switch cards in slot 15 and the other two gateways, slots 3 and 5, are configured to use switch cards in slot 16. (To locate the slot numbers on the Cisco SFS 3012R Server Switch, see Figure 4-4.) Thus if any of the switch cards fail, two gateways continue to remain operational. If the Ethernet ports of the gateways are required to be connected to two different Ethernet switches, connect the gateways to the same IB switch card that are connected to different Ethernet switches. Thus if any one Ethernet switch and any one IB switch card were to fail, at least one gateway continues to remain operational.

Note For the purpose of this example, gateways in slots 2 and 3 must be connected to different Ethernet switches than gateways in slots 4 and 5.

The topology used in this example is shown in Figure 7-3.


OL-12957-02


Figure 7-3 Ethernet Gateway Redundancy with Single Cisco SFS 3012R Server Switch

Once the redundancy groups are configured, a primary bridge is selected and forwarding is enabled.

To configure redundancy in a single Cisco SFS 3012R Server Switch, perform the following steps:



SFS-3012R> enableSFS-3012R# configure terminal

Step 2 Configure and connect an IB in-band management interface

Although there is only one chassis in this configuration, the interface must be configured and connected in order for the redundancy to work.


The following example shows how to configure and connect an IB in-band management interface:

SFS-3012R(config)# interface mgmt-ibSFS-3012R(config-if-mgmt-ib)# 192.168.0.1 255.255.255.0SFS-3012R(config-if-mgmt-ib)# no shutdownSFS-3012R(config-if-mgmt-ib)# exit

Step 3 Create and configure all four link aggregation groups (trunks) by assigning Ethernet ports to each link aggregation group.

The following example shows how to create and configure four link aggregation groups:

SFS-3012R(config)# interface trunk 1SFS-3012R(config-if-trunk)# enableSFS-3012R(config-if-trunk)# distribution-type src-dst-ipSFS-3012R(config-if-trunk)# interface ethernet 2/1-2/6SFS-3012R(config-if-ether-2/1-2/6)# trunk-group 1

Ethernet Gateway 1

InfiniBand Switch 1 InfiniBand Switch 2

Ethernet Gateway 3Ethernet Gateway 2 Ethernet Gateway 4SFS 3012R

Ethernet Fabric

InfiniBand Fabric

1826

39


OL-12957-02




SFS-3012R(config)# interface trunk 4SFS-3012R(config-if-trunk)# enableSFS-3012R(config-if-trunk)# distribution-type src-dst-ipSFS-3012R(config-if-trunk)# interface ethernet 5/1-5/6SFS-3012R(config-if-ether-5/1-5/6)# trunk-group 4SFS-3012R(config-if-ether-5/1-5/6)# exit

Step 4 Configure all four bridge groups and assign ports to them.

Two of the bridge groups use IB port 1 to connect to the switch card in slot 15 and the other two bridge groups use IB port 2 to connect to the switch card in slot 16.

Note The IP address must be from the Data IP subnet and must be unique for each bridge group.


SFS-3012R(config)# bridge-group 1 subnet-prefix 10.0.0.0 8SFS-3012R(config)# bridge-group 1 ip-addr 10.0.0.101SFS-3012R(config)# bridge-group 1 ib-next-hop 10.0.0.1SFS-3012R(config)# interface trunk 1SFS-3012R(config-if-trunk)# bridge-group 1SFS-3012R(config)# interface gateway 2/1SFS-3012R(config-if-gw-2/1)# bridge-group 1SFS-3012R(config-if-gw-2/1)# exit




OL-12957-02



Step 5 Configure a redundancy group, and assign all four bridge groups to it.

The following example shows how to configure a redundancy group and assign bridge groups to it:

SFS-3012R(config)# redundancy-group 1SFS-3012R(config)# bridge-group 1 redundancy-group 1SFS-3012R(config)# bridge-group 2 redundancy-group 1SFS-3012R(config)# bridge-group 3 redundancy-group 1SFS-3012R(config)# bridge-group 4 redundancy-group 1


The following example shows how to enable load balancing between bridge groups:

SFS-3012R(config)# redundancy-group 1 load-balancingSFS-3012R(config)# exit

Verifying Redundancy Group Configuration for a Single Cisco SFS 3012R Server Switch

This section describes how to verify redundancy group configuration for a single Cisco SFS 3012R Server Switch.

The following is sample output from the show redundancy-group command to check the redundancy group configuration and status. This command shows the redundancy group properties and all the members.

Note Make sure the properties match the configuration and that all members are reported.

SFS-3012R# show redundancy-group

================================================================================ Redundancy Groups================================================================================ rlb-id : 1 name : group-p_key : ff:ff load-balancing : enabled broadcast-forwarding : false multicast : false gratuitous-igmp : false igmp-version : v2 num-members : 4 new-member-force-reelection : false


OL-12957-02


================================================================================ Redundancy Group Members================================================================================bridge-group src-addr last-receive --------------------------------------------------------------------------------1 192.168.0.1 Thu Jan 1 01:44:48 19702 192.168.0.1 Thu Jan 1 01:44:47 19703 192.168.0.1 Thu Jan 1 00:02:12 19704 192.168.0.1 Thu Jan 1 00:03:04 1970

Verifying Bridge Group Configuration for a Single Cisco SFS 3012R Server Switch

This section describes how to verify bridge group configuration for a single Cisco SFS 3012R Server Switch.

To check the bridge group configuration and status use the show bridge-group CLI command. Make sure all bridge groups are members of the same redundancy group. When a bridge group is a member of a redundancy group, most of the properties are inherited from the redundancy group. Also make sure only one bridge group is primary and the rest are secondary. This is true even when load balancing is enabled (active-active mode). In the active-passive mode, only the primary bridge group is forwarding and all others are in hot standby state. In the active-active mode all bridge groups are forwarding unicast traffic and only the primary bridge group is forwarding broadcast and multicast if enabled.

The following is sample output from the show bridge-group command, and it checks the bridge group configuration and status.

SFS-3012R# show bridge-group

================================================================================ Bridge Groups================================================================================ bridge-group-id : 1 bridge-group-name : ip-addr : 10.0.0.101 eth-bridge-port : trunk 1 (not tagged) ib-bridge-port : 2/1(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : primary

bridge-group-id : 2 bridge-group-name : ip-addr : 10.0.0.102 eth-bridge-port : trunk 2 (not tagged) ib-bridge-port : 3/2(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group


OL-12957-02


igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : secondary

bridge-group-id : 3 bridge-group-name : ip-addr : 10.0.0.103 eth-bridge-port : trunk 3 (not tagged) ib-bridge-port : 4/1(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : secondary


Configuring Ethernet Gateway Redundancy Using Dual Cisco SFS 3012R Server Switches

This section describes how to configure a dual Cisco SFS 3012R Server Switch redundancy configuration and includes the following topics:

• Verifying Redundancy Group Configuration for Dual Cisco SFS 3012R Server Switches, page 7-19

• Verifying Bridge Group Configuration for Dual Cisco SFS 3012R Server Switches, page 7-20

This typical example shows a dual Cisco SFS 3012R Server Switch setup with a total of four Ethernet gateways that are available as two for each server switch. Both gateways in each chassis are configured to use different switch cards. The gateway in slot 2 uses the switch card in slot 15 and the gateway in slot 3 uses the switch card in slot 16. (To locate the slot numbers on the Cisco SFS 3012R Server Switch, see Figure 4-4.) Thus if any of the switch cards fail, the other gateway continues to remain operational. If the Ethernet ports of the gateways are connected to two different Ethernet switches, the gateways in the same chassis must be connected to different Ethernet switches. This topology allows one chassis and one Ethernet switch failure without interruption to the service. The topology used in this example is shown in Figure 7-4.


OL-12957-02


Figure 7-4 Ethernet Gateway Redundancy with Dual Cisco SFS 3012R Server Switches

Once redundancy group is configured, a primary bridge group is elected and forwarding is enabled.

To configure the first Cisco SFS 3012R Server Switch, perform the following steps:




Step 2 Configure and connect the IB in-band management interface.


Note If the out-of-band Ethernet interface on the controller card is also configured, it must be on a different IP subnet.

The following example shows how to configure and connect the IB in-band management interface:

SFS-3012R(config)# interface mgmt-ibSFS-3012R(config-if-mgmt-ib)# ip address 192.168.0.1 255.255.255.0SFS-3012R(config-if-mgmt-ib)# no shutdownSFS-3012R(config-if-mgmt-ib)# exit

Step 3 Create and configure two link aggregation groups (trunks). Assign Ethernet ports to the link aggregation groups.

The following example shows how to create and configure link aggregation groups:

SFS-3012R(config)# interface trunk 1SFS-3012R(config-if-trunk)# enableSFS-3012R(config-if-trunk)# distribution-type src-dst-ipSFS-3012R(config-if-trunk)# interface ethernet 2/1-2/6SFS-3012R(configif-ether-2/1-2/6)# trunk-group 1

Ethernet Gateway 1

InfiniBand Switch 1

Ethernet Gateway 2

InfiniBand Switch 2

Ethernet Gateway 1

InfiniBand Switch 1

Ethernet Gateway 2

InfiniBand Switch 2

SFS 3012R-1 SFS 3012R-2

Ethernet Fabric

InfiniBand Fabric

1826

40


OL-12957-02


SFS-3012R(config)# interface trunk 2SFS-3012R(config-if-trunk)# enableSFS-3012R(config-if-trunk)# distribution-type src-dst-ipSFS-3012R(config-if-trunk)# interface ethernet 3/1-3/6SFS-3012R(configif-ether-3/1-3/6)# trunk-group 2SFS-3012R(configif-ether-3/1-3/6)# exit


One of the bridge groups uses the IB port 1 to connect to the switch card in slot 15 and the other bridge group uses the IB port 2 to connect to the switch card in slot 16.

Note IP addresses must be from Data IP subnet and must be unique for each bridge group.

The following example shows how to configure two bridge groups and assign ports to them:



Step 5 Configure the redundancy group and assign both bridge groups to it.


The following example shows how to configure the redundancy group and assign bridge groups to it:

SFS-3012R(config)# redundancy-group 1SFS-3012R(config)# bridge-group 1 redundancy-group 1SFS-3012R(config)# bridge-group 2 redundancy-group 1



SFS-3012R(config)# redundancy-group 1 load-balancing


OL-12957-02


To configure the second Cisco SFS 3012R Server Switch, perform the following steps:


The following example shows how to enable configuration mode:


Step 2 Configure and connect the IB in-band management interface.


Note If the out-of-band Ethernet interface on the controller card is also configured it must be on a different IP subnet.

The following example shows how to configure and connect the IB in-band management interface:

SFS-3012R(config)# interface mgmt-ibSFS-3012R(config-if-mgmt-ib)# ip address 192.168.0.2 255.255.255.0SFS-3012R(config-if-mgmt-ib)# no shutdownSFS-3012R(config-if-mgmt-ib)# exit

Step 3 Create and configure two link aggregation groups (trunks). Assign Ethernet ports to link aggregation groups.

The following example shows how to create and configure two link aggregation groups:


SFS-3012R(config)# interface trunk 4SFS-3012R(config-if-trunk)# enableSFS-3012R(config-if-trunk)# distribution-type src-dst-ipSFS-3012R(config-if-trunk)# interface ethernet 3/1-3/6SFS-3012R(config-if-ether-3/1-3/6)# trunk-group 4SFS-3012R(config-if-ether-3/1-3/6)# exit


One of the bridge groups uses the IB port 1 to connect to the switch card in slot 15, and the other bridge group uses the IB port 2 to connect to the switch card in slot 16.

Note IP addresses must be from Data IP subnet and must be unique for each bridge group.

The following example shows how to configure two bridge groups and assign ports to them:



OL-12957-02






SFS-3012R(config)# redundancy-group 1SFS-3012R(config)# bridge-group 3 redundancy-group 1SFS-3012R(config)# bridge-group 4 redundancy-group 1



SFS-3012R(config)# redundancy-group 1 load-balancingSFS-3012R(config)# exit

Verifying Redundancy Group Configuration for Dual Cisco SFS 3012R Server Switches

This section describes how to verify redundancy group configuration for the Cisco SFS 3012R Server Switch.

To check redundancy group configuration and status use the show redundancy-group CLI command. This command shows redundancy group properties and all members. It is important to make sure the properties match the configuration and all members are reported. The redundancy group must be checked on both chassis.

The following is sample output from the show redundancy-group command for the first switch:




OL-12957-02



The following is sample output from the show redundancy-group command for the next switch:




Verifying Bridge Group Configuration for Dual Cisco SFS 3012R Server Switches

This section describes how to verify bridge group configuration for the Cisco SFS 3012R Server Switch.

To check bridge group configuration and status use the show bridge-group CLI command. Inspect both chassis and confirm all bridge groups are members of the same redundancy group. When a bridge group is a member of a redundancy group, most of the properties are inherited from the redundancy group. Also make sure only one bridge group, across all chassis, is primary and the rest are secondary. This is true even when load balancing is enabled (active-active mode). In active-passive mode only the primary bridge group is forwarding. All others are in hot standby state. In active-active mode all bridge groups are forwarding unicast traffic and only primary is forwarding broadcast and multicast if enabled.


OL-12957-02


Step 1 Verify the bridge group configuration for the first Cisco SFS 3012R Server Switch.

The following is sample output from the show bridge-group command that shows how to verify the bridge group configuration for the first Cisco SFS 3012R Server Switch:





OL-12957-02


Step 2 Verify the bridge group configuration for the second Cisco SFS 3012R Server Switch.

The following is sample output from the show bridge-group command that shows how to verify the bridge group configuration for the second Cisco SFS 3012R Server Switch:


================================================================================ Bridge Groups================================================================================ bridge-group-id : 3 bridge-group-name : ip-addr : 10.0.0.103 eth-bridge-port : trunk 3 (not tagged) ib-bridge-port : 2/1(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : secondary



OL-12957-02

Chapter 7 Ethernet Gateway and IPoIB RedundanciesConfiguring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch

Configuring Ethernet Gateway Redundancy for the Cisco SFS 3001 Server Switch

This section describes how to configure Ethernet gateway redundancy for two Cisco SFS 3001 Server Switches. and includes the following topics:

• Verifying Redundancy Group Configuration for Cisco SFS 3001 Server Switches, page 7-27


Two or more Cisco SFS 3001 Server Switches must be used to provide high availability redundancy. A very typical deployment consists of two Cisco SFS 3001 Server Switches with one Ethernet gateway in each (see Figure 7-5).

Figure 7-5 Ethernet Redundancy with Dual Cisco SFS 3001 Server Switches

A single Cisco SFS 3001 Server Switch provides power supply redundancy only. A single Cisco SFS 3001 Server Switch does not provide Ethernet gateway redundancy, because it contains a single gateway slot. For more on the Cisco SFS 3001 redundancy, see Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy.”

For each Cisco SFS 3001 Server Switch, once the redundancy groups are configured, a primary bridge is selected and forwarding is enabled.




SFS-3001> enableSFS-3001# configure terminal

Ethernet Gateway

InfiniBand Switch

Ethernet Gateway

InfiniBand Switch

SFS 3001-1 SFS 3001-2

Ethernet Fabric

InfiniBand Fabric

1826

38


OL-12957-02






SFS-3001(config)# interface mgmt-ibSFS-3001(config-if-mgmt-ib)# ip address 192.168.0.1 255.255.255.0SFS-3001(config-if-mgmt-ib)# no shutdownSFS-3001(config-if-mgmt-ib)# exit

Step 3 Create and configure a link aggregation group (trunk) by assigning Ethernet ports to the link aggregation group.

The following example shows how to create and configure a link aggregation group by assigning Ethernet ports:

SFS-3001(config)# interface trunk 1SFS-3001(config-if-trunk)# enableSFS-3001(config-if-trunk)# distribution-type src-dst-ipSFS-3001(config-if-trunk)# interface ethernet 2/1-2/6SFS-3001(config-if-ether-2/1-2/6)# trunk-group 1SFS-3001(config-if-ether-2/1-2/6)# exit

Step 4 Configure a bridge group and assign ports to it.

Note The IP address must be from the data IP subnet and must be unique for each bridge group.

The following example shows how to configure a bridge group and assign ports:

SFS-3001(config)# bridge-group 1 subnet-prefix 10.0.0.0 8SFS-3001(config)# bridge-group 1 ip-addr 10.0.0.101SFS-3001(config)# bridge-group 1 ib-next-hop 10.0.0.1SFS-3001(config)# interface trunk 1SFS-3001(config-if-trunk)#bridge-group 1SFS-3001(config)# interface gateway 2SFS-3001(config-if-gw-2/2)# bridge-group 1SFS-3001(config-if-gw-2/2)# exit

Step 5 Configure a redundancy group, and assign a bridge group to it.

Note The redundancy group ID must be the same in all chassis.

The following example shows how to configure a redundancy group and assign a bridge group.

SFS-3001(config)# redundancy-group 1SFS-3001(config)# bridge-group 1 redundancy-group 1



SFS-3001(config)# redundancy-group 1 load-balancingSFS-3001(config)#


OL-12957-02


Step 7 (Optional) Change the redundancy group parameters.

Note Enable multicast forwarding on the redundancy group and not on individual bridge groups.

The following example shows how to change the redundancy group parameters:

SFS-3001(config)# redundancy-group 1 multicastSFS-3001(config)#

To configure the next Cisco SFS 3001 Server Switch, perform the following steps:

Note While configuring the bridge group IDs, it is good practice to maintain unique bridge group IDs even if it is not required.



SFS-3001> enableSFS-3001# configure terminal




The following example shows how to configure IB management interface:

SFS-3001(config)# interface mgmt-ibSFS-3001(config-if-mgmt-ib)# ip address 192.168.0.2 255.255.255.0SFS-3001(config-if-mgmt-ib)# no shutdownSFS-3001(config-if-mgmt-ib)# exit

Step 3 Create and configure a link aggregation group (trunk) by assigning Ethernet ports to the link aggregation group.

The following example shows how to create and configure a link aggregation group by assigning Ethernet ports:

SFS-3001(config)# interface trunk 2SFS-3001(config-if-trunk)# enableSFS-3001(config-if-trunk)# distribution-type src-dst-ipSFS-3001(config-if-trunk)# interface ethernet 2/1-2/6SFS-3001(config-if-ether-2/1-2/6)# trunk-group 2SFS-3001(config-if-ether-2/1-2/6)# exit


OL-12957-02


Step 4 Configure a bridge group and assign ports to it.

Note The IP address must be from the data IP subnet and must be unique for each bridge group.

The following example shows how to configure a bridge group and assign ports:

SFS-3001(config)# bridge-group 2 subnet-prefix 10.0.0.0 8SFS-3001(config)# bridge-group 2 ip-addr 10.0.0.102SFS-3001(config)# bridge-group 2 ib-next-hop 10.0.0.1SFS-3001(config)# interface trunk 2SFS-3001(config-if-trunk)#bridge-group 2SFS-3001(config)# interface gateway 2SFS-3001(config-if-gw-2/2)# bridge-group 2SFS-3001(config-if-gw-2/2)# exit

Step 5 Configure a redundancy group, and assign a bridge group to it.

Note The redundancy group ID must be the same in all chassis.

The following example shows how to configure a redundancy group and assign a bridge group.

SFS-3001(config)# redundancy-group 1SFS-3001(config)# bridge-group 2 redundancy-group 1



SFS-3001(config)# redundancy-group 1 load-balancingSFS-3001(config)#

Step 7 (Optional) Change the redundancy group parameters.

Note Enable multicast forwarding on the redundancy group and not on individual bridge groups.

The following example shows how to change the redundancy group parameters:

SFS-3001(config)# redundancy-group 1 multicastSFS-3001(config)#


OL-12957-02


Verifying Redundancy Group Configuration for Cisco SFS 3001 Server SwitchesTo check the redundancy group configuration and status, use the show redundancy-group CLI command. This command shows the redundancy group properties and all its members. It is important to make sure the properties match the configuration and all members are reported. The redundancy groups must be checked on both chassis.

The following is sample output from the show redundancy-group command for the first switch:

SFS-3001# show redundancy-group

================================================================================ Redundancy Groups================================================================================ rlb-id : 1 name : group-p_key : ff:ff load-balancing : enabled broadcast-forwarding : false multicast : true gratuitous-igmp : false igmp-version : v2 num-members : 2 new-member-force-reelection : false

================================================================================ Redundancy Group Members================================================================================bridge-group src-addr last-receive --------------------------------------------------------------------------------1 192.168.0.1 Thu Jan 1 00:06:50 19702 192.168.0.2 Thu Jan 1 00:03:39 1970

The following is sample output from the show redundancy-group command for the next switch:

SFS-3001# show redundancy-group

================================================================================ Redundancy Groups================================================================================ rlb-id : 1 name : group-p_key : ff:ff load-balancing : disabled broadcast-forwarding : false multicast : true gratuitous-igmp : false igmp-version : v2 num-members : 2 new-member-force-reelection : false

================================================================================ Redundancy Group Members================================================================================bridge-group src-addr last-receive --------------------------------------------------------------------------------1 192.168.0.1 Thu Jan 1 00:06:50 19702 192.168.0.2 Thu Jan 1 00:03:39 1970


OL-12957-02


Verifying Bridge Group Configuration for Cisco SFS 3001 Server SwitchesTo check the bridge group configuration and status, use the show bridge-group CLI command. Check both the Cisco SFS 3001 Server Switches to ensure all the bridge groups are members of the same redundancy group.

When a bridge group is a member of the redundancy group, most of the properties are inherited from the redundancy group. Also make sure only one bridge group, across all chassis, is primary and the rest are secondary. This is true even when load balancing is enabled (active-active mode). In active-passive mode, only the primary bridge group is forwarding. All others are in hot standby state. In active-active mode, all bridge groups are forwarding unicast traffic and only the primary bridge group is forwarding broadcast and multicast traffic if enabled.

Step 1 Verify the bridge group configuration for the first Cisco SFS 3001 Server Switch.

The following is sample output from the show bridge-group command that shows how to verify the bridge group configuration for the first switch:

SFS-3001# show bridge-group



OL-12957-02


Step 2 Verify the bridge group configuration for the next SFS 3001 Server Switch.

The following is sample output from the show bridge-group command that shows how to verify the bridge group configuration for the next switch:

SFS-3001# show bridge-group

================================================================================ Bridge Groups================================================================================ bridge-group-id : 2 bridge-group-name : ip-addr : 10.0.0.102 eth-bridge-port : trunk 2 (not tagged) ib-bridge-port : 2/2(gw) (pkey: ff:ff) broadcast-forwarding : false broadcast-forwarding-mode : inherit-from-redundancy-group loop-protection-method : one multicast : false multicast-mode : inherit-from-redundancy-group gratuitous-igmp : false gratuitous-igmp-mode : inherit-from-redundancy-group igmp-version : v2 igmp-version-mode : inherit-from-redundancy-group redundancy-group : 1status-in-redundancy-group : secondary


OL-12957-02



OL-12957-02

CisOL-12957-02

C H A P T E R 8
Fibre Channel Gateway and SRP Redundancies
This chapter describes Fibre Channel gateway redundancies and includes the following sections:

• Dynamic Load Balancing, page 8-2

• Dynamic Gateway Failover, page 8-2

• Path Affinity, page 8-2

• Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch, page 8-3

• Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch, page 8-8

• Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch, page 8-16


Fibre Channel gateway redundancy is based on the concept of using all available paths if the configured policy allows for it. Each ITL has a redundancy policy set and the gateway paths are used according to those policies. Access to the available paths can either be granted or denied. The connection manager that is resident on the controller determines the path of the Initiator/Target.

A logical diagram of Fibre Channel gateways in a redundant configuration is shown in Figure 8-1.


Chapter 8 Fibre Channel Gateway and SRP RedundanciesDynamic Load Balancing

Figure 8-1 Logical Diagram of Fibre Channel Gateways in a Redundant Configuration

The examples in the next sections show the most popular deployment of different types of I/O chassis.

The following features enable Fibre Channel gateways to support redundancy and high availability:

• Dynamic Load Balancing

• Dynamic Gateway Failover

• Path Affinity

Dynamic Load BalancingThese features work within a chassis for the Cisco SFS 3504, SFS 3012R, and SFS 3001 Server Switches. Dynamic load balancing automatically distributes traffic from a host to a target evenly across all available paths and thus provides increased availability. It prevents against single points of failure or performance bottlenecks.

Dynamic Gateway FailoverThis feature is supported by the Cisco SFS 3504, SFS 3012R, and SFS 3001 Server Switches. Dynamic gateway failover enables available gateways to assume the traffic of gateways that fail by having one or more redundant gateways available that are ready to provide service. Fibre Channel gateways support greater granular high availability at the port level and thereby ensure load balancing is utilized most efficiently.

Path AffinityThis feature is supported by the SFS 3012R and the Cisco SFS 3001 Server Switches. Path affinity compensates for the lack of load balancing capabilities of the storage systems, including the storage systems for which multiple paths are available. When multiple I/Os are initiated to a storage system with a queue depth that is greater than one, the host has an affinity to the path that was selected and the I/Os are transmitted on that path until I/O count to that host reaches zero. The next set of I/Os can have a

Fibre Channel Gateway Fibre Channel Gateway

SAN Fabric

InfiniBand Fabric

1830

52

Additional Fibre Channelgateways areadded as required


OL-12957-02

Chapter 8 Fibre Channel Gateway and SRP RedundanciesConfiguring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch

different path affinity. Therefore, at any given time, there is only one path in use, although multiple paths are available. However, each time the path could be different. This is different from port failover, where only one path is used until the path no longer exists.

Note The Cisco SFS 3504 Server Switch does not support path affinity.

Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3504 Server Switch

This section describes how to configure Fibre Channel gateway redundancy for the Cisco SFS 3504 chassis and includes the following topics:

• Verifying Configured Initiator, page 8-5

• Verifying IT, page 8-6

• Verifying LU, page 8-7

Two or more Cisco SFS 3504 Server Switches must be used to provide high availability redundancy. A very typical deployment would consist of two Cisco SFS 3504 Server Switches with two Fibre Channel gateways in each (see Figure 8-2).

The Product ID number for the Fibre Channel gateway card is SFS-3500-FCGW-4G. Each Fibre Channel gateway card is a 4-port, 4 Gbps per-port capable InfiniBand-to-Fibre Channel gateway module.

The Cisco SFS 3504 Server Switches support VSANs. For more details about VSANs, see the Cisco SFS InfiniBand Software Configuration Guide and the Cisco SFS Product Family Command Reference.

Figure 8-2 Fibre Channel Redundancy with Dual Cisco SFS 3504 Server Switches

InfiniBand Switch

Fibre Channel Gateway Fibre Channel Gateway Fibre Channel Gateway Fibre Channel Gateway

InfiniBand Switch

SFS 3504-1 SFS 3504-2

SAN Fabric

InfiniBand Fabric

1830

53


OL-12957-02





SFS-3504> enableSFS-3504# config

Step 2 Configure the initiator.

The following example shows how to configure the initiator:

SFS-3504(config)# fc srp initiatorinitiator initiator-wwpnSFS-3504(config)# fc srp initiator 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 wwnn ? <wwnn> - Initiator wwnn suggested wwnn = 20:08:00:1b:0d:00:12:00 SFS-3504(config)# fc srp initiator 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 wwnn 20:08:00:1b:0d:00:12:00SFS-3504(config)#SFS-3504(config)# fc srp initiator-wwpn 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 3/1 ?<wwpn> - wwpn suggested wwpn = 20:08:00:1b:0d:00:12:16SFS-3504(config)# fc srp initiator-wwpn 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 3/1 20:08:00:1b:0d:00:12:16 vsan 20SFS-3504(config)#

Step 3 Apply port masking.

The following example shows how to apply port masking:

SFS-3504(config)# no fc srp it 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 20:01:00:20:c2:03:31:99 gateway-portmask-policy restricted 3/1-3/4

Step 4 Discover the LUNs.

The following example shows how to discover the LUNs:

SFS-3504(config)# fc srp initiator 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 discover-itl

Step 5 Set the logical unit policy to load balancing, failover, or path-affinity.

The following example shows how to set the logical unit policy to load balancing:

SFS-3504(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancingSFS-3504(config)# exit

To configure the second Cisco SFS 3504 Server Switch, perform Step 1 to Step 5 on the switch.


OL-12957-02


Verifying Configured InitiatorThe following is sample output from the show fc srp initiator command that shows how to verify the configured initiator for the first Cisco SFS 3504 Server Switch:

SFS-3504# show fc srp initiator

================================================================================ SRP Initiators================================================================================ guid: 00:05:ad:00:00:00:22:3c extension: 00:00:00:00:00:00:00:00 description: svbu-fc-host-12 wwnn: 20:08:00:1b:0d:00:12:00 credit: 0 active-ports: none pkeys: bootup-target: 00:00:00:00:00:00:00:00 bootup-lu: 00:00:00:00:00:00:00:00 alt-bootup-target: 00:00:00:00:00:00:00:00 alt-bootup-lu: 00:00:00:00:00:00:00:00 action: none result: none wwpns: port wwpn fc-addr vsan 3/1 20:08:00:1b:0d:00:12:16 14:01:01 20 3/2 20:08:00:1b:0d:00:12:16 14:04:01 20 3/3 20:08:00:1b:0d:00:12:16 14:04:01 20 3/4 20:08:00:1b:0d:00:12:16 14:04:01 20

To verify the configured initiator for the second Cisco SFS 3504 Server Switch, use the command after configuring the switch.


OL-12957-02


Verifying IT The following is sample output from the show fc srp it command that shows how to verify IT for the first Cisco SFS 3504 Server Switch:

SFS-3504# show fc srp it 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 20:01:00:20:c2:03:31:99

================================================================================

SRP IT

================================================================================

guid: 00:05:ad:00:00:00:22:3c

extension: 00:00:00:00:00:00:00:00

target-wwpn: 20:01:00:20:c2:03:31:99

description: it

non-restricted-ports: 3/1-3/4

active-ports: 3/1

physical-access: 3/1

mode: normal-mode

action: none

result: none

To verify the IT for the second Cisco SFS 3504 Server Switch, use the command after configuring the switch.


OL-12957-02


Verifying LUThe following is sample output from the show fc srp itl command that shows how to verify LU for the first Cisco SFS 3504 Server Switch:

SFS-3504# show fc srp itl 00:05:ad:00:00:00:22:3c 00:00:00:00:00:00:00:00 20:01:00:20:c2:03:31:99 00:00:00:00:00:00:00:00================================================================================ SRP ITL================================================================================ guid: 00:05:ad:00:00:00:22:3c extension: 00:00:00:00:00:00:00:00 target-wwpn: 20:01:00:20:c2:03:31:99 fc-lunid: 00:00:00:00:00:00:00:00 srp-lunid: 00:00:00:00:00:00:00:00 logical-id (raw 64 bytes): 02:01:00:22:54:4d:53:20:20:20:20:20:46:43:36:35 : 20:20:20:20:20:20:20:20:20:20:20:20:30:33:33:31 : 39:39:30:30:30:30:00:00:00:00:00:00:00:00:00:00 : 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 logical-id (formatted display): TMS FC65 0331990000 description: itl device-category: random lun-policy: restricted non-restricted-ports: 3/1-3/4 active-ports: none physical-access: none hi-mark: 16 max-retry: 5 min-io-timeout: 10 dynamic-path-affinity: false dynamic-gateway-port-loadbalancing: true dynamic-storage-port-loadbalancing: dynamic-gateway-port-failover: false dynamic-storage-port-failover: active-slots: none


OL-12957-02

Chapter 8 Fibre Channel Gateway and SRP RedundanciesConfiguring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch

Configuring Fibre Channel Gateway Redundancy Using the Cisco SFS 3012R Server Switch

This section describes how to configure Fibre Channel gateway redundancy using the Cisco SFS 3012R Server Switch and includes the following topics:

• Configuring Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012 Server Switch, page 8-8

• Verifying Configuration for a Single Cisco SFS 3012R Server Switch, page 8-10

• Configuring Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R Server Switches, page 8-12

• Verifying Configuration for Two Cisco SFS 3012R Server Switches, page 8-13

The Cisco SFS 3012R Server Switch has various redundancies built into the chassis. It has twelve gateway slots, two controller modules, and two switch cards. Each gateway is connected to both switch cards through the backplane. A single Cisco SFS 3012R Server Switch is therefore capable of being configured for high availability.

For more information about this product, see Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy.”

Configuring Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012 Server Switch

This section describes how to configure Fibre Channel gateway redundancy using a single Cisco SFS 3012R Server Switch.

The example in this section shows a typical single-switch configuration with four Fibre Channel gateways and two switch cards. Two of the gateways, slots 2 and 4, are configured to use switch cards in the first slot and the other two gateways, slots 3 and 5, are configured to use switch cards in the next slot. (For information about the location of the slot numbers, see Figure 4-4.) Thus if any of the switch cards fail, two gateways continue to remain operational. If the Fibre Channel ports of the gateways are required to be connected to two different Fibre Channel switches, connect the gateways to the same IB switch card that are connected to different Fibre Channel switches. Thus if any one Fibre Channel switch and any one IB switch card were to fail, at least one gateway continues to remain operational.

Note For the purpose of this example, gateways in slots 2 and 3 must be connected to different Fibre Channel switches than gateways in slots 4 and 5.

The topology used in this example is shown in Figure 8-3.


OL-12957-02


Figure 8-3 Fibre Channel Gateway Redundancy Using a Single Cisco SFS 3012R Server Switch

Once the redundancy groups are configured, a primary bridge is selected and forwarding is enabled.

To configure a single Cisco SFS 3012R Server Switch, perform the following steps:

Step 1 Telnet to the Cisco SFS 3012R Server Switch.

The following example shows how to Telnet to the server switch and enter the configuration mode:

telnet ip-address <switch address>SFS-3012R-1> enable SFS-3012R-1# configure

Step 2 Configure the IB initiator using the IB GUID and the GUID extension.

The following example shows how to configure the IB initiator and then change the initiator description:

SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 auto-bindSFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 description ib-fc-init-1



SFS-3012R-1(config)# no fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70 gateway-portmask-policy restricted 2/1-2/2,3/1,3/2



SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 discover-itl



SFS-3012R-1(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancing

Fibre Channel Gateway 1 Fibre Channel Gateway 2 Fibre Channel Gateway 3 Fibre Channel Gateway 4

InfiniBand Switch 1 InfiniBand Switch 2

SFS 3012R

SAN Fabric

InfiniBand Fabric

1830

50


OL-12957-02


Step 6 Exit the switch.

The following example shows how to exit the switch:

SFS-3012R-1(config)# exit

Verifying Configuration for a Single Cisco SFS 3012R Server SwitchTo verify the redundancy configuration for a single Cisco SFS 3012R Server Switch, perform the following steps:

Step 1 Telnet to the server switch.

The following example shows how to Telnet to the server switch:

telnet ip-address <switch address>SFS-3012R-1> enable

Step 2 Verify the configured initiator.

The following sample output from the show fc srp initiator command shows how to verify the configured initiator:

SFS-3012R-1# show fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00

================================================ SRP Initiators================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 description: ib-fc-init-1 wwnn: 20:01:00:05:ad:00:1b:4f credit: 0 active-ports: none pkeys: bootup-target: 00:00:00:00:00:00:00:00 bootup-lu: 00:00:00:00:00:00:00:00 alt-bootup-target: 00:00:00:00:00:00:00:00 alt-bootup-lu: 00:00:00:00:00:00:00:00 action: discover-itl result: success wwpns: port wwpn fc-addr 2/1 20:01:00:05:ad:20:1b:4f 67:0f:04 2/2 20:01:00:05:ad:24:1b:4f 67:0e:04 3/1 20:01:00:05:ad:30:1b:4f 67:11:18 3/2 20:01:00:05:ad:34:1b:4f 67:10:18


OL-12957-02


Step 3 Verify the configured Initiator/Target.

The following sample output from the show fc srp it command shows how to verify the configured Initiator/Target:

SFS-3012R-1# show fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70

================================================SRP IT================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 target-wwpn: 50:02:0f:23:00:00:08:70 description: it non-restricted-ports: 2/1-2/2,3/1-3/2 active-ports: none physical-access: 2/1-2/2,3/1-3/2 mode: normal-mode action: none result: none

Step 4 Verify the configured logical unit.

The following is sample output from the show fc srp itl command that shows how to verify the configured logical unit:

SFS-3012R-1# show fc srp itl 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70 00:00:00:00:00:00:00:00================================================SRP ITL================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 target-wwpn: 50:02:0f:23:00:00:08:70 fc-lunid: 00:00:00:00:00:00:00:00 srp-lunid: 00:00:00:00:00:00:00:00 logical-id (raw 64 bytes): 01:03:00:10:60:06:01:60:a2:70:0d:00:70:ad:1c:0a : 3b:a8:db:11:00:00:00:00:00:00:00:00:00:00:00:00 : 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 : 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 logical-id (formatted display): 60060160A2700D0070AD1C0A3BA8DB11 description: itl device-category: random lun-policy: non restricted non-restricted-ports: 2/1-2/2,3/1-3/2 active-ports: none physical-access: 2/1-2/2,3/1-3/2 hi-mark: 16 max-retry: 5 min-io-timeout: 10 dynamic-path-affinity: false dynamic-gateway-port-loadbalancing: true dynamic-storage-port-loadbalancing: dynamic-gateway-port-failover: false dynamic-storage-port-failover: active-slots: none


OL-12957-02





Step 6 To verify the redundancy configuration for the second Cisco SFS 3012R Server Switch, repeat Step 1 to Step 5 for the next switch.

Configuring Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R Server Switches

This section describes how to configure Fibre Channel gateway redundancy using two Cisco SFS 3012R Server Switches.

This typical example shows a setup of two Cisco SFS 3012R Server Switches with a total of four Fibre Channel gateways that are available as two for each server switch. Both gateways in each chassis are configured to use different switch cards. The gateway in slot 2 uses the switch card in one slot and the gateway in slot 3 uses the switch card in the other slot. Thus if any of the switch cards fail, the other gateway continues to remain operational. If the Fibre Channel ports of the gateways are connected to two different Fibre Channel switches, the gateways in the same chassis must be connected to different Fibre Channel switches. This topology allows one chassis and one Fibre Channel switch failure without interruption to the service. The topology used in this example is shown in Figure 8-4.

Figure 8-4 Fibre Channel Gateway Redundancy Using Two Cisco SFS 3012R Server Switches

Once a redundancy group is configured, a primary bridge group is elected and forwarding is enabled.

SFS 3012R-2

1830

51

InfiniBand Switch 1 InfiniBand Switch 2 InfiniBand Switch 1 InfiniBand Switch 2

Fibre Channel Gateway 1 Fibre Channel Gateway 2 Fibre Channel Gateway 3 Fibre Channel Gateway 4SFS 3012R

SAN Fabric

InfiniBand Fabric


OL-12957-02


To configure two Cisco SFS 3012R Server Switches, perform the following steps:

Step 1 Telnet to the first Cisco SFS 3012R Server Switch.


telnet ip-address <switch address>SFS-3012R-1> enable SFS-3012R-1# configure

Step 2 Configure the IB initiator using the IB GUID and the GUID extension.

The following example shows how to configure the IB initiator and then change the initiator description:

SFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 auto-bindSFS-3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 description ib-fc-init-1



SFS-3012R-1(config)# no fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70 gateway-portmask-policy restricted 2/1-2/2,3/1,3/2



3012R-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 discover-itl



SFS-3012R-1(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancing




Step 7 To configure the second Cisco SFS 3012R Server Switch, repeat Step 1 to Step 6 for the next switch.

Verifying Configuration for Two Cisco SFS 3012R Server SwitchesTo verify configuration for the two Cisco SFS 3012R Server Switches, perform the following steps:

Step 1 Telnet to the Cisco SFS 3012R Server Switch.


telnet ip-address <switch address>SFS-3012R-1> enable


OL-12957-02



The following is sample output from the show fc srp initiator command that shows how to verify the configured initiator:

SFS-3012R-1# show fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00

================================================ SRP Initiators================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 description: ib-fc-init-1 wwnn: 20:01:00:05:ad:00:1b:4f credit: 0 active-ports: none pkeys: bootup-target: 00:00:00:00:00:00:00:00 bootup-lu: 00:00:00:00:00:00:00:00 alt-bootup-target: 00:00:00:00:00:00:00:00 alt-bootup-lu: 00:00:00:00:00:00:00:00 action: discover-itl result: success wwpns: port wwpn fc-addr 2/1 20:01:00:05:ad:20:1b:4f 67:0f:04 2/2 20:01:00:05:ad:24:1b:4f 67:0e:04 3/1 20:01:00:05:ad:30:1b:4f 67:11:18 3/2 20:01:00:05:ad:34:1b:4f 67:10:18


The following is sample output from the show fc srp it command that shows how to verify the configured Initiator/Target:

SFS-3012R-1# show fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70

================================================SRP IT================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 target-wwpn: 50:02:0f:23:00:00:08:70 description: it non-restricted-ports: 2/1-2/2,3/1-3/2 active-ports: none physical-access: 2/1-2/2,3/1-3/2 mode: normal-mode action: none result: none


OL-12957-02




SFS-3012R-1# show fc srp itl 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70 00:00:00:00:00:00:00:00

================================================SRP ITL================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 target-wwpn: 50:02:0f:23:00:00:08:70 fc-lunid: 00:00:00:00:00:00:00:00 srp-lunid: 00:00:00:00:00:00:00:00 logical-id (raw 64 bytes): 01:03:00:10:60:06:01:60:a2:70:0d:00:70:ad:1c:0a : 3b:a8:db:11:00:00:00:00:00:00:00:00:00:00:00:00 : 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 : 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 logical-id (formatted display): 60060160A2700D0070AD1C0A3BA8DB11 description: itl device-category: random lun-policy: non restricted non-restricted-ports: 2/1-2/2,3/1-3/2 active-ports: none physical-access: 2/1-2/2,3/1-3/2 hi-mark: 16 max-retry: 5 min-io-timeout: 10 dynamic-path-affinity: false dynamic-gateway-port-loadbalancing: true dynamic-storage-port-loadbalancing: dynamic-gateway-port-failover: false dynamic-storage-port-failover: active-slots: none




Step 6 To verify the configuration for the second Cisco SFS 3012R switch, repeat Step 1 to Step 5 for the switch.


OL-12957-02


Configuring Fibre Channel Gateway Redundancy for the Cisco SFS 3001 Server Switch

This section describes how to configure Fibre Channel gateway redundancy for the Cisco SFS 3001 Server Switch and includes the following topics:

• Configuring Two Cisco SFS 3001 Server Switches, page 8-16

• Verifying Redundancy Configuration for Cisco SFS 3001 Server Switches, page 8-17

Two or more Cisco SFS 3001 Server Switches must be used to provide high availability redundancy. A typical deployment consists of two Cisco SFS 3001 Server Switches using one Fibre Channel gateway in each (see Figure 8-5).

Figure 8-5 Fibre Channel Gateway Redundancy Using Two Cisco SFS 3001 Server Switches

A single Cisco SFS 3001 Server Switch provides power supply redundancy only. A single Cisco SFS 3001 Server Switch cannot provide Fibre Channel gateway redundancy, because it contains a single gateway slot. For more information about this product, see Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy.”

Configuring Two Cisco SFS 3001 Server SwitchesThis section describes how to configure two Cisco SFS 3001 Server Switches. To configure two Cisco SFS 3001 Server Switches, perform the following steps:

Step 1 Telnet to the server switch.

The following example shows how to Telnet to a server switch:

telnet ip-address <switch address>SFS-3001-1> enable SFS-3001-1# configure

Fibre Channel Gateway

InfiniBand Switch

Fibre Channel Gateway

InfiniBand Switch

SFS 3001-1 SFS 3001-2

SAN Fabric

InfiniBand Fabric

1830

49


OL-12957-02


Step 2 Configure the IB initiator using the IB GUID and the GUID-extension.

The following is sample output from the show fc srp initiator command that shows how to configure the IB initiator and then change the initiator description:

SFS-3001-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 auto-bindSFS-3001-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 description ib-fc-init-1



SFS-3001-1(config)# no fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70 gateway-portmask-policy restricted 2/1-2/2



SFS-3001-1(config)# fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 discover-itl

Step 5 Set the logical unit policy to load balancing, failover, or path affinity.


SFS-3001-1(config)# fc srp lu <64 byte lu-id string> dynamic-gateway-port-loadbalancing

Step 6 To configure the second Cisco SFS 3001 Server Switch, repeat Step 1 to Step 5 for the next switch.

Verifying Redundancy Configuration for Cisco SFS 3001 Server SwitchesTo verify redundancy for the Cisco SFS 3001 Server Switches, perform the following steps:

Step 1 Telnet to the first Cisco SFS 3001 Server Switch.


telnet ip-address <switch address>SFS-3001-1> enable


OL-12957-02



The following example shows how to verify the configured initiator:

SFS-3001-1# show fc srp initiator 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00

================================================ SRP Initiators================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 description: ib-fc-init-1 wwnn: 20:01:00:05:ad:00:1b:4f credit: 0 active-ports: none pkeys: bootup-target: 00:00:00:00:00:00:00:00 bootup-lu: 00:00:00:00:00:00:00:00 alt-bootup-target: 00:00:00:00:00:00:00:00 alt-bootup-lu: 00:00:00:00:00:00:00:00 action: discover-itl result: success wwpns: port wwpn fc-addr 2/1 20:01:00:05:ad:20:1b:4f 67:0f:04 2/2 20:01:00:05:ad:24:1b:4f 67:0e:04


The following is sample output from the show fc srp it command that shows how to verify the configured Initiator/Target:

SFS-3001-1# show fc srp it 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70

================================================SRP IT================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 target-wwpn: 50:02:0f:23:00:00:08:70 description: it non-restricted-ports: 2/1-2/2 active-ports: none physical-access: 2/1-2/2 mode: normal-mode action: none result: none



SFS-3001-1# show fc srp itl 29:01:00:05:ad:00:24:fc 00:00:00:00:00:00:00:00 50:02:0f:23:00:00:08:70 00:00:00:00:00:00:00:00================================================SRP ITL================================================ guid: 29:01:00:05:ad:00:24:fc extension: 00:00:00:00:00:00:00:00 target-wwpn: 50:02:0f:23:00:00:08:70 fc-lunid: 00:00:00:00:00:00:00:00 srp-lunid: 00:00:00:00:00:00:00:00 logical-id (raw 64 bytes): 01:03:00:10:60:06:01:60:a2:70:0d:00:70:ad:1c:0a : 3b:a8:db:11:00:00:00:00:00:00:00:00:00:00:00:00


OL-12957-02


: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 : 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 logical-id (formatted display): 60060160A2700D0070AD1C0A3BA8DB11 description: itl device-category: random lun-policy: non restricted non-restricted-ports: 2/1-2/2 active-ports: none physical-access: 2/1-2/2 hi-mark: 16 max-retry: 5 min-io-timeout: 10 dynamic-path-affinity: false dynamic-gateway-port-loadbalancing: true dynamic-storage-port-loadbalancing: dynamic-gateway-port-failover: false dynamic-storage-port-failover: active-slots: none

Step 5 Exit from the switch.

The following example shows how to exit from the switch:

SFS-3001-1# exit

Step 6 To verify the configuration for the second Cisco SFS 3001 Server Switch, repeat Step 1 to Step 5 for the next switch.


OL-12957-02



OL-12957-02

CisOL-12957-02

C H A P T E R 9
Typical Redundancy Use Case
This chapter describes a typical redundancy use case of an Oracle Real Application Clusters (RAC) 10g installation.



Chapter 9 Typical Redundancy Use Case

Figure 9-1 Example of a Fully-Redundant Oracle RAC 10g Installation

Figure 9-1 is an example of a fully-redundant Oracle RAC10g installation. The environment for this cluster is set such that there is no single point of failure. The following redundancies are built into this environment:

• HCA redundancy:

For more details about HCA redundancy, see Chapter 6, “Host Redundancy, and IPoIB and SRP Redundancies.”

• IPoIB and SRP redundancy:

For more details about IPoIB and SRP redundancy, see Chapter 6, “Host Redundancy, and IPoIB and SRP Redundancies.”

• Cisco SFS 7008P and SFS 7000 Series Server Switches redundancy:

For more details about Cisco SFS 7008P and SFS 7000 Series Server Switches redundancy, see Chapter 2, “Cisco SFS 7008P and SFS 7000 Series Server Switch Redundancy.”

2417

74

IB Host 1

SFS 7000D-1

16 1B Hosts

SFS 7000D-2

SFS 3504-2

HCA-1 HCA-2

IB Host 16


HCA-1 HCA-2

SFS 3504-1

LUN

LAN Fabric

ENGW-2ENGW-1

FCGW-1

ENGW-2ENGW-1

FCGW-2 FCGW-2

VSAN Fabric-2

FCGW-1

VSAN Fabric-1


OL-12957-02


• Subnet Manager redundancy:

For more details about Subnet Manager redundancy, see Chapter 5, “Subnet Manager Redundancy.”

• Cisco SFS 3504 and SFS 3000 Series Multifabric Switches redundancy:

For more details about Cisco SFS 3504 and SFS 3000 Series Multifabric Switches redundancy, see Chapter 4, “Cisco SFS 3504 and Cisco SFS 3000 Series Server Switch Redundancy.”

• Ethernet gateways redundancy:

For more details, see Chapter 7, “Ethernet Gateway and IPoIB Redundancies.”

• Fibre Channel gateway redundancy:

For more details, see Chapter 8, “Fibre Channel Gateway and SRP Redundancies.”


OL-12957-02



OL-12957-02

Cisco SFSOL-12957-02

A
P P E N D I X A Acronyms and Abbreviations
Table A-1 defines the acronyms and abbreviations that are used in this publication.

Table A-1 List of Acronyms and Abbreviations

Acronym Expansion

API Application Program Interface

APM Automatic Path Migration

CLI command-line interface

HCA Host Channel Adapter

HSM High-Performance Subnet Manager

IB InfiniBand

IPoIB IP over InfiniBand

ITL Initiator/Target/LUN

LID Local ID

LU logical unit

LUN logical unit number

MTBF mean time between failure

OFED OpenFabrics Enterprise Distribution

RAC Real Application Clusters

RHEL Red Hat Enterprise Linux

SFS Server Fabric Switch

SLES SuSE Linux Enterprise Server

SNMP Simple Network Management Protocol

SRP SCSI RDMA Protocol

SSH Secure Shell

VLAN virtual local area network

A-1 InfiniBand Redundancy Configuration Guide

Appendix A Acronyms and Abbreviations

A-2Cisco SFS InfiniBand Redundancy Configuration Guide

OL-12957-02

CisOL-12957-02

I N D E X

A

abbreviations 1-1

acronyms 1-1

active controller 4-6

active standby controller 4-6

audience vii

B

blanking panel 2-4

blower module redundancy 4-6

bonding driver 6-6

bridge group 7-1

broadcast traffic 7-1

C

CLI session 2-2

controller card 5-3

controller module redundancy 4-6

conventions, document viii

core slot 2-6

D

database synchronization 5-8

protocol 5-1

data traffic 5-1

Device Mapper Multipath 6-8

document

audience vii

conventions viii

organization vii

related ix

driver operation 6-6

dual-port HCA 6-1

dynamic gateway failover 8-2

dynamic load balancing 8-2

dynamic storage reconfiguration 6-9

E

embedded processors 5-4

Embedded Subnet Manager 5-1

end storage unit 6-8

Enterprise solutions 1-1

Ethernet gateway redundancy 4-4, 4-7, 7-1

Ethernet port 2-5

F

fabric card 2-3

fabric controller 2-5, 2-6

fabric redundancy 2-8, 4-7, 4-10

failover 6-8

fan redundancy 4-9

fan tray 2-4

fan tray module 4-9

Fibre Channel gateway redundancy 4-4, 4-7, 8-1

Fibre Channel network 4-4

H

high availability, IPoIB 6-6

High-Performance Subnet Manager 5-1, 5-4

IN-1co SFS InfiniBand Redundancy Configuration Guide

Index

host redundancy 6-1

hot-standby 2-2

hot swap 4-3, 4-5

HTTP 2-3

I

IBM Blade Center 3-1

IP network 4-4

IPoIB

configure 6-6

failover 6-8

high availability 6-6

verify 6-8

IPoIB redundancy 6-1

IPoIB specification 6-4

IP subnet 7-2

L

large clusters 5-8

large fabric 5-4

Line Interface Module 2-6

load balancing 4-7, 6-8

M

management interface module 2-3, 2-5, 2-6

master-poll-interval 5-7

master-poll-retries 5-7

master Subnet Manager 5-1

merge, physical ports 6-4

MTBF 5-8

multicast traffic 7-1

multi-IC switches 4-7

multiple HCAs 6-3

IN-2Cisco SFS InfiniBand Redundancy Configuration Guide

N

Netperf 6-8

node card 2-3

node slot 2-6

O

operating system 4-2

optimal routing 5-4

Oracle RAC 10g 9-1

organization, document vii

P

partition 4-2

path affinity 8-2

port aggregation 4-7

port failover 6-6

port redundancy 2-10

power-fan module 4-3

power supply module 2-4

power supply module redundancy 4-9

power supply redundancy 2-9, 4-5

primary controller 2-2

primary interface 6-8

priority number 5-4, 5-6

product ID 8-3

R

reboot 2-2

recovery mode 2-3

Red Hat Enterprise Linux. See RHEL

redundancy 1-1

software 2-2

related documentation ix

reset 2-2

OL-12957-02

Index

RHEL 6-8

S

Secure Shell. See SSH

serial console port 2-3

serial port 2-5

Server Fabric Switch 1-1

SFS. See Server Fabric Switch

Simple Network Management Protocol. See SNMP.

SLES 6-8

SNMP 2-3

software redundancy 2-2, 4-2

SRP redundancy 6-1

SSH 2-3

standby controller 2-2

standby Subnet Manager 5-1

Subnet Manager

Embedded 5-1

High-Performance 5-1, 5-4

master 5-1

redundancy 5-1

standby 5-1

SuSE Linux Enterprise Server. See SLES

switch module redundancy 4-7

system mastership 2-6

T

Telnet 2-3

trap mechanism 5-4

U

unmerge, physical ports 6-5

OL-12957-02

V

verbose mode 6-6

virtual local area network. See VLAN

virtual port 6-6

VLAN 7-1

W

world-wide names. See WWN.

WWN 6-8


Index


OL-12957-02

Cisco SFS InfiniBand Redundancy Configuration Guide · iii Cisco SFS InfiniBand Redundancy Configuration Guide 78-12957-02 CONTENTS Preface vii Audience vii Organization vii Conventions

Documents

Cisco SFS InfiniBand Redundancy Configuration Guide · iii Cisco SFS InfiniBand Redundancy Configuration Guide 78-12957-02 CONTENTS Preface vii Audience vii Organization vii Conventions