IPMP Howto

IPMP HOWTOIP Multipathing..................................................................................1-1Objectives ........................................................................................... 1-1Introducing IPMP .............................................................................. 1-2Self-Test ............................................................................................... 1-3Hardware and Software Requirements...........................................2-1Objectives ........................................................................................... 2-1Supported Hardware......................................................................... 2-2Supported Software........................................................................... 2-3Self-Test ............................................................................................... 2-4Data Path...........................................................................................3-1Objectives ........................................................................................... 3-1The ifconfig Output ....................................................................... 3-2Standby Configuration...................................................................... 3-3Example Failure ....................................................................... 3-5Multipath Configuration................................................................... 3-7Self-Test ............................................................................................. 3-10Control Path ......................................................................................4-1Objectives ........................................................................................... 4-1The ifconfig Subcommands.......................................................... 4-2The if_mpadm Command ................................................................. 4-3Alternate Pathing and Dynamic ReconfigurationInteraction With IPMP .......................................................... 4-3The in.mpathd Daemon................................................................... 4-4Multipathing Configuration File ............................................ 4-5Self-Test ............................................................................................... 4-7Installation.........................................................................................5-1Objectives ........................................................................................... 5-1Checking the IPMP Package............................................................. 5-2Configuration ................................................................................... 6-1Objectives ........................................................................................... 6-1System Preparation............................................................................ 6-2Standby Configuration...................................................................... 6-3Performing a Manual Configuration With IPv4................... 6-3Performing an Automatic Configuration With IPv4 ........... 6-4Performing a Manual Configuration With IPv6................... 6-6Performing an Automatic Configuration With IPv6 ........... 6-7Multipath Configuration................................................................... 6-8Performing a Manual Multipath ConfigurationWith IPv4................................................................................. 6-8Performing an Automatic Multipath ConfigurationWith IPv4................................................................................. 6-9Performing a Manual Multipath ConfigurationWith IPv6.............................................................................. 6-12Performing an Automatic Multipath ConfigurationWith IPv6............................................................................... 6-12Systems Administration Tasks....................................................... 7-1Objectives ........................................................................................... 7-1Stopping and Starting the in.mpathd Daemon........................... 7-2Automatically Starting the in.mpathd Daemon.......................... 7-2

Self-Test ............................................................................................... 7-3Message Logs and Errors ............................................................... 8-1Objectives ........................................................................................... 8-1Messages.............................................................................................. 8-2The in.mpathd Diagnostics ............................................................. 8-3Self-Test ............................................................................................... 8-5Troubleshooting and Tuning .......................................................... 9-1Objectives ........................................................................................... 9-1Debug Mode for the in.mpathd Daemon ....................................... 9-2The snoop IP Packets......................................................................... 9-4Tracing Routing With the netstat Command................................. 9-7Self-Test ............................................................................................... 9-9IPMP Lab......................................................................................... 10-1Objectives: ........................................................................................ 10-1Exercise: Configuring IPMP........................................................... 10-2Preparation.............................................................................. 10-3Task 1 Create an IPMP Standby IPMP Configuration....... 10-3Preparation.............................................................................. 10-4Task 2 Create a Multipath IPMP Configuration................. 10-4Task 3 Verify Load Spreading.............................................. 10-5Task 4: Verify an IPMP Failover and FailbackOperation .............................................................................. 10-5Exercise Summary............................................................................ 10-6

IP MultipathingObjectivesUpon completion of this module, you should be able to: Describe IP multipathing (IPMP) Define load spreadingIntroducing IPMP

Introducing IPMPIPMP provides the Solaris operating environment (Solaris OE) with theability to recover from single point failures with a network adapter, andalso increased throughput when multiple network adapters are connectedto the same IP link (for example, the same ethernet switch).If a failure occurs in the network adapter, and if an alternate adapter hasbeen configured, the system fails over, with all the network accesschanging automatically from the failed adapter to the alternate adapterallowing uninterrupted access to the network. Also, when there aremultiple network adapters connected to the IP link, increased throughputwill be achieved by spreading the load across multiple network adapters.Following are the summary of features available from IP multipathing: Outbound load spreading – Outbound network packets are spreadacross multiple network adapters without affecting the ordering ofpackets to achieve higher throughput. The load spreading is doneonly when the server is talking to more than one client. Failure detection – Ability to detect when a network adapter hasfailed, and automatically failing over the network access to analternate network adapter. This assumes that an alternate networkadapter was configured. Repair detection – Ability to detect when a network adapter that

failed previously has been repaired and automatically failing backthe network access to an alternate network adapter. This processassumes that failbacks are enabled.

Hardware and Software RequirementsObjectivesUpon completion of this module, you should be able to: List supported hardware List software requirements List limitations of the Solaris operating environment (Solaris OE)implementationSupported Hardware

Supported HardwareIP Multipathing (IPMP) is supported on both the SPARC and IntelArchitecture (IA). Only similar interfaces are supported in a multipathinggroup. For example, you cannot mix Token Ring and Ethernet in the samemultipath group. You can mix different Ethernets, such as le, hme, ge, andqfe, in one multipath group.Asynchronous Transfer Mode (ATM) is only supported in LAN Emulation(LANE) mode.On SPARC, the OpenBoot PROM (OBP) local-mac-address? variable

must be set to true if multipathing is being used..Supported SoftwareIPMP is supported on the Solaris 8 OE Update 2 (10/00). Currently thereis no backport to previous versions of the Solaris OE.The /sbin/in.mpathd daemon detects failed interfaces. The daemon isstarted with the ifconfig utility when the subcommand group is used.IPMP is configured with the ifconfig utility. There are a few newsub-commands in the ifconfig utility to configure multipathing.Configuration with the ifconfig utility and in.mpathd daemon isdescribed in detail in Unit 6, ‘‘Configuration.”

Self-Test1. Which platform type is supported by IPMP?A. SPARC onlyB. Intel onlyC. Sun4uD. SPARC and Intel architectureAnswer:D. SPARC and Intel architecture

ObjectivesUpon completion of this module, you should be able to: Identify test address interfaces Identify multipath configuration Identify standby configuration Identify possible paths for multipath Internet Protocol (IP)datagrams between client and server.

Diagram IP failover data flow between client and server

The ifconfig output has new flags and information to identify ifmultipathing is being used. The flags are: DEPRECATED, NOFAILOVER,STANDBY, INACTIVE, and FAILED. Other information given by this outputis groupname, which should be a non-null name to identify the physicalgroup name to which the interface belongs. A description of the flags forifconfig output follows: DEPRECATED – The interface address should not be preferred as thesource address for outbound packets. If any non-deprecatedaddresses are present, they are preferred over deprecated. Theinterface still responds to pings and can make a connection. NOFAILOVER – The interface address does not fail over when a failedinterface is detected. STANDBY – The interface is marked as a standby interface for thephysical interface group. INACTIVE – Use this flag with STANDBY, and it indicates that theinterface is not currently being used. FAILED – This flag marks the interface as failed, and normal traffic isre-routed to another interface (if possible).

Standby ConfigurationThe following is an example of partial ifconfig -a output producedwhen a system is set up with a standby interface:hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname standby-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname standby-testether 8:0:20:b7:41:d8The hme0 interface has the primary IP address for this system, and it isdesignated 172.20.4.181. The hme0 interface is a part of the physicalinterface group called standby-test. The hme0:1 flags DEPRECATED andNOFAILOVER indicate that this interface is not to be used by a normalapplication and the IP address will not be moved if the interface isdetected as having failed.Standby Configuration

qfe0 interface is the standby interface and is only used when there is a

detected failure on the hme0 interface72.20.4.183hme0IPhme0:1 qfe0 172.20.4.183 group standby-test deprecated-failover standby# cat /etc/hostname.qfe0172.20.4.181 group standby-test up \addif 172.20.4.182 -failover deprecated up# cat /etc/hostname.hme0lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4>mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname standby-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255# ifconfig -aStandby Configuration

This is an example of a failure on the hme0 interface, which is a partiallisting of the output from the ifconfig -a command:hme0: flags=19000842<BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER,FAILED> mtu 0 index 2inet 0.0.0.0 netmask 0groupname standby-testether 8:0:20:b3:22:f2hme0:1: flags=19040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=29040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname standby-testether 8:0:20:b7:41:d8qfe0:1: flags=21000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,STANDBY> mtu1500 index 3inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.25The preceding example shows how the hme0 interface has failed and nowhas 0.0.0.0 as its Internet address. There is an new logical interface on

the qfe0 interface. It is qfe0:1 and now has the internet address of hme0(127.20.4.181). All connections that were on hme0 transferred toqfe0:1. Notice the flags on hme0 are FAILED and NOFAILOVER and theflag on hme0:1 is FAILED.Note – The reason that hme0 has 0.0.0.0 as the address is because hme0is the name of an interface and also represents the zeroth address (forexample, the hme0 interface is actually the hme0:0 interface). To addressthe interface as hme0 (for example in ioctls), even after a failover, youneed to have a zeroth address.Standby ConfigurationWhen the in.mpathd daemon detects that the hme0 interface has beenrepaired, failback occurs and the routes and interfaces return to a normalconfiguration. The messages in the following output show that link downmessage (Line 1) of the original failure, to the failback (Line 9), to theoriginal interface:1 Aug 2 11:54:44 proto181 hme: SUNW,hme0 : No response from Ethernetnetwork : Link down -- cable problem?2 Aug 2 11:54:52 proto181 in.mpathd[31]: NIC failure detected on hme03 Aug 2 11:54:52 proto181 in.mpathd[31]: Successfully failed over fromNIC hme0 to NIC qfe04 Aug 2 11:54:55 proto181 hme: SUNW,hme0 : No response from Ethernetnetwork : Link down -- cable problem?5 Aug 2 11:56:35 proto181 last message repeated 9 times6 Aug 2 11:56:36 proto181 hme: SUNW,hme0 : Internal TransceiverSelected.7 Aug 2 11:56:36 proto181 hme: SUNW,hme0 : Auto-Negotiated 100 MbpsHalf-Duplex Link Up8 Aug 2 11:56:51 proto181 in.mpathd[31]: NIC repair detected on hme09 Aug 2 11:56:51 proto181 in.mpathd[31]: Successfully failed back toNIC hme0

Multipath ConfigurationThe following code example, which is a partial listing of the ifconfig -acommand, shows a system that is set up with a multipath interface:hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.25groupname multipath-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b7:41:d8qfe0:1:

flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.25The hme0 interface, which is normally running a 172.20.4.181 IPaddress, has a physical interface group name multipath-test. Thehme0:1 interface is the test address for the hme0 interface with an172.20.4.182 IP address. Its flags are DEPRECATED and NOFAILOVER andare special for multipathing. No application should select this as a sourceaddress, and the address will not be failed over.Caution – This does not prevent someone from connecting to the testaddress from another host. It is still a legal address that can be used butthe address will not failover.The qfe0 interface, which is normally running a 172.20.4.183 IPaddress, has a physical interface group name multipath-test. Theqfe0:1 interface is the test address for the hme0 interface with an172.20.4.184 IP address. Its flags are also DEPRECATED and NOFAILOVERso no application should select this as a source address, and the addresswill not fail over.The hme0 interface address can be relocated to the qfe0 interface if there isa failed interface detected. The reverse is true if the qfe0 interface failsthen, the hme0 interface takes over.Multipath Configuration

You can use the qfe0 (172.20.4.183) and the hme0 (172.20.4.181)addresses as a usable source addresses for applications from this host. Forinstance, a telnet session can have hme0 address as its source address toone host, and during the next telnet session, it might have the address ofthe qfe0 as the source address to another host.Figure 3-2 shows the network when it is configured for IP Multipathing(IPMP). There are four possible paths that IPMP can use with IP traffic toand from this host with two interfaces. If there are more interfaces, thenmore possible paths are added: In/out on hme0 In/out on qfe0 In on hme0/out on qfe0 In on qfe0/out on hme0IP172.20.4.181 group multipath-test up \addif 172.20.4.182 -failover deprecated up172.20.4.183 group multipath-test up \addif 172.20.4.184 -failover deprecated up# cat /etc/hostname.hme0lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255

qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b7:41:d8qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 3inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255# ifconfig -aMultipath Configuration

Logical interfaces are also supported on IPMP. All connections work asnormal.There is a change in source address selection in Solaris 8 OE Update 2.Now, the Solaris OE uses the inbound packet destination address as thesource address of the outbound packet. This means that packets that comein on a certain IP address now leave with that IP address as its sourceaddress and not the primary physical address interface as in previousreleases of the Solaris OE.

1. Which flag is not new to the ifconfig utility in the Solaris 8 OEUpdate 2?A. DEPRECATEDB. NOFAILOVERC. STANDBYD. INACTIVEE. FAILEDF. IPv4Answer:F. IPv4

Control PathObjectivesUpon completion of this module, you should be able to: Locate the configuration file Identify parameters in the configuration file Look up possible values and actions for each parameterThe ifconfig Subcommands

The ifconfig SubcommandsThe ifconfig(1M) tool configures Internet Protocol (IP) Multipathing(IPMP). Several new sub-commands were added for use with IPMP: deprecated – Do not use the interface with normal IP traffic. failover – The interface address does not fail over when a failedinterface is detected. group – Use this sub-command to create an physical interface group. standby - Use this sub-command to mark an interface as a standbyfor the physical interface group. addif - Use this sub-command to create the next unused logicalinterface on the specified physical interface. This is a sub-commandnew to Solaris 8 Operating Environment (Solaris 8 OE) FCS.

The if_mpadm Command.

Alternate Pathing and Dynamic ReconfigurationInteraction With IPMPCurrently, there is no support for using IPMP concurrently with AlternatePathing (AP) on the same network interfaces. You can use AP forsupported storage arrays and use IPMP for your network interfaces.Currently, IPMP does not automatically fail over an active networkinterface when a Dynamic Reconfiguration operation is initiated. Thesystem administrator must manually issue the following command:/usr/sbin/if_mpadm -d network_interface_name/usr/sbin/if_mpadm -r network_interface_namewhere: The -d option detaches the network interface or to take it offline The -r option reattaches a previously detached network interface. network_interface_name is the network interface to be managed.The if_mpadm command allows administrators to manually force IPMP tostop using a redundant network interface and then make it possible to useDynamic Reconfiguration to remove the network interface card. Thisfeature of IPMP is in the Solaris 8 OE Update 3 (1/01).The in.mpathd Daemon

The in.mpathd DaemonThe in.mpathd multipathing daemon detects failures and repairs bysending out probes on all the interfaces that are part of a group. When aninterface is part of a group and has a test address, the daemon startssending out probes for determining failures on that interface. If there areno replies to five consecutive probes, the interface is considered to havefailed. The probing rate depends on the failure detection time. By default,failure detection time is 10 seconds. Thus, the probing rate is one probeevery two seconds.To avoid synchronization in the network, probing is not periodic. If fiveconsecutive probes fail, the in.mpathd daemon considers the interface asfailed and performs a failover of the network access from the failedinterface to another interface in the group that is functioning properly. If astandby interface is configured, it is chosen for failover of the IPaddresses, and broadcasts and multicasts memberships. If no standbyinterface exists, the interface with the least number of IP addresses ischosen.Because the in.mpathd daemon determines what targets to probedynamically, you cannot configure the targets. Routers connected to thelink are chosen as targets for probing. If no routers exist on the link,arbitrary hosts on the link are chosen. A multicast packet sent to the “allhosts” multicast address (224.0.0.1 in Internet Protocol version 4 [IPv4]and ff02::1 in Internet Protocol version 6 [IPv6]) determines thearbitrary hosts. The first few hosts that respond to the echo packets arechosen as targets for probing. If the in.mpathd daemon cannot findrouters or hosts that responded to ICMP echo packets, the in.mpathddaemon cannot detect failures.

The in.mpathd daemon detects an NIC failure and repair by sending andreceiving ICMP echo requests and replies on each NIC. The in.mpathddaemon sends the probes to on-link routers. If no routers are available, itsends the probes to neighboring hosts. Thus, for network failure detectionand repair, there must be at least one neighbor on each link that respondsto ICMP echo request probes.

The in.mpathd daemon works on both IPv4 and IPv6. If IPv4 is plumbedon a NIC, an IPv4 test address is configured on the NIC, and the NIC isconfigured as part of a network multipathing group, then the in.mpathddaemon starts sending ICMP probes on the NIC using IPv4.In the case of IPv6, you must configure the link-local address as the testaddress. The in.mpathd daemon does not accept a non-link-local addressas a test address. If the NIC is part of a multipathing group and the testaddress has been configured, then the in.mpathd daemon probes the NICfor failures using IPv6.Even if both the IPv4 and IPv6 protocol streams are plumbed, it issufficient to configure only one of the two, that is, either an IPv4 testaddress or an IPv6 test address on a NIC. If only an IPv4 test address is configured, it probes using onlyICMPv4. If only an IPv6 test address is configured, it probes using onlyICMPv6. If both type test addresses are configured, it probes using bothICMPv4 and ICMPv6.

Multipathing Configuration FileThe in.mpathd daemon uses the settings in the /etc/default/mpathdconfiguration file to invoke multipathing. Changes to this file are read bythe in.mpathd daemon at startup and on a SIGHUP signal. This filecontains the following default settings and information:## Time taken by mpathd to detect a NIC failure in ms. The minimum time# that can be specified is 100 ms.#FAILURE_DETECTION_TIME=10000## Failback is enabled by default. To disable failback turn off thisoption#FAILBACK=yes## By default only interfaces configured as part of multipathing groups# are tracked. Turn off this option to track all network interfaces# on the system#TRACK_INTERFACES_ONLY_WITH_GROUPS=yesThe in.mpathd Daemon

For more information on configuring the /etc/default/mpathd

configuration file, refer to the IP Multipathing Administration Guide forSolaris 8 OE and for Solaris 9 see Solaris 9 Systems Administration Guide: IPServices.Failure Detection TimeYou can set a lower value of failure detection time. Sometimes thesevalues might not be achieved if the load on the network is too high. Then,the in.mpathd daemon prints a message on the console, indicating thatthe time cannot be met. The daemon also prints the time that it can meetcurrently. If the response comes back correctly, the in.mpathd daemonmeets the failure detection time provided in this file.FailbackAfter a failover, failbacks take place when the failed interface is repaired.However, the in.mpathd daemon does not fail back the interface if theFAILBACK flag is set to no.Tracking Interfaces Only With the groups OptionBy turning off the groups option, the in.mpathd daemon tracks allinterfaces in the system. When a failure is detected, an appropriatemessage is logged on the console. For this option to function properly,Ethernet addresses on all the interfaces must be unique.

Installing IPMP

Checking the IPMP PackageInternet Protocol Multipathing (IPMP) does not require specialinstallation. It is installed with the Solaris OE core installation. To checkfor the package, use the pkginfo command:% pkginfo | grep -i csrsystem SUNWcsr Core Solaris, (Root)

System PreparationThese conditions must exist before Internet Protocol Multipathing (IPMP)can be used: The set of STREAMS modules on all network adapters in themultipathing group must be the same. If you have plumbed Internet Protocol version 4 (IPv4) on onenetwork adaptor, then you must plumb IPv4 on all network adaptorsin the multipathing group.When you plumb a device, you open the device associated with thephysical interface name and set up the streams needed for IP to usethe device. When used with a logical interface name, this commandcreates a a specific named logical interface. An interface must beseparately plumbed for use by IPv4 and IPv6. The address_familyparameter controls whether the ifconfig command applies to IPv4or IPv6. If you have plumbed Internet Protocol version 6 (IPv6) on onenetwork adapter, then you must plumb IPv6 on all network adaptorsin the multipathing group. All Ethernet network adapters in the system should have uniqueMedia Access Control (MAC) addresses. This is achieved by settingthe local-mac-address? parameter to true in the OpenBoot

PROM (OBP) for SPARC platforms. Nothing needs to be done forIntel architecture (IA) platforms. All network adapters of the multipathing group must be connectedto the same IP link. The multipathing group should not contain dissimilar interfaces. Theinterfaces that are grouped together should be of the same interfacetype that is defined in the /usr/include/net/if_types.h file. Forexample, you cannot combine Ethernet with Token Ring, and youcannot combine a Token bus with asynchronous transfer mode(ATM). When you use IP network multipathing with ATM, you mustconfigure the ATM for LAN emulation (multipathing over classicalIP instances is not currently supported).

Performing a Manual Configuration With IPv4To perform a manual configuration with IPv4, follow these steps:1. Assume that the primary interface (hme0, 172.20.4.181) is running.2. Make sure that the local-mac-address? parameter is set to true.# eeprom local-mac-address?local-mac-address?=true3. Join an interface group called standby-test.# ifconfig hme0 group standby-test4. Create a test address (172.20.4.182) for hme0.# ifconfig hme0 addif 172.20.4.182 deprecated -failovernetmask + broadcast + up5. Use the plumb command on the second interface.# ifconfig qfe0 plumb6. Join interface group standby-test.# ifconfig qfe0 group standby-test7. Configure qfe0 as a standby with test address of 172.20.4.183.# ifconfig qfe0 addif 172.20.4.183 deprecated -failoverstandby netmask + broadcast + up8. Check your work.# ifconfig -alo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname standby-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255

groupname standby-test

Performing an Automatic Configuration With IPv4To make the configuration a part of the boot process, configure the/etc/hostname.hme0 and /etc/hostname.qfe0 files similar to thefollowing example:# cat /etc/hostname.hme0172.20.4.181 group standby-test upaddif 172.20.4.182 -failover deprecated up# cat /etc/hostname.qfe0172.20.4.183 group standby-test deprecated -failover standbyThe system takes care of the netmask and broadcast as a part of startup.Standby Configuration

Performing a Manual Configuration With IPv6To perform a manual configuration with IPv6, follow these steps:1. Make sure that the MAC addresses on your interfaces are unique. Ifthe interfaces are not already members of an interface group, theyneed to join one. Again, assuming that IPv6 addresses are alreadyconfigured, issue the following commands:# ifconfig hme0 group standby-test# ifconfig qfe0 group standby-test172.20.4.184172.20.4.254cat # /etc/hostname.qfe0172.20.4.181 group multipath-test up \addif 172.20.4.182 -failover deprecated up172.20.4.183 group multipath-test up \addif 172.20.4.184 -failover deprecated up# cat /etc/hostname.hme0lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b7:41:d8qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 3inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255# ifconfig -a2. When the interfaces are joined, set up qfe0 as the standby interface:# ifconfig qfe0 inet6 standby -failoverCaution – You must not mark IPv6 link-local addresses as DEPRECATED.

Performing an Automatic Configuration With IPv6To make the configuration a part of the boot process, configure the

/etc/hostname6.hme0 and /etc/hostname6.qfe0 files similar to thefollowing example:# cat /etc/hostname6.hme0group standby-test -failover up# cat /etc/hostname6.qfe0group standby-test -failover standby up

Performing a Manual Multipath Configuration WithIPv4To perform a manual multipath configuration with IPv4, follow thesesteps:1. Assume that the primary interface (hme0, 172.20.4.181) isrunning. Make sure that local-mac-address? is set to true.# eeprom local-mac-address?local-mac-address?=true2. Join an interface group called multipath-test.# ifconfig hme0 group multipath-test3. Create a test address (172.20.4.182) for hme0.# ifconfig hme0 addif 172.20.4.182 deprecated -failovernetmask + broadcast + upThe second interface is already running with the IP address172.20.4.183.4. Join interface group multipath-test.# ifconfig qfe0 group multipath-test5. Configure qfe0 as a standby with test address of 172.20.4.184.# ifconfig qfe0 addif 172.20.4.184 deprecated -failovernetmask + broadcast + up6. Check your work.# ifconfig -alo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b3:22:f2hme0:1:flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3Multipath Configurationinet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b7:41:d8qfe0:1:flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>

mtu 1500 index 3inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255

Performing an Automatic Multipath Configuration WithIPv4To make the configuration a part of the boot process, configure the/etc/hostname.hme0 and /etc/hostname.qfe0 files similar to thefollowing example:# cat /etc/hostname.hme0:172.20.4.181 group multipath-test upaddif 172.20.4.182 -failover deprecated up# cat /etc/hostname.qfe0:172.20.4.183 group multipath-test upaddif 172.20.4.184 -failover deprecated upMultipath Configuration

hme0 hme0:1 qfe0 qfe0:# cat /etc/hostname.qfe0172.20.4.181 group multipath-test up \addif 172.20.4.182 -failover deprecated up172.20.4.183 group multipath-test up \addif 172.20.4.184 -failover deprecated up# cat /etc/hostname.hme0lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b7:41:d8qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 3inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255# ifconfig -aMultipath Configuration172.20.4.254cat # /etc/hostname.qfe0172.20.4.181 group multipath-test up \addif 172.20.4.182 -failover deprecated up172.20.4.183 group multipath-test up \addif 172.20.4.184 -failover deprecated up# cat /etc/hostname.hme0lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b3:22:f2hme0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255

qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b7:41:d8qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 3inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255# ifconfig -a

Performing a Manual Multipath Configuration WithIPv6To perform a manual multipath configuration with IPv6, follow thesesteps:1. Make sure the MAC addresses are unique. If the interfaces are notalready members of an interface group, they need to join one. Againassuming that IPv6 addresses are already configured, issue thefollowing command:# ifconfig hme0 group multipath-test# ifconfig qfe0 group multipath-test2. When the interfaces are joined, hme0 and qfe0 can be set up formultipath:# ifconfig hme0 inet6 -failover# ifconfig qfe0 inet6 -failoverCaution – You must not mark IPv6 link-local addresses as DEPRECATED.

Performing an Automatic Multipath Configuration WithIPv6To make the configuration a part of the boot process, configure the/etc/hostname6.hme0 and /etc/hostname6.qfe0 files similar to thefollowing example:# cat /etc/hostname6.hme0group multipath-test -failover up# cat /etc/hostname6.qfe0group multipath-test -failover up

Stopping and Starting the in.mpathd DaemonTo start the in.mpathd daemon by manually, issue the followingcommand as root:# /sbin/in.mpathdTo start the in.mpathd daemon with debugging turned on:# /sbin/in.mpathd -dNote – The -d flag is an undocumented command-line option.

Automatically Starting the in.mpathd DaemonThe multipath daemon does not have a start and stop script in the/etc/init.d directory. The in.mpathd daemon is executed from theifconfig command when using the group sub-command. Thein.mpathd daemon does not create multiple incarnations.Self-Test

in.mpathd messages in the /var/adm/messages file, issue thefollowing command as root:# grep mpathd /var/adm/messages*

in.mpathd DiagnosticsThis section describes the diagnostics associated with the in.mpathddaemon.IFF_NOFAILOVER addressThe address is not unique, and failure detection is not possible. Everytest address that is configured must be unique on the system. Otherwise,the in.mpathd daemon cannot perform failure detection. Because theIPv6 test address is a link-local address, which in turn is derived from theEthernet address, each network interface card (NIC) must have a uniqueMAC address. Also, when multiple NICs are connected to the samesubnet, one should have a unique Media Access Control (MAC) addressto avoid duplicate packets being picked up by the interface.NIC interface_name of group group_name is not plumbed for IPv[4|6] andmay affect failover capabilityAll NICs in a multipathing group must be homogeneously plumbed. Forexample, if a NIC is plumbed for IPv4, then all NICs in the group must beplumbed for IPv4. The streams modules pushed on all NICs must beidentical.Failures cannot be detected on interface_name as no IFF_NOFAILOVERaddress is availableEvery NIC that is configured as part of a multipathing group must have atest address, which can be either IPv4, IPv6, or both. Otherwise, thein.mpathd daemon cannot probe the NIC, and the preceding messageappears.Invalid failure detection time assuming default 10000An invalid value was encountered for FAILURE_DETECTION_TIME in the/etc/default/mpathd file.Too small failure detection time of time assuming minimum 100The minimum value that can be specified for FAILURE_DETECTION_TIMEis currently 100 milliseconds.Invalid value for FAILBACK valueValid values for the boolean variable FAILBACK are yes or no.The in.mpathd Diagnostics

Invalid value for TRACK_INTERFACES_ONLY_WITH_GROUPS valueValid values for the Boolean variableTRACK_INTERFACES_ONLY_WITH_GROUPS are yes or no.Cannot meet requested failure detection time of time ms on (inet[4|6]interface_name) new failure detection is time msThe round trip time for ICMP probes is higher than the specified failuredetection time. The network is probably congested or the probe targetsare loaded. The in.mpathd daemon automatically increases the failuredetection time to whatever it can achieve under these conditions.Improved failure detection time time ms

The round trip time for ICMP probes has now decreased and thein.mpathd daemon has lowered the failure detection timecorrespondingly.NIC failure detected on interface_nameThe in.mpathd daemon has detected NIC failure on interface_name,and has set the IFF_FAILED flag on NIC interface_name.Successfully failed over from NIC interface_name1 to NIC interface_name2The in.mpathd daemon has caused the network traffic to failover fromNIC interface_name1 to NIC interface_name2, which is part of themultipathing group.NIC repair detected on interface_nameThe in.mpathd daemon has detected that NIC interface_name isrepaired and operational. If the IFF_FAILED flag on the NIC waspreviously set, it will be reset.Successfully failed back to NIC interface_nameThe in.mpathd daemon has restored network traffic back to NICinterface_name, which is now repaired and operational.

Debug Mode for the in.mpathd Daemon1. Place the in.mpathd daemon into debug mode, type:/sbin/in.mpath -dCaution – Debug mode is undocumented. Debugging output isconsidered unstable and scripts or tools should not depend on it to be thesame from release to release.The following is sample output from debug mode:run_timeouts(inet qfe0): next scheduled for this phyint inst 997, nextscheduled global 997 msphyint_inst_timer(inet hme0)cur_time 159084781 snxt_time 159085886 snxt_basetime 159086211 interval1105probe(inet hme0 4660 159084781)run_timeouts(inet hme0): next scheduled for this phyint inst 1105, nextscheduled global 997 msrun_timeouts: 997 mstimer_schedule(997)timer_schedule(997): sec 0 usec 997000in_data(inet hme0)incoming_echo_reply: inet hme0 172.20.4.254 seq 2pi_set_crtt: target - m 2in_signal() got 14in_signal(SIGALRM) delta 1run_timeouts()phyint_inst_timer(inet qfe0)cur_time 159085780 snxt_time 159086917 snxt_basetime 159087423 interval1137probe(inet qfe0 4660 159085780)run_timeouts(inet qfe0): next scheduled for this phyint inst 1137, nextscheduled global 1137 ms2. Print statistics either in normal daemonize or debug modes. Send aUSR1 signal to the daemon, type:

# pkill -USR1 mpathDebug Mode for the in.mpathd DaemonThe following is sample output with an explanation of the followingterms:Probe stats on (inet qfe0)Number of probes sent 4142Number of probe acks received 4142Number of probes/acks lost 0Number of valid unacknowled probes 0Number of ambiguous probe acks received 0Probe stats on (inet hme0)Number of probes sent 4142Number of probe acks received 4142Number of probes/acks lost 0Number of valid unacknowled probes 0Number of ambiguous probe acks received 0Note – These statistics are on a per interface, per protocol basis.Number of probes – The total number of ICMPv4 echo requests sent outon interface after the in.mpathd daemon was started until now.Number of probe acks received – The total number of ICMPv4 echoreplies received in response to the preceding echo requests.Number of probes/acks lost – The number of echo requests that didnot get a reply.Number of valid unacked probes – The number of echo requests towhich you have not received replies and which have not yet timed out.The probe is considered timed out if more than one RTT has elapsed sinceit was sent.Number of ambiguous probe acks – If an echo reply is received longafter it is sent, that is after the circular buffer has wrapped around, andyou cannot determine anything about the packet, it is consideredambiguous. The same applies to unsolicited echo replies; for example, aduplicate.The snoop IP Packets

The snoop IP PacketsThe hme0 and qfe0 interfaces are configured to use multipathing.# ifconfig -alo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1inet 127.0.0.1 netmask ff000000hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2inet 172.20.4.181 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b3:22:f2hme0:1:flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>mtu 1500 index 2inet 172.20.4.182 netmask ffffff00 broadcast 172.20.4.255qfe0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

inet 172.20.4.183 netmask ffffff00 broadcast 172.20.4.255groupname multipath-testether 8:0:20:b7:41:d8qfe0:1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3inet 172.20.4.184 netmask ffffff00 broadcast 172.20.4.255The test address is 172.20.4.182 on hme0:1. The snoop command wastaken on the hme0 interface and filtering on ICMP packets. The sourceEthernet address matches hme0, and source IP address is 172.20.4.182.This ICMP is directed at a router.For the test address, 172.20.4.184 on qfe0:1, you should see a similarpacket with the ethernet source address of the qfe0 interface and source IPaddress of 172.20.4.184:10 ETHER: ----- Ether Header -----11 ETHER:12 ETHER: Packet 1 arrived at 15:28:27.1913 ETHER: Packet size = 50 bytes14 ETHER: Destination = 0:d0:d3:35:46:d0,15 ETHER: Source = 8:0:20:b3:22:f2, Sun16 ETHER: Ethertype = 0800 (IP)17 ETHER:18 IP: ----- IP Header -----19 IP:20 IP: Version = 421 IP: Header length = 20 bytes22 IP: Type of service = 0x00The snoop IP Packets23 IP: xxx. .... = 0 (precedence)24 IP: ...0 .... = normal delay25 IP: .... 0... = normal throughput26 IP: .... .0.. = normal reliability27 IP: Total length = 36 bytes28 IP: Identification = 4093629 IP: Flags = 0x430 IP: .1.. .... = do not fragment31 IP: ..0. .... = last fragment32 IP: Fragment offset = 0 bytes33 IP: Time to live = 1 seconds/hops34 IP: Protocol = 1 (ICMP)35 IP: Header checksum = 781436 IP: Source address = 172.20.4.182, proto182.Central.Sun.COM37 IP: Destination address = 172.20.4.254, protorouter.Central.Sun.COM38 IP: No options39 IP:40 ICMP: ----- ICMP Header -----41 ICMP:42 ICMP: Type = 8 (Echo request)43 ICMP: Code = 0 (ID: 7170 Sequence number: 60558)44 ICMP: Checksum = 586045 ICMP:46

4748 ETHER: ----- Ether Header -----49 ETHER:50 ETHER: Packet 2 arrived at 15:28:27.1951 ETHER: Packet size = 60 bytes52 ETHER: Destination = 8:0:20:b3:22:f2, Sun53 ETHER: Source = 0:d0:d3:35:46:d0,54 ETHER: Ethertype = 0800 (IP)55 ETHER:56 IP: ----- IP Header -----57 IP:58 IP: Version = 459 IP: Header length = 20 bytes60 IP: Type of service = 0x0061 IP: xxx. .... = 0 (precedence)62 IP: ...0 .... = normal delay63 IP: .... 0... = normal throughput64 IP: .... .0.. = normal reliability65 IP: Total length = 36 bytes66 IP: Identification = 4093667 IP: Flags = 0x468 IP: .1.. .... = do not fragmentThe snoop IP Packets69 IP: ..0. .... = last fragment70 IP: Fragment offset = 0 bytes71 IP: Time to live = 255 seconds/hops72 IP: Protocol = 1 (ICMP)73 IP: Header checksum = 7a1374 IP: Source address = 172.20.4.254, protorouter.Central.Sun.COM75 IP: Destination address = 172.20.4.182, proto182.Central.Sun.COM76 IP: No options77 IP:78 ICMP: ----- ICMP Header -----79 ICMP:80 ICMP: Type = 0 (Echo reply)81 ICMP: Code = 0 (ID: 7170 Sequence number: 60558)82 ICMP: Checksum = 606083 ICMP:84Note – Currently, the snoop command cannot watch multiple interfaces atonce. To make sure that you capture all packets destined for an IP address,you must have multiple snoop commands running. Tracing Routing With the netstat CommandYou can use netstat and route to verify load spreading in the SolarisOE.1. First, verify the Transmission Control Protocol (TCP) connectionsinto the system that you are examining. In this case, examine thetelnet connections into the host:1 # netstat -n23 TCP: IPv44 Local Address Remote Address Swind Send-Q Rwind Recv-Q

State5 -------------------- -------------------- ----- ------ ----- ------ -------6 172.20.4.181.513 129.147.48.184.532 24820 0 24820 0ESTABLISHED7 172.20.4.181.23 129.147.28.54.35430 24820 0 24820 0ESTABLISHED8 172.20.4.181.23 172.20.4.170.33065 24820 0 24820 0ESTABLISHED9 172.20.4.181.23 172.20.13.122.32863 8760 1 24820 0ESTABLISHEDLines 7, 8, and 9 are the telnet connections into the host. You cansee this because it is on port number 23 and the fact that it is on thelocal address.2. Examine the routing tables using the following command:10 # netstat -ran | grep UHA11 129.147.62.15 172.20.4.254 UHA 1 18 qfe012 172.20.4.254 -- UHA 1 788 qfe013 172.20.4.254 -- UHA 1 788 hme014 129.147.48.184 172.20.4.254 UHA 1 1 qfe015 172.20.13.122 172.20.4.254 UHA 2 100 qfe0Tracing Routing With the netstat CommandThe 172.20.13.122 remote host is using the qfe0 interface, whichindicates that multipathing is working. Another way to verify thatmultipathing is working is to use the following command:16 # route -n get 172.20.13.12217 route to: 172.20.13.12218 destination: default19 mask: default20 gateway: 172.20.4.25421 interface: qfe022 flags: <UP,GATEWAY,DONE,STATIC>23 recvpipe sendpipe ssthresh rtt,ms rttvar,ms hopcount mtuexpire24 0 0 0 0 0 0 15000Line 21 show IP is using the qfe0 interface to communicate with the172.20.13.122 remote host.Self-Test