BRKCOM-3001 (1)

7/22/2019 BRKCOM-3001 (1)

http://slidepdf.com/reader/full/brkcom-3001-1 1/103

BRKCOM-3001

Troubleshooting the Cisco UCSCompute Deployment

www.ciscolivevirtual.com

http://www.ciscolivevirtual.com/


7/22/2019 BRKCOM-3001 (1)


© 2012 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCOM-3001

Agenda

Overview

Physical Setup

Software Setup

Configuration

Path Trace

Maintenance

Key Takeaways

Q & A

7/22/2019 BRKCOM-3001 (1)


Overview

7/22/2019 BRKCOM-3001 (1)



UCS Building Blocks

UCS ManagerEmbedded – manages entire system

UCS Fabric InterconnectNexus Switch

UCS Fabric ExtenderRemote line card

UCS Blade Server ChassisFlexible bay configurations

UCS Blade ServerIndustry-standard architecture

UCS Virtual Adapters

Choice of multiple adapters

7/22/2019 BRKCOM-3001 (1)

http://slidepdf.com/reader/full/brkcom-3001-1 5/103© 2012 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCOM-3001

7/22/2019 BRKCOM-3001 (1)


UCS Manager 2.0 Release

Hardware

UCS 6248UP Fabric InterconnectUCS 2208XP IOM

UCS 1280 VIC

Features

Unified Ports

80 Gb Host connectivityFabric port-channeling

L2 disjoint upstream in EHM

VM-FEX for Red Hat KVM

iSCSI Boot

7/22/2019 BRKCOM-3001 (1)


Physical Setup

7/22/2019 BRKCOM-3001 (1)


UCS Fabric Interconnect

Standalone or Clustered

Primary / Subordinate

Data Management Engine (DME)

FI-A#

Virtual IP

IP #BIP #A

Management Network

Cluster links

DB

7/22/2019 BRKCOM-3001 (1)


IO Module HIF to NIF Pinning (2104X

Fabric Interconnect1 link

2 links

4 links

Server slots pinn

Uplink: slots 1,2

Uplink 1: slots

Uplink 2: slots 2

Uplink 1: slots

Uplink 2: slots 2

Uplink 3: slots 3

Uplink 4: slots 4

slot 1

slot 2slot 3slot 4slot 5slot 6slot 7slot 8

slot 1slot 2slot 3slot 4

slot 5slot 6slot 7slot 8

slot 1slot 2slot 3slot 4slot 5slot 6slot 7

slot 8

NIF

NIF

NIF

F

E

X

F

EX

Fabric Interconnect

Fabric Interconnect

FE

X

HIF

7/22/2019 BRKCOM-3001 (1)



1-4

5-8

9-12

13-16

17-20

21-24

25-28

29-32

FEX Fabric

Interconnect

Slot 1

Slot 2

Slot 3

Slot 4

Slot 5

Slot 6

Slot 7

Slot 8

7/22/2019 BRKCOM-3001 (1)




FEX Fabric

Interconnect

29-32

1-4

5-8

9-12

13-16

17-20

21-24

25-28

Slot 1

Slot 2

Slot 3

Slot 4

Slot 5

Slot 6

Slot 7

Slot 8

7/22/2019 BRKCOM-3001 (1)




FEX Fabric

Interconnect

29-32

1-4

5-8

9-12

13-16

17-20

21-24

25-28

Slot 1

Slot 2

Slot 3

Slot 4

Slot 5

Slot 6

Slot 7

Slot 8

7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)



IO Module and Failover

IO

Blade1 Blade2 Blade3 Blade4 Blade5 Blade6 Blade7 B

4

1 2 3 4

Lose IOM link 1

Lose connectivity on

mezzanine port mapped

to IOM 1 for blades 1

and 5

Blades 2,3,4,6,7,8 w

continue to use all 3

links

Blades 1 and 5 will

have connectivity o

the mezzanine port

connected to IOM 2

If you re-acknowledge

we will fall back to a 2

topology, and blades

use one uplink, and b

will use a different up

Note: Only a re-ack

required, not a chas

discover policy cha

Which of the 2 links

of the remaining 3 i

“random” decision.

7/22/2019 BRKCOM-3001 (1)



Port-Channel Pinning

No slot based pinning

No invalid link count for NIF ports

Pinned

to Po

IOMVIC1280

adaptor with

DCE links inPort-Channel

Gen-1 adaptor

with single 10G

link

7/22/2019 BRKCOM-3001 (1)



Chassis Discovery Policy

Discovery policy only defines the minimum nu

of links necessary before a chassis can bediscovered and NOT how many links will be ut

7/22/2019 BRKCOM-3001 (1)


Software Setup

7/22/2019 BRKCOM-3001 (1)


Upgrade

7/22/2019 BRKCOM-3001 (1)



Upgrade Considerations

Release Notes will cover pre-requisites and concerns in

upgrade process

Schedule an outage window

FI and IOM will reboot during upgrade

Make sure network and storage fabric are redundant

Highly recommended to backup UCSM configuration

Upgrade process is not quick so be patient

Follow the upgrade procedure for each version

7/22/2019 BRKCOM-3001 (1)



Upgrade Process

Backup UCS Config (Full & All Config)

Download firmware

Update components (adapters, CIMC, IOMs)

Activate components in order of:

Adapter cards – Set Startup Only

CIMC

UCSM

IOM – Set Startup Only

Fabric Interconnect

BIOS/Storage Controller/etc (through firmware package)

7/22/2019 BRKCOM-3001 (1)


Fabric Interconnect Modes

7/22/2019 BRKCOM-3001 (1)



FI Ethernet Modes: End Host

• No Spanning Tree

Protocol

• Maintains MAC tabServers only

• Allows Multiple Acti

Uplinks per VLAN

• Prevents Loops bypreventing Uplink-tUplink switching

Server 1

VNIC 0

Server 2

VNIC 0

Fabric A

L2

Switching

6x00 A

MAC

Learning

MAC

Learning

VLAN 10

Spanning

Tree

LAN

vEth 3 vEth 1

7/22/2019 BRKCOM-3001 (1)



VLAN 10

End Host Mode Unicast Forwarding

• Server to server traffic on thesame VLAN is locally switched

• Each server link is pinned to anuplink port / port-channel

• Network to server unicast traffic isforwarded to server only if itarrives on pinned uplink port. Thisis termed as the Reverse PathForwarding—(RPF) check

• Packet with source MACbelonging to a server received onan uplink port is dropped (Deja-VuCheck)

Unknown unicast traffic is dropped

UplinkPorts

6x00

Deja

vEth 1

S

VNIC 0

Server 2

LAN

7/22/2019 BRKCOM-3001 (1)



Uplink

Ports

6x00

End Host Mode Broadcast/Multicast For

vEth 1

S

VNIC 0

Server 2

LAN

Broadcast

Listener All VLANs

B

B Pre 2.0 - Broadcast traffic is

pinned on exactly oneuplink port (or port-channel)i.e., it is dropped whenreceived on other uplinks

All multicast groups are pinned

to same uplink port

Server to server multicasttraffic is locally switched

RPF and deja-vu check alsoapplies for multicast traffic

FI Ethernet Modes: Switch

7/22/2019 BRKCOM-3001 (1)



FI Ethernet Modes: Switch

Server 1

VNIC 0

Server 2

VNIC 0

Fabric A

L2

Switching

6x00 A MAC

Learning

VLAN 10

LAN

vEth 3 vEth 1

Root • FI acts as L2 switch

• Server vNIC traffic follows

forwarding

• Rapid PVST+ is used

• Configuration of STP parapriority, Hello Timers etc) supported

• MAC learning/aging happserver and uplink ports likLayer 2 switch

7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)



Unsupported Storage Topologies Direct connect without zoning (no upstream

MDS/N5K)

Multi-Hop FCoE or FCoE “northbound” on EthUplinks

FC connections between the two FabricInterconnects running in FC switch mode (ISL

Connecting any other L2 based object off of th Appliance Ports

7/22/2019 BRKCOM-3001 (1)


Disjoint Layer 2

7/22/2019 BRKCOM-3001 (1)



End Host Mode – Disjoint L2

SW1 SW2 SW3 SW4

7/22/2019 BRKCOM-3001 (1)



Production

VLAN 10-20

Backup

VLAN 30-40

Fabric Interconnect - A

Eth 1Eth 0

Fabric Interconnect - B

Bare Metal

OS

Production

VLAN 10-20 V


EtEth 0

Bare Meta

OS

L2 Switch -A L2 Prune VLANs

Switch Mode End Host Mode

Disjoint L2 – Pre 2.0 Workarounds

7/22/2019 BRKCOM-3001 (1)



EHM Disjoint L2 Upstream >= 2.0

Hardware independent: 6100 & 6200 supportthis feature with UCS 2.0 and above

The ability to selectively assign VLANs touplinks

vNIC pinning decision based on - VLANmembership on uplink ports

Allocating a designated broadcast/multicastreceiver on a per VLAN rather than globalbasis

Production

VLAN 10-20


Eth 0

Bare Met

OS

7/22/2019 BRKCOM-3001 (1)



Disjoint L2 – Do NOT

1. Configure different range of

VLANs across uplinks of adisjoint network

2. Overlap VLANs across L2networks

3. Configure vNIC with VLANsacross multiple L2 networks.vNIC does not get pinned ifthe uplink vlan membership ispartial, vNIC stays down Half Width Bl

IOM-A

Adapter

v N I C - A

v N I C - B


End Host Mode

Production

VLAN 10-20

10 -20 11,13,15

VLAN 10,30

1

3

Production

VLAN 10-20

7/22/2019 BRKCOM-3001 (1)



VLAN Designated Receiver

1.4 = Connect nxos : show platform software e

internal info global | grep - A 6 'Global Params’

2.0 = Connect nxos : show platform software einternal info vlandb id <vlanID>

7/22/2019 BRKCOM-3001 (1)



Disjoint L2 Verification Overview

From UCSM CLI – NXOS Context:

UCS-CiscoLive(NXOS)# show interface trunk

Should expect to see your associated VLANs the assigned interfaces only.

Port Vlans Allowed on Trunk

---------------------------------------------------

Eth1/16 1,5-7,1000,3000 Only Eth1/16 allow

Eth1/17 1,6-7,1000,3000

<snip>

7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)



Finite State Machine (FSM)

FSM runs as a workflow (FSM-Stages)

Workflows are predefined -stages can be skipnot needed (in HA if remote is down, no vHBA configu

FSM Flags (shallow checkpoint vs deep checkpoint)

Each Stage is an interaction between DME

Application Gateway -> End Point

FSM D il

7/22/2019 BRKCOM-3001 (1)



Error Description for that stage Stage DescriOperation (workflow)

FSM Details

Bl d I

7/22/2019 BRKCOM-3001 (1)



Blade Issues

Server Blade discovery failed

KVM connectivity

Hardware problems

Adapter issues

7/22/2019 BRKCOM-3001 (1)


Configuration

C fi ti V lid ti & E

7/22/2019 BRKCOM-3001 (1)



Configuration Validation & Errors

7/22/2019 BRKCOM-3001 (1)


Configuration Boot issues

7/22/2019 BRKCOM-3001 (1)


Boot from SAN

SAN Boot in UCS

7/22/2019 BRKCOM-3001 (1)



SAN Boot in UCS

Must be defined in the boot policy applied to th

service profile

SAN boot and Local HDD are mutually exclusthe boot policy

During SAN boot, the local HDDs are disabled

Some O/S will not install if more than one pathLUN is seen – check O/S installation guide

7/22/2019 BRKCOM-3001 (1)


SAN Boot Failure Checklist

7/22/2019 BRKCOM-3001 (1)



SAN Boot Failure Checklist

SAN configuration errors Zoning

Host and Target WWPNsService Profile fc0/fc1

SAN boot order configuration

Zones and zonesets

Matching VSANs

Lun Masking

Check that the access control configuration allows the host WWPNaccess the array WWPN

LunID value is correct

SAN Boot Policy

7/22/2019 BRKCOM-3001 (1)



SAN Boot Policy

SAN Failures Unable to locate Lun

7/22/2019 BRKCOM-3001 (1)



SAN Failures – Unable to locate Lun

Both adapters (0 and 1) can’t find the configur

LunID

ROM BIOS fails and reports to setup there is no Sdevice to boot from

SAN Failures Unable to locate Lun

7/22/2019 BRKCOM-3001 (1)




In Option ROM confirm that adapter was enab

by boot policy

In Option ROM confirm that adapter Boot wasenabled and check target + LunID


7/22/2019 BRKCOM-3001 (1)




In Option ROM scan the fabric for visible LUN

Notice how the LunID configured is not seen, another is

7/22/2019 BRKCOM-3001 (1)


iSCSI Boot

iSCSI Boot Support

7/22/2019 BRKCOM-3001 (1)



iSCSI Boot Support

iSCSI boot configuration is added to UCSM 2.0(1)

Allows stateless model with no FC or FCoE in the environment

Limited adapter support

• Cisco VIC and VIC-1280 (iBFT only, no iSCSI offloads)

• Broadcom 57711 M51KR-B (full offload, iSCSI HBA)

Operating System support

• ESXi 4.1U1, 4.1U2, 5.0

• Windows 2008 R2 and R2 SP1

• RHEL 5.6, 6.0 and 6.1

iSCSI Hardware Offload is not a requirement to support booting,supporting iSCSI Boot Firmware Table (iBFT) in the option ROM

iSCSI Setup Flow

7/22/2019 BRKCOM-3001 (1)



iSCSI Setup Flow

Create iSCSI vNICs and iSCSI Boot policy

Provide UCSM with iSCSI boot information

Target IP, IQN

Initiator IP/Mask/Gateway, IQN

Broadcom only – Modify adapter policy for install

vMedia map the OS and drivers if required

Adapter successfully initialises

Install OS and Drivers (if required)

Broadcom only – Modify adapter policy for boot

iSCSI Boot Service Policy – iSCSI v

7/22/2019 BRKCOM-3001 (1)



iSCSI Boot Service Policy iSCSI v

Overlay vNIC

iSCSI adapter policy Identify native VLAN

MAC address:

M81KR – none (do not set)

M51KR-B – must be set(manual or pool)

iSCSI Boot Service Policy – Boot Po

7/22/2019 BRKCOM-3001 (1)



iSCSI Boot Service Policy Boot Po

Set the iSCSI parameters – Select the first iSCdevices (you can have multiple iSCSI initiators

7/22/2019 BRKCOM-3001 (1)


Cisco M81KR VIC Verification

7/22/2019 BRKCOM-3001 (1)



7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)


Ethernet troubleshooting andtracing

7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)


Service Profile Interfaces

7/22/2019 BRKCOM-3001 (1)



show service-profile circuit server

<chassis#>/<slot#>

VIF Identities

7/22/2019 BRKCOM-3001 (1)



Show host-eth-if

Port Information – M81KR Adapter

7/22/2019 BRKCOM-3001 (1)



MCP level : vnic

Port Information – M81KR Adapter

7/22/2019 BRKCOM-3001 (1)



If too many vNICs, can find correct interface w

‘vnicfind’

7/22/2019 BRKCOM-3001 (1)


FEX Port Information

7/22/2019 BRKCOM-3001 (1)



Connect nxos : show fex <chassis#> detail

FEX Ports Information

7/22/2019 BRKCOM-3001 (1)



2.0 = show system internal fex info satport eth

<chassis#>/<adapter#>/<slot#>

1.4 = show platform software fex info satport ethernet<chassis#>/<adapter#>/<slot#>

IOM Ports Information

7/22/2019 BRKCOM-3001 (1)



Connect iom <chassis #>

show platform software <redwood|woodside> sts

IOM Ports Information

7/22/2019 BRKCOM-3001 (1)



show platform software <redwood|woodside>

7/22/2019 BRKCOM-3001 (1)


VIF Information

7/22/2019 BRKCOM-3001 (1)



Connect nxos : show interface brief

Show interface vethernet <id>

MAC Information

7/22/2019 BRKCOM-3001 (1)



show mac address-table

VIF Pinning

7/22/2019 BRKCOM-3001 (1)



show pinning border-interface

show pinning server-interfaces

Port Channel Information

7/22/2019 BRKCOM-3001 (1)



show port-channel summary

Port Channel Information

7/22/2019 BRKCOM-3001 (1)



show port-channel load-balance forwarding-pa

interface port-channel <#> ...

VLAN Designated Receiver

7/22/2019 BRKCOM-3001 (1)



2.0 = show platform software enm internal info

vlandb id <vlanID>

1.4 = show platform software enm internal info global | grep -A 6 'GloParams’

7/22/2019 BRKCOM-3001 (1)


SAN troubleshooting and trac

Service Profile Interfaces

7/22/2019 BRKCOM-3001 (1)



show service-profile circuit server

<chassis#>/<slot#>

VIF Identities

7/22/2019 BRKCOM-3001 (1)



Show host-fc-if

FC Interfaces – M81KR

7/22/2019 BRKCOM-3001 (1)



Attaching to the fls of the adapter provides the

ability to view information related to SAN bootattributes

The vnic command provides us with numberinassociated with vNIC interfaces

FC Login Information – M81KR

7/22/2019 BRKCOM-3001 (1)



Login information (Target PWWN and FCID) fo

each port can be obtained with the login <vniccommand:

FC Login Information – M81KR

7/22/2019 BRKCOM-3001 (1)



The lunmap command will confirm if the interfhave logged into the target or not

7/22/2019 BRKCOM-3001 (1)


FC Statistics

7/22/2019 BRKCOM-3001 (1)



VIF Information


7/22/2019 BRKCOM-3001 (1)




Show interface vfc <id>

VFC Membership and Login

7/22/2019 BRKCOM-3001 (1)



show vsan membership

show npv flogi-table

7/22/2019 BRKCOM-3001 (1)


7/22/2019 BRKCOM-3001 (1)


Maintenance

System Event LogsChassis

7/22/2019 BRKCOM-3001 (1)



Server

SEL Configuration

7/22/2019 BRKCOM-3001 (1)



Users can define rules (policies) for backing uclearing SEL across all servers in the UCS sys

Events LifecycleFaults Lifecycle

UCSM – Faults and Events Lifecycle

7/22/2019 BRKCOM-3001 (1)



Events Lifecycley

Clear Condition

Retention Interval

or Acknowledged

Clear

Action

Retain

Delete

Raise Fault Clear Fault

Delete Fault

Create Event

Delete Event

7/22/2019 BRKCOM-3001 (1)



Data Gathering for Support—UCS

7/22/2019 BRKCOM-3001 (1)



7/22/2019 BRKCOM-3001 (1)


O/S Troubleshooting Tools

7/22/2019 BRKCOM-3001 (1)



Ping, mping, arping

Arp Esxcfg-nics, esxcfg-vmknic, ifconfig, ipconfig

Ethtool or lsmod or lspci or vmkload_mod or modin

Esxtop, htop

Tcpdump, Wireshark

O/S Logs (/var/log/messages or event viewer)

O/S Core Files (BSOD or PSOD)

7/22/2019 BRKCOM-3001 (1)


Key Takeaways

Narrowing Down the Problem

7/22/2019 BRKCOM-3001 (1)



Define the problem

From which point to what other point is the problem?

Do we see the problem in one direction or both?

Eliminate variables

Is the problem seen between traffic traversing the samfabric?

Is the problem only happening on a specific fabric pat

List all the ports in the traffic path

VIFs, HIFs, NIFs, Server ports and Uplink ports

What to do next?

7/22/2019 BRKCOM-3001 (1)



Ask questions on Cisco Support Community

Review documents at the links provided in theappendix section

Raise TAC cases if you encounter issues

If interested, speak with your Account Team ab

participating in Beta programs. Alternatively, cspeak to me afterwards or email me:[email protected]

7/22/2019 BRKCOM-3001 (1)


Q & A

Complete Your Online SessionEvaluation

7/22/2019 BRKCOM-3001 (1)



Complete your session evaluation:

Directly from your mobile device by visitingwww.ciscoliveaustralia.com/mobile and loginby entering your username and password

Visit one of the Cisco Live internetstations located throughout the venue

Open a browser on your own computerto access the Cisco Live onsite portal Don’t forget to activate

Virtual account for acce

materials, communities

live activities throughou

account at any internet

www.ciscolivevirtual.co

http://www.ciscoliveaustralia.com/mobile



http://www.ciscoliveaustralia.com/mobile

7/22/2019 BRKCOM-3001 (1)



BRKCOM-3001 (1)

Documents