Top Banner

of 38

VMware vSphere 5 on Cisco UCS and EMC VNXe

Jun 03, 2018

Download

Documents

dktyler
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    1/38

    VMware vSphere 5

    on Cisco UCS and

    EMC VNXeTechnical Integration Howto

    Kenny Garreau

    @kennega

    10/28/2012

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    2/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    1

    Table of Contents

    Introduction ............................................................................................................................................ 2

    Intended Audience .............................................................................................................................. 2

    Special Thanks ..................................................................................................................................... 2

    Vendor Products...................................................................................................................................... 3

    VMware vSphere 5 .............................................................................................................................. 3

    Cisco UCS B-series Servers ................................................................................................................... 3

    EMC VNXe Unified Storage .................................................................................................................. 3

    Implementation....................................................................................................................................... 4

    Architecture ........................................................................................................................................ 4

    Challenges ........................................................................................................................................... 5

    Best Practices ...................................................................................................................................... 7

    Cisco UCS Configuration .......................................................................................................................... 7

    Create Pools and Policies ..................................................................................................................... 7

    Create vNIC Templates ........................................................................................................................ 9

    Create Service Profile Template ......................................................................................................... 10

    Deploy and Modify Service Profiles .................................................................................................... 17

    EMC VNXe Configuration ....................................................................................................................... 18

    Add Generic iSCSI Storage.................................................................................................................. 18

    vSphere Installation ............................................................................................................................... 24

    ESXi Network Configuration ............................................................................................................... 28

    ESXi iSCSI Configuration ..................................................................................................................... 30

    EMC VNXe Configuration Redux ............................................................................................................ 32

    Add VMware Storage ......................................................................................................................... 32

    Conclusion ............................................................................................................................................. 36

    References ............................................................................................................................................ 37

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    3/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    2

    Introduction

    Being in the technical solutions and integration space will always present challenges, and the more

    technologies involved in that integration, the greater the challenge. From a technological perspective,

    the protocols and products in this whitepaper arent the most advanced in the market, but I found some

    unique issues that arose during the work. Many of the troubleshooting steps I followed that were

    outlined on the Cisco, EMC and VMware forums didnt apply, or ended up being red herrings to the true

    problem. I spent a lot of time sifting through reference documentation from each vendor, and

    unfortunately the trusty Google search for EMC VNXe UCS yielded less fruitful information than I had

    hoped to find. That is why I decided to pen this whitepaper.

    Intended Audience

    This document is meant to provide specific technical guidance with regards to deploying VMware

    vSphere 5 in an environment backed by Cisco UCS B-series blade servers and EMCs VNXe storage

    offering. These are a collection of best practices from Cisco, VMware and EMC, as well as my

    professional recommendations and experiences from several of the deployments I have been charged

    with.

    The knowledge level assumed is the ability to deploy all three technologies in at least an SMB

    environment. Standard setup tasks such as rack and stack, array/chassis cabling and basic setup and

    initialization procedures are outside the scope of this document.

    Special Thanks

    Before I get started, putting this together would not have been possible without the help of a couple

    good friends.

    Jason Goldrick (@J_Goldrick)for all your help with the VNXe minutiae.

    Anderson Nichols (@badrockjones)the best technical writer I know!

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    4/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    3

    Vendor Products

    Covered in this whitepaper is the integration of the following technologies:

    VMware vSphere 5 Cisco UCS B-series Servers EMC VNXe Unified Storage

    VMware vSphere 5

    The deployment here utilized VMware vSphere 5.0, with ESXi 5.0 U1 hosts deployed from Ciscos

    vendor-specific ISO. This is available from VMwares Downloads site, under the vSphere 5.0 download

    page, under Drivers and Tools.

    vCenter 5.0 U1b was installed and configured as the management platform for the underlying ESXi

    hypervisor hosts.

    Cisco UCS B-series Servers

    Im a huge advocate of Ciscos UCS blade servers. They are flexible, scalable and highly customizable.

    However, Cisco assumes a certain level of knowledge when using all of its products, and UCS is no

    different. The platform is extremely powerful, but you will need to know what you are configuring and

    more importantly, whyyoure configuring it in said fashion. The level of control you have over even the

    smallest parts or policies within the system, Service Profile mobility, as well as a low entry cost into 10

    Gigabit networking makes it the best server platform on the market.

    Components used in this whitepaper are Ciscos 5108 UCS blade chassis, 6248UP Fabric Interconnects,

    2208XP Fabric Extenders, B200 M3 blade servers and the Cisco VIC1280 mezzanine card.

    EMC VNXe Unified Storage

    For SMB customers looking to deploy an entry-level unified storage platform, EMC offers the VNXe

    platform in two models, the VNXe 3150 and the VNXe 3300. Configuration options will vary based on the

    customers storage, performance and connectivity requirements. Packing advanced storage features

    such as Flash drives, multi-protocol support and 10GbE connectivity, it is a fantastic entry-level platform

    for the SMB market. This installation included 10GbE iSCSI modules for block storage.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    5/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    4

    Implementation

    Architecture

    Now that weve covered the key components involved, lets discuss how all of this is going to fit

    together. Keep in mind that most SMB customers who have purchased this type of solution arent going

    to have an existing 10GbE switching infrastructure. This will be their first exposure to 10Gb, and this may

    even be their first time using a dedicated storage or virtualization platform! There are lots of moving

    parts, so lets cover the basics.

    We have four 10GbE SFP+ Twinax cables going from each 2208 Fabric Extender (FEX) to its

    corresponding 6248UP Fabric Interconnect (FI). The customer does NOT have an existing 10GbE

    switching infrastructure, so in this scenario we have opted to utilize UCS Appliance Ports to connect the

    VNXes 10GbE iSCSI modules. Cisco UCS bundles will typically include four SFP-10GB-SRs, which are

    10GbE optical SFPs youll need for connectivity between those 10GbE modules in the VNXe and the6248 FIs1. Two appliance ports will be configured per FI, so you should cable module ports A-0 and B-1

    to FI-A, then module ports A-1 and B-0 to FI-B.

    This customer had a single-switch core, and many SMBs will have a similar setup, or perhaps a stack of

    Cisco Catalyst 3750 switches. Connectivity choices upstream can vary wildly, but a quick Google search

    for Cisco UCS Networking Best Practices will yield plenty of results2. For this install, the customer has

    two two-port 1GbE LACP bundles from each FI into a single switch, so we acquired four GLC-T GBICs and

    placed two into each FI and configured two LACP port-channels within UCS Manager.

    In order to enable Service Profile mobility, each blade server has a VIC1280 mezzanine card installed

    with no local drives. We will boot each Service Profile (ESXi host) from an iSCSI LUN presented via the

    VNXe, and shared iSCSI datastores for VMware virtual machine storage will be presented to each host

    and formatted with VMFS-5.

    The following diagram shows the Visio layout of how weve connected the components:

    1You may be able to unplug the 10GbE SFPs in the VNXes modules and use 10GbE SFP+ Twinax cabling, but I have

    not tested this. EMC VNX requires Active Twinax cabling, which may also be a requirement for the VNXe. Hit me up

    on Twitter (@kennega) if youve tested this!2See also the References section of this document.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    6/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    5

    Challenges

    Throughout the course of testing and validating this solution, I ran into quite a few issues. As of now

    they seem quite simple and obvious, but during the integration and deployment, things arent always so

    clear.

    The biggest challenge with this V+C+E integration is the lack of Storage Groups within the VNXe. In a

    VNX, LUNs are assigned to a storage group, and that storage group is in turn masked to a host.

    Therefore, masking a boot LUN ID of 0 to each host is uniform throughout the environment.

    The VNXe however, has no concept of Storage Groups. LUNs are assigned at the iSCSI Server level within

    the system, at which point you assign Virtual Disk or Datastore access to a Host. Its a slightly different

    way of thinking than with a mid-range or enterprise storage array. While this impacts how you end up

    deploying and organizing your storage, it also has an effect on your UCS Service Profiles and Service

    Profile Templates.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    7/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    6

    In a VNX-backed UCS deployment on FC or iSCSI, the array storage group decides how to mask LUN IDs.

    Therefore each Service Profile (host) could be assigned a 5GB boot LUN within the storage group.

    Service Profiles can always stay bound to, and in compliance with, a Service Profile Template within UCS,

    since LUN 0 will always be the boot LUN. This is not so with the VNXe. Since the iSCSI Server will assign

    LUN IDs in ascending order, the Service Profiles will have to be unbound from the Service Profile

    Template in order to modify the iSCSI Boot Parameters.

    Theres also an order of operations to be followed that I will outline shortly. Its certainly not efficient by

    any means, but its something youll need to take into consideration for properly automated

    VNXe/vSphere integration.

    Another thing to keep in mind is that, as an integrator, when you leave the customer site, youll have to

    be aware that if and when the customer purchases a new blade or set of blades to install into the

    chassis, the above considerations should be outlined to them, or scoped into any future professional

    services engagement. The Service Profile Templates will get them 95% of the way there with the VNXe,

    but that last 5% will come in the form of unbinding the new Service Profiles and editing the iSCSI Boot

    Parameters under the Boot Order tab of the Service Profile.

    Finally, be sure you size your pools to at least the capacity of the UCS chassis youre deploying. You

    could create your MAC/IQN/Initiator IP pools with a size of 4, but why not size it to the capacity of the

    chassis? This will make future expansion a plug and play operation until you fill the capacity of that

    chassis.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    8/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    7

    Best Practices

    The following best practices are followed as part of this implementation:

    Dedicated storage VLANs for iSCSI fabrics; one for A side, one for B side. Differing subnets in each storage VLAN. In this deployment, we utilize 172.16.10.0/24 and

    172.16.11.0/24

    LUN masking at the array for boot LUNs and vSphere datastores Do not use link aggregation for any iSCSI interfaces iSCSI vmkernel port binding within vSphere iSCSI vmkernel port groups are contained within a dedicated vSwitch Balanced storage presentation from iSCSI Server A and iSCSI Server B All boot LUNs presented from one iSCSI Server Boot LUNs should always utilize the lowest tier of storage available iSCSI Overlay vNICs do not utilize a MAC address (inherited from parent vNIC) Do not enable Fabric Failover for iSCSI-bound vNICs All iSCSI connections are set as access ports to their respective iSCSI VLANs Jumbo frames are enabled end-to-end Utilize Round Robin PSP within vSphere for storage multipathing

    Cisco UCS Configuration

    From this point forward, I assume that all racking, stacking, cabling, uplinks, hardware/software

    initialization and storage pool creation have been completed. At this stage, the environment is ready to

    have Service Profile Templates and Services Profiles configured, boot LUNs to be created and assigned

    to initiators, and for ESXi to be installed on the boot LUN.

    Create Pools and Policies

    Create the following pools within UCS Manager: Management IP, iSCSI Initiator IP, IQN, MAC, Server,

    UUID. For MAC pools, I recommend creating several MAC pools, and in my example here I create four

    MAC pools. One pool each for iSCSI vNICs on FI-A, iSCSI vNICs on FI-B, SC/VM vNICs on FI-A and SC/VMvNICs on FI-B. This is extremely useful in troubleshooting connectivity issues!

    Under the LAN tab, expand PoliciesAppliancesNetwork Control Policies, right-click and select Create

    Network Control Policy. Name the policy something descriptive, in my example Ive called it iSCSI_CDP.

    Choose to Enable CDP, and leave all other settings at their default. Click OK:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    9/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    8

    Create your Appliance Interfaces, and be sure to set them as Access ports to the appropriate iSCSI VLAN.

    For ease, Ive created a VLAN label iSCSI and assigned VLAN 500 on FI-A and VLAN 501 on FI-B.

    Remember, youll need to create your iSCSI VLANs in two places: under Appliances Fabric A/BVLANs,

    and under LANLAN CloudFabric A/BVLANs. While creating the Appliance Interfaces, be sure to

    assign the Network Control Policy you just created:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    10/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    9

    Under the LAN tab, under PoliciesrootNetwork Control Policies, right-click and Create Network

    Control Policy. Wait... Didnt we do this above? Sort of. We created a Network Control Policy beforefor

    our Appliance Ports. Thispolicy were creating will be applied to our vNIC Templates were about to

    create. Set the same options as before however: Enable CDP, leave all others at their defaults. Give it adescriptive name and click OK.

    Create vNIC Templates

    Next well be creating vNIC Templates. For an SMB customer, I generally recommend the following vNIC

    layout:

    One vNIC per Fabric for Service Console and vMotion traffic One vNIC per Fabric for Virtual Machine traffic One vNIC per Fabric for iSCSI traffic (do not enable Fabric Failover for these vNIC Templates!)

    Create your six vNIC Templates, assign the appropriate VLANs, and be sure you set MTU 9000 for your

    iSCSI vNIC Templates. Unfortunately I didnt get a screen grab of my vNIC Templates as an example, but I

    did get a screen cap of the vNICs within the Service Profile Templates as they were built from the vNIC

    Connectivity Templates:

    Youll see above that this iSCSI vNIC within the Service Profile Template is bound to my iSCSI_A LAN

    template, is owned by Fabric A WITHOUT Failover, has an MTU of 9000, Adapter Policy set to VMware

    and Network Control Policy set to my CDP-enabled NCP I created earlier.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    11/38

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    12/38

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    13/38

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    14/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    13

    Click OK, click OK, and click Next.

    Under the vNIC/vHBA Placement screen, I recommend moving the iSCSI vNICs to last in the placement

    order. If you leave the iSCSI vNICs as first in the placement order, they will be assigned as vmnic0 and

    vmnic1 in the vmkernel. This in itself is not a bad thing until you attempt to load ESXi onto the LUN you

    assign to the Service Profile. At the end of the install process, ESXi will attempt to re-initialize vmnic0 as

    DHCP and the Service Profile will lose IP connectivity to the iSCSI target. Youll wind up with an error asfollows:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    15/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    14

    So be safe and change the placement order:

    On the Server Boot Order screen, create a Boot Policy to use. Boot off the CD-ROM first, and then add

    an iSCSI Boot option. Be sure you use the same iSCSI Overlay vNIC name you created earlier. Click OK.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    16/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    15

    Select the boot policy from the drop-down. Select the iSCSI option, and click Set iSCSI Boot Parameters.

    Select a pool for Initiator Name assignment, and set Initiator Addressing to Pool. Now select to Add an

    iSCSI Target Interface. At this point youll need to head over to the VNXe quickly to copy/paste in your

    target IQN. Under SettingsiSCSI Server Settingsyou can select your iSCSI Server and select Details,

    and on this screen youll be able to copy the IQN.

    Paste the IQN of the VNXe iSCSI Server into the Name field, set the IPv4 address to the iSCSI Server A-

    side IP address. The default LUN ID of 0 is acceptable, this will change for the Service Profiles anyway

    when we un-bind them from the Template after deploying.

    Note: I only deploy one iSCSI vNIC at this time. The reason for this is that UCS currently does not have

    the ability to deploy multiple IP pools for iSCSI vNICs. It would be great if this feature was enabled aswed be able to simply create two iSCSI vNICs, bind distinct IPs to each initiator on differentsubnets, and

    VMware would be crafty enough to recognize both of them and assign vmkernel ports to them. Our goal

    is to automate what we can though, so that is why I do not create a second iSCSI vNIC.

    Click OK, and your screen should look like this:

    Click OK, then click Next.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    17/38

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    18/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    17

    Set any operational policies you wish to set, and click Finish. Your Service Profile Template should be

    created, and at this point you can deploy Service Profiles from this template.

    Deploy and Modify Service Profiles

    At this point you should deploy X number of Service Profiles, where X is the number of blades within the

    chassis that youre deploying. Its easiest to deploy all at once, then you can unbind each individually to

    apply specific iSCSI Boot Parameters to each blade. When deploying, IPs, MACs, IQNs, etc. will all be

    assigned and visible within each Service Profile.

    Deploy your Service Profiles, and then select the first in the list. Right-click and select Unbind from the

    Template. Select the Boot Order tab and select the iSCSI boot at the bottom. Click Set iSCSI Boot

    Parameters. Youll see that you can now modify all of the settings, including Pools and IQNs, but you

    should leave these as-is. Remember, automate!

    Select the Static Target at the bottom and select Modify. Youll be able to change/update the boot LUN

    ID that the iSCSI Server on the VNXe will present. When I am doing a net-new installation for a

    customer, I utilize the first several LUNs created on the VNXe iSCSI Server as my boot LUNs. You can see

    in the following screenshots the difference between Service Profile 1s boot LUN config and Service

    Profile 2s boot LUN config: (note the LUN ID column, and difference in initiator IPs and IQNs)

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    19/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    18

    After you modify these parameters and click OK and Save Settings, youll be prompted with a User

    Acknowledgment to reboot the blade to apply the new parameters. Click Yes, and under Pending

    Activities, acknowledge the reboot. You can do these one at a time for each Service Profile, or edit and

    reboot them all in bulk. Dont close UCS Manager quite yet;well need some information for the VNXe.

    EMC VNXe Configuration

    Add Generic iSCSI Storage

    Well need to add Generic iSCSI Storage at first so that we can provide boot LUNs to the ESXi hosts.

    Creating VMware iSCSI Storage within the VNXe attempts to create VMFS volumes automatically, and

    thats not what were looking to do initially.

    At the Dashboard screen after logging into the VNXe, select Create storage for generic iSCSI. The first

    screen asks you to name the generic storage. Give it a descriptive name; here Ive named it ESX13:

    3Id recommend something a little more descriptive, maybe ESXnBootLUN, where n is the Service Profile number.I

    did a bad job here of clearly delineating between the Host ID and Storage ID.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    20/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    19

    Click Next. Choose a Pool from which to provision the LUN, preferably the pool with the lowest

    performance. Set your LUN size to 5GB, and since you can have two iSCSI Servers, make sure youre

    provisioning from iSCSI Server A:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    21/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    20

    Click Next. Select to Not Configure Protection Storage:

    Click Next. Under Configure Host Access, select Create Host:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    22/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    21

    Specify a name for the host:

    Click Next. Under the Operating System dropdown, select VMware ESX:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    23/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    22

    Click Next. Select IP address, and enter the IP address of the iSCSI initiator that corresponds with this

    host. Youll find that under the Service Profiles iSCSI Boot Parameters that we modified earlier.

    Click Next. Grab the IQN off the iSCSI Boot Parameters property sheet as well, and input that on this

    screen.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    24/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    23

    Click Next. Verify the Summary information and click Finish. Youll return to this page:

    Select Virtual Disk from the Access drop-down. Click Next. Verify the summary information, and click

    Finish.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    25/38

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    26/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    25

    This is actually a bad example as only one port is active currently (Eth1/24), which doesnt have an IP

    assigned to it within the VNXe yet. But a show mac address-table vlan 500 will show the MAC

    membership for your iSCSI VLAN on FI-A. If you are seeing MAC addresses on the opposite FIs that

    youre expecting, you may need to edit your iSCSI Server configuration within the VNXe, or do some

    cable swaps between FIs.

    If all is well, mount your Cisco custom ISO in the Virtual Media tab:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    27/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    26

    After booting from the ISO, youll land at the ESXi Installer welcome screen:

    Run through the install prompts, and when the installer scans for disks, you should see the following:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    28/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    27

    After walking through the rest of the prompts, youll again be prompted to install to your iSCSI boot

    device. Hit F11 to confirm and run the install.

    After the install finishes, the virtual media ISO will be ejected and the Service Profile will reboot.

    Youll need to perform thisprocedure for each Service Profile. If you dont see your boot LUN at first,dont panic. Sometimes its as simple as a Service Profile Reset and on the next boot it will discover the

    target and login successfully.5

    5CLI tools are invaluable for troubleshooting. From the adapter prompt, the iscsi_ping and iscsi_get_config

    commands display a wealth of information. However, they are only available when booting or when the server has

    entered the BIOS screen.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    29/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    28

    ESXi Network Configuration

    You should now complete basic setup of the ESXi host. From the DCUI, configure IP addressing and

    continue basic configuration via the vSphere Client. The applicable configuration items for iSCSI

    vmkernel port binding, network configuration and iSCSI multipathing are represented here via the GUI,

    but you can also complete these tasks via the command line.

    In the vSphere Client, when you connect to the ESXi host and look at the Configuration tab under

    Networking, you should see two vSwitches created for you; vSwitch0 will contain your Management

    Network vmkernel port. The ESXi install will also have created an iScsiBootvSwitch with an iScsiBootPG

    vmkernel port. The following graphic shows this, as well as the fact that Ive created a second Port

    Group and vmkernel port attached to iScsiBootvSwitch, as well as vmnic5, my second iSCSI vNIC.

    A pre-requisite for iSCSI vmkernel port binding is to bind each vmkernel port in the iSCSI vSwitch to its

    own vmnic by overriding the vSwitch Failover order.

    For the iScsiBootPG port group, edit the NIC Teaming properties and select to override the switch

    failover order:

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    30/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    29

    Commit this change and repeat the same process for the other port group, except youll reverse the

    adapter usage:

    At this point youll be able to configure iSCSI vmkernel port binding.6

    6Im not quite sure why these vmnics show a speed of 20000. This usually happens when Fabric Failover is

    enabled, as the NIC technically sees two 10Gb connections. I can assure you FF is not enabled for the vNICs backing

    these vmnics however.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    31/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    30

    ESXi iSCSI Configuration

    To configure iSCSI vmkernel port binding, youll need to edit the vmhba properties for the iSCSI adapter.

    From the vSphere Client, go to ConfigurationStorage Adapters, right-click the iSCSI Software Initiator,

    and select Properties. Select the Network Configuration tab:

    Click Add, and select a vmkernel port to use. Both of the vmkernel ports you created before should show

    as compliant; if they do not, go back and double-check your bindings for the port groups.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    32/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    31

    Click OK to bind one vmkernel port, and then click Add and add the other vmkernel port. Your Network

    Configuration screen should show a summary as follows:

    You can choose to add discovery targets now, but the VNXes vSphere integrationwill actually do all of

    this for you.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    33/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    32

    EMC VNXe Configuration Redux

    Our Service Profile hosts have already been added to the VNXe, but theyve been added as generic

    initiators. To take advantage of the VNXes vSphere integration features, its necessary to delete the

    Generic Host and re-add it as a VMware Host. After that well need to re-add the generic boot LUN to

    the host, as well as configure its shared storage. This should be done while the Service Profile is online. If

    you remove the host and try to re-add it while the server is offline, the operation will fail as this process

    requires an active Management Network connection to the ESXi host.

    Add VMware Storage

    From the VNXe Unisphere window, select Hosts and highlight your first Service Profile. Select Delete to

    delete the Host from the VNXe. This does not delete any storage associated with the Host; it simply

    removes the Host record from the array:

    Navigate to HostsVMware and select Find ESX Hosts.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    34/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    33

    Click Next, and enter the IP address of the Service Profiles Management Network vmkernel port.Then

    click Find.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    35/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    34

    Youll need to then enter credentials for the host7, and click OK.

    At this point the VNXe should find and recognize the host as an ESXi server.

    7If the password is ever changed (or the account deleted if you use a non-root account) youll need to be sure to

    update the credentials for the host(s) affected.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    36/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    35

    Click Next, and click Finish on the Summary screen. You should now see a host listed in the Virtualization

    Hosts list:

    The host has now been re-added to the VNXe, but well need to make sure we re-assign its boot LUN to

    the host. Navigate to StorageGeneric iSCSI Storage and select the boot LUN for the host you just re-

    added. Click Details, and then select Host Access.

  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    37/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    36

    Here youll see a list of available hosts that you can present the LUN to. Notice the difference between

    the newly-created VMware host and the pre-existing Generic hosts. The Generic hosts have only

    registered a single initiator IP, whereas the VMware host has registered three. Two will be your iSCSI

    vmkernel ports, and one will be the Management Network vmkernel port.8

    Select the Access drop-down for the host youll be addingthe disk to, and select Virtual Disk. Click ApplyChanges. That is the only modification required for re-adding the boot LUN. Remember, there are no

    Storage Groupsthe LUNs are assigned per iSCSI Server, so the LUN ID will never change.

    At this point, youll be able to add shared datastores to your ESXi hosts by selecting Create Storage for

    VMware from the Dashboard. This is a well-covered topic in EMCs documentation, as well as on several

    blogs and YouTube, so for the sake of brevity Ill leave thoseinstructions to their docs.9

    Conclusion

    I hope this document is of assistance to anyone looking to virtualize with VMware vSphere on Cisco

    Unified Compute Servers backed by EMCs VNXe on iSCSI. If you have any feedback or questions, or just

    want to give a shout out that this was helpful, I can be found on Twitter @kennega, or via my website:

    http://www.dudewheresmycloud.com/

    8This is for reference only. The host will never try to use the Management Network to log in via iSCSI.

    9Depending on the VNXe code revision, the datastores created by the VNXe vSphere engine may create the

    datastores as VMFS-3 instead of VMFS-5. The newest code rev from EMC is supposed to address this.

    http://www.dudewheresmycloud.com/http://www.dudewheresmycloud.com/http://www.dudewheresmycloud.com/
  • 8/12/2019 VMware vSphere 5 on Cisco UCS and EMC VNXe

    38/38

    VMware vSphere 5 on Cisco UCS and EMC VNXe

    References

    EMC VNXe High Availability

    https://community.emc.com/docs/DOC-12551

    Multipathing Configuration for Software iSCSI Using Port Binding

    http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-

    binding.pdf

    Cisco UCS Manager GUI Configuration Guide

    http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Conf

    iguration_Guide_2_0_chapter_011101.html#concept_D7BF302366F24CF5A602B0E0BD18787C

    Cisco UCS Networking Best Practices

    http://bradhedlund.com/2010/06/22/cisco-ucs-networking-best-practices/

    https://community.emc.com/docs/DOC-12551https://community.emc.com/docs/DOC-12551http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdfhttp://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdfhttp://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdfhttp://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_011101.html#concept_D7BF302366F24CF5A602B0E0BD18787Chttp://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_011101.html#concept_D7BF302366F24CF5A602B0E0BD18787Chttp://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_011101.html#concept_D7BF302366F24CF5A602B0E0BD18787Chttp://bradhedlund.com/2010/06/22/cisco-ucs-networking-best-practices/http://bradhedlund.com/2010/06/22/cisco-ucs-networking-best-practices/http://bradhedlund.com/2010/06/22/cisco-ucs-networking-best-practices/http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_011101.html#concept_D7BF302366F24CF5A602B0E0BD18787Chttp://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_011101.html#concept_D7BF302366F24CF5A602B0E0BD18787Chttp://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdfhttp://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdfhttps://community.emc.com/docs/DOC-12551