Enabling VMware Enhanced VMotion Compatibility on … Enhanced VMotion Compatibility on H… · Enabling VMware Enhanced VMotion Compatibility on HP ProLiant servers Technical white

Enabling VMware Enhanced VMotion

Compatibility on HP ProLiant servers

Technical white paper

Table of contents

Executive summary ............................................................................................................................... 2

VMware VMotion overview................................................................................................................... 2 VMotion CPU compatibility ............................................................................................................... 2 Enhanced VMotion Compatibility ....................................................................................................... 3

VMware Enhanced VMotion compatibility requirements ........................................................................... 7 Intel-based HP ProLiant servers allowed in an EVC cluster ..................................................................... 8 AMD Opteron-based HP ProLiant servers allowed in an EVC cluster ....................................................... 9

Summary .......................................................................................................................................... 10

Appendix A – enabling processor virtualization options ......................................................................... 11

Appendix B – migration logs ............................................................................................................... 17

For more information .......................................................................................................................... 18

2

Executive summary

VMware VMotion technology allows running virtual machines to move from one physical machine to

another with no impact to the virtual machines. VMotion offers improved system utilization with load

balancing, increased serviceability and manageability, as well as enhanced flexibility. Administrators

can reduce unplanned downtime and can eliminate planned downtime to perform hardware

maintenance, such as disruptive firmware updates. Successful VMotion migration requires CPU

compatibility between source and destination ESX hosts. This document explains the use of HP

ProLiant servers with VMware Enhanced VMotion Compatibility (EVC) to ensure all hosts in a cluster

are VMotion compatible.

Target audience: The intended audience for this document is VMware administrators who intend to

deploy HP ProLiant servers in EVC clusters, and purchasing managers who wish to add new HP

ProLiant servers to an EVC cluster. It is assumed that you have working knowledge of VMware

Infrastructure 3 (VI3) and/or VMware vSphere 4.

VMware VMotion overview

The VMotion process starts by VMware vCenter performing several checks to verify that the virtual

machine to be migrated is in a stable state on the source host and that the destination host is

compatible. Next, vCenter begins an iterative pre-copying of the memory state of the source guest to

the destination host. The memory pre-copy completes when memory changed is below a given

threshold or no forward progress is made. The amount of time needed to perform the memory pre-

copy depends on the workload, amount of memory and type of network used for VMotion. This step

can happen in seconds, or it can take minutes.

Next the virtual machine is quiesced, and the remaining state is sent to the destination host. At this

point, control is transferred from the source host to the destination host. This step typically takes under

one second.

The last step is to send the remaining modified memory, and start the virtual machine on the

destination host. The amount of time needed for this step depends on workload and memory size.

In the event that the VMotion operation fails, the virtual machine continues to run on the source host.

A VMotion operation can fail for several reasons, such as network latency or unresponsive storage. In

many cases, vCenter will have an error message detailing the cause of the failed migration. The ESX

hosts also log migration information. See Appendix B – migration logs for information on logs to

check to help diagnose a failed VMotion migration.

VMotion CPU compatibility

Successful VMotion migration requires that the processors of the destination host be able to execute

using equivalent instructions to those the processors of the source host were using when the virtual

machine was migrated off of the source host. Processor clock speeds, cache sizes, and the number of

processor cores can vary, but processors must come from the same vendor (Intel® or AMD™) and

present an identical CPU feature set. VMotion compatibility rules prevent unsafe migration that can

make a virtual machine unstable.

By default, vCenter only allows live migration with VMotion between source and destination

processors with a compatible feature set. If processors do not have a compatible feature set, a CPU

mask must be used to make the CPU feature set on the destination host appear identical to the CPU

feature set on the source host. For information on VMotion CPU compatibility requirements for Intel

processors, see VMware KB Article 1991, http://kb.vmware.com/kb/1991. For information on

VMotion CPU compatibility requirements for AMD processors, see VMware KB Article 1992,

http://kb.vmware.com/kb/1992.

3

Note

VMware does not recommend or support the use of CPU compatibility

masks in production environments. For more information on CPU

compatibility masks, see http://kb.vmware.com/kb/1993.

Enhanced VMotion Compatibility

Enhanced VMotion Compatibility (EVC) removes the need to set CPU masks manually. EVC is a

cluster setting that automatically configures all hosts in the cluster to be VMotion compatible with each

other. All guests in the cluster can migrate live to any host in the cluster because guests always see an

identical CPU feature set from all hosts in the EVC cluster. Figures 1 and 2 show the EVC options

available in VI3 vCenter cluster settings. VI3 has one EVC mode for Intel hosts and one EVC mode for

AMD hosts.

Note

Before enabling EVC, or adding a new host to an EVC cluster, it is

recommended that the host be updated to the latest HP ProLiant system

ROM version.

Figure 1. VI3 EVC cluster setting for AMD hosts

4

Figure 2. VI3 EVC cluster setting for Intel hosts

Figures 3 shows the VMware EVC modes available in vSphere 4 for AMD hosts. vSphere 4 has four

EVC modes for AMD hosts (AMD Opteron™ Generation 1, AMD Opteron Generation 2, AMD

Opteron Generation 3 (no 3DNow!), and AMD Opteron Generation 3).

Figure 3. vSphere EVC cluster setting for AMD hosts

5

Figures 4 shows the EVC modes available in vSphere 4 for Intel hosts. vSphere 4 has four EVC modes

for Intel hosts (Intel Xeon® Core™ 2, Intel Xeon 45nm Core 2, Intel Xeon Core i7, and Intel Xeon

32nm Core i7).

Figure 4. vSphere EVC cluster setting for Intel hosts

EVC uses Intel VT FlexMigration and AMD-V Extended Migration, concepts jointly developed by

VMware and the CPU manufacturers, to dynamically turn off selected CPUID feature bits. Intel VT

FlexMigration is available in Intel processors with the Intel Core 2 microarchitecture and newer.

AMD-V Extended Migration is available in Second-Generation AMD Opteron processors and newer.

In VI3, with vCenter 2.5 U2 and hosts using ESX 3.5 U2 or later, there is one EVC baseline for each

CPU vendor. For Intel Xeon processor-based EVC clusters, the baseline is CPU features supported by

Intel Core 2 (Merom) processors. For AMD processor-based EVC clusters, the baseline is CPU features

supported in AMD Opteron First and Second Generation (Revision E/F) processors.

vSphere 4 includes support for multiple baselines, such as Penryn and Nehalem baselines for Intel-

based EVC clusters, and a Greyhound baseline for AMD-based EVC clusters. This allows more control

over which CPU features are exposed to the guest. There is a tradeoff between compatibility and

capability. The most capable baseline will expose the largest subset of CPU features supported by

CPUs in the cluster, but older hardware may not be added to the cluster. The most compatible

baseline will expose a minimal set of CPU features to the guest so older hardware may be added to

the cluster. Table 1 lists HP ProLiant server processors supported in EVC clusters.

6

Table 1. Processors in HP ProLiant Servers Supported in EVC Clusters

Baseline Processors Supported

Intel Core 2

Intel Xeon Core 2 (Merom)

51xx, 53xx, 72xx, 73xx series

Intel Xeon 45nm Core 2 (Penryn)

33xx, 52xx, 54xx, 74xx series

Intel Xeon Core i7 (Nehalem)

35xx, 55xx series

Intel Xeon 32nm Core i7 (Westmere)

56xx series

Intel Xeon 45nm Core 2

Intel Xeon 45nm Core 2 (Penryn)

Intel Xeon 33xx, 52xx, 54xx, 74xx series

Intel Xeon Core i7 (Nehalem)

35xx, 55xx series

Intel Xeon 32nm Core i7 (Westmere)

56xx series

Intel Xeon Core i7

Intel Xeon Core i7 (Nehalem)

35xx, 55xx series

Intel Xeon 32nm Core i7 (Westmere)

56xx series

Intel Xeon 32nm Core i7 Intel Xeon 32nm Core i7 (Westmere)

56xx series

First Generation AMD Opteron (vSphere)

Second Generation AMD Opteron (VI3)

First Generation AMD Opteron

Rev. E based CPUs

2xx , 8xx

Second Generation AMD Opteron

Rev. F based CPUs

22xx, 82xx

Third Generation AMD Opteron

Greyhound based CPUs

23xx, 83xx, 24xx, 84xx

Second Generation AMD Opteron

Second Generation AMD Opteron

Rev. F based CPUs

22xx, 82xx

Third Generation AMD Opteron

Greyhound based CPUs

23xx, 83xx, 24xx, 84xx

Third Generation AMD Opteron

Third Generation AMD Opteron

Greyhound based CPUs

23xx, 83xx, 24xx, 84xx

7

VMware Enhanced VMotion compatibility requirements

All hosts in the cluster must be running ESX Server 3.5 Update 2 or later, and be connected to

vCenter Server 2.5 U2 or later. Hosts with Nehalem processors must be running ESX Server 3.5

Update 4 or later.

All hosts must be licensed for VMotion.

All hosts in the cluster must use shared storage for guests. Shared storage can be implemented

using a Fibre Channel (FC) storage area network (SAN), iSCSI, or network attached storage (NAS).

All hosts must have access to the same subnets, and network labels for each virtual machine port

group should match.

All hosts require a private gigabit Ethernet network for VMotion.

All hosts in the cluster must have CPUs from a single vendor, either Intel or AMD.

All hosts in the cluster must either have hardware live migration support (Intel VT FlexMigration or

AMD-V Extended Migration) or have the CPU feature set you intend to enable as the EVC cluster

baseline.

All hosts in the cluster must have hardware virtualization enabled in the BIOS if it is available (Intel

Virtualization Technology or AMD Virtualization). See Appendix A – enabling processor

virtualization options for more information on enabling these features on HP ProLiant servers.

All hosts in the cluster must have execute protection enabled in the BIOS (No-Execute Memory

Protection on Intel processors and No-Execute Page-Protection on AMD processors). See Appendix

A – enabling processor virtualization options for more information on enabling these features on HP

ProLiant servers.

All virtual machines in the cluster must be powered off or migrated out of the cluster when EVC is

enabled. If the virtual machines are migrated to a host with the same processor type that will be

enabled as the baseline for the EVC cluster, VMotion can be used to migrate these virtual machines

into the EVC cluster after it is configured.

– Any new hosts added to an existing EVC cluster must have virtual machines powered off or

evacuated prior to the new hosts being added to the EVC cluster.

Note

EVC clusters are supported only if applications running in the guest are

well-behaved. This means an application uses relevant CPUID feature flags

to detect the existence of a feature. See VMware KB 1005763 for details.

http://kb.vmware.com/kb/1005763.

8

Intel-based HP ProLiant servers allowed in an EVC cluster

Table 2 lists HP ProLiant server models that have been certified for VMware ESX 3.5 U2 or later, and

contain processors compatible with an EVC baseline.

Table 2. HP ProLiant servers allowed in an EVC cluster for Intel hosts

Merom Penryn Nehalem Westmere

Inte

l X

eon 5

1xx S

eri

es

Inte

l X

eon 5

3xx S

eri

es

Inte

l X

eon 7

2xx S

eri

es

Inte

l X

eon 7

3xx S

eri

es

Inte

l X

eon 3

3xx S

eri

es

Inte

l X

eon 5

2xx S

eri

es

Inte

l X

eon 5

4xx S

eri

es

Inte

l X

eon 7

4xx S

eri

es

Inte

l X

eon 3

5xx S

eri

es

Inte

l X

eon 5

5xx S

eri

es

Inte

l X

eon 5

6xx S

eri

es

BL20p G4 √ √

BL2x220c G6 √ √

BL260c G5 √ √ √

BL280c G6 √ √

BL460c G1 √ √ √ √

BL460c G5 √

BL460c G6 √ √

BL480c G1 √ √ √ √

BL490c G6 √ √

BL680c G5 √ √ √

DL160 G6 √ √

DL170h G6 √ √

DL180 G6 √ √

DL360 G5 √ √ √ √

DL360 G6 √ √

DL360 G7 √

DL370 G6 √

DL380 G5 √ √

DL380 G6 √ √

DL380 G7 √

DL580 G5 √ √ √

9

Merom Penryn Nehalem Westmere

Inte

l X

eon 5

1xx S

eri

es

Inte

l X

eon 5

3xx S

eri

es

Inte

l X

eon 7

2xx S

eri

es

Inte

l X

eon 7

3xx S

eri

es

Inte

l X

eon 3

3xx S

eri

es

Inte

l X

eon 5

2xx S

eri

es

Inte

l X

eon 5

4xx S

eri

es

Inte

l X

eon 7

4xx S

eri

es

Inte

l X

eon 3

5xx S

eri

es

Inte

l X

eon 5

5xx S

eri

es

Inte

l X

eon 5

6xx S

eri

es

ML330 G6 √ √ √

ML350 G5 √ √ √

ML350 G6 √ √

ML370 G5 √ √ √

ML370 G6 √

SL160z G6 √ √

SL170z G6 √ √

AMD Opteron-based HP ProLiant servers allowed in an EVC cluster

Table 3. HP ProLiant servers allowed in an EVC cluster for AMD hosts

Generation 1 Generation 2 Generation 3

AM

D O

pte

ron 2

xx S

eri

es

AM

D O

pte

ron 8

xx S

eri

es

AM

D O

pte

ron 2

2xx S

eri

es

AM

D O

pte

ron 8

2xx S

eri

es

AM

D O

pte

ron 2

3xx S

eri

es

AM

D O

pte

ron 8

3xx S

eri

es

AM

D O

pte

ron 2

4xx S

eri

es

AM

D O

pte

ron 8

4xx S

eri

es

BL25p G1 √

BL25p G2 √ √

BL35p G1 √

BL45p G1 √

BL45p G2 √

BL465c G1 √ √

BL465c G5 √

BL465c G6 √

BL495c G5 √

10

Generation 1 Generation 2 Generation 3

AM

D O

pte

ron 2

xx S

eri

es

AM

D O

pte

ron 8

xx S

eri

es

AM

D O

pte

ron 2

2xx S

eri

es

AM

D O

pte

ron 8

2xx S

eri

es

AM

D O

pte

ron 2

3xx S

eri

es

AM

D O

pte

ron 8

3xx S

eri

es

AM

D O

pte

ron 2

4xx S

eri

es

AM

D O

pte

ron 8

4xx S

eri

es

BL495c G6 √

BL685c G1 √ √

BL685c G5 √

BL685c G6 √ √

DL365 G1 √ √

DL365 G5 √

DL385 G1 √

DL385 G2 √ √

DL385 G5 √

DL385 G6 √

DL585 G1 √

DL585 G2 √ √

DL585 G5 √

DL585 G6 √

DL785 G5 √

DL785 G6 √

Summary

VMware VMotion allows virtual machines to migrate from one physical server to another with no

downtime. A live migration is undetectable to end users of the virtual machine. Successful migration

requires CPU compatibility between source and destination hosts. Maintaining VMotion compatibility

between hosts in a cluster can become challenging, especially as new hardware enters the

environment. Enhanced VMotion Compatibility simplifies maintaining compatibility between hosts by

enabling a baseline set of features for all hosts in an EVC cluster. In order to use an HP ProLiant server

in an EVC cluster, several conditions beyond VMotion requirements must be met. As outlined in Tables

2 and 3, the server model and processor combination must be certified for ESX 3.5 U2 or later, and

the processor must be compatible with an EVC cluster.

11

Appendix A – enabling processor virtualization options

In order to enable an HP ProLiant server for VMotion or for use in an EVC cluster, BIOS settings for the

processors must enable Hardware Virtualization (if available) and Execute Protection. On HP ProLiant

servers, Hardware Virtualization is specified as Intel Virtualization Technology on Intel-based ProLiant

servers and AMD Virtualization on AMD-based ProLiant servers. Execute Protection is specified as No-

Execute Memory Protection on Intel-based ProLiant servers and No-Execute Page-Protection on AMD-

based ProLiant servers. The Figures A-1a – A-3c below show the paths to enable these features using

the ROM Based Setup Utility (RBSU). The RBSU is accessed by pressing F9 during POST. The path to

the relevant BIOS options is dependent on the server model.

Figure A-1a shows a common step for both Intel- and AMD-based ProLiant G5 (or earlier) servers.

Advanced Options is selected first.

Figure A-1a. Advanced Options selection

12

Figure A-1b shows RBSU Processor Options on an Intel-based HP ProLiant BL460c G6 server. System

Options is selected first.

Figure A-1b. System Options selection on an HP ProLiant BL460c G6 server

13

Some ProLiant servers list processor options such as Hardware Virtualization in the Advanced Options

section of RBSU, as shown in Figure A-2a. In those servers, Hardware Virtualization can be enabled

in the Advanced Options section. RBSU of most ProLiant servers will have a Processor Options

selection, as shown in Figures A-2b and Figure A-2c.

Figure A-2a. RBSU Advanced Options list with Hardware Virtualization options for an Intel-based ProLiant server

14

Figure A-2b. RBSU Processor Options selection

15

Figure A-2c shows RBSU Processor Options on an Intel-based HP ProLiant BL460c G6 server.

Figure A-2c. Processor Options selection of a HP ProLiant BL460c G6 server

Figure A-3a shows the RBSU processor option for many AMD-based HP ProLiant servers. In Figure

A-3a, No-Execute Page-Protection and AMD Virtualization (if available) must be enabled for AMD-

based HP ProLiant servers. Some AMD-based HP ProLiant servers, such as the HP ProLiant DL585 G1,

will not have the AMD Virtualization option available.

Figure A-3a. RBSU AMD processor options

16

Figure A-3b shows RBSU Processor Options on many Intel-based HP ProLiant servers. No-Execute

Memory Protection and Intel Virtualization Technology (if available) must be enabled.

Figure A-3b. Intel Virtualization Technology processor option

Figure A-3c shows RBSU Processor Options on an Intel-based HP ProLiant BL460c G6 server. No-

Execute Memory Protection and Intel Virtualization Technology must be enabled.

Figure A-3c. No-Execute Memory Protection processor option on an HP ProLiant BL460c G6 server

Once the Hardware Virtualization options have been enabled, press <Esc> three times, then F10 to

exit the RBSU and reboot the server.

17

Appendix B – migration logs

Several logs can be used to troubleshoot a failed migration on an ESX host. The first step is to obtain

the migration ID. The migration ID is listed on the left of each row of

/proc/vmware/migration/failed. /proc/vmware/migration/active, and

/proc/vmware/migration/history can also be used to obtain the migration ID. The following

command can be used in the ESX Service Console to retrieve the migration ID in the case of a failed

VMotion operation:

cat /proc/vmware/migration/failed

Once the migration ID is obtained for a failed VMotion operation, search for the migration ID in

several of the other ESX logs, such as the /var/log/vmkernel logs. The following command will

return the log entries related to the migration:

grep <migration id> /var/log/vmkernel*

The logs vmware*.log are located in the guest’s home directory on the source host. The location of

these logs will depend on the datastore that holds the virtual machine files and the name of the virtual

machine. For a virtual machine on a VMFS datastore, the path to the location of the logs would be

similar to /vmfs/volumes/<datastore>/<vm name>/vmware-*.log. The following command will

return the log entries related to the migration:

grep <migration id> /vmfs/volumes/<datastore>/<vm name>/vmware-*.log

It may also be useful to grep the migration ID of hostd logs, /var/log/vmware/hostd*.log, on

source and destination hosts.

grep <migration id> /var/log/vmware/hostd*.log

Examination of these logs may give you clues as to what caused the migration to fail.

For more information

HP Virtualization with VMware, http://www.hp.com/go/vmware

HP ProLiant servers, http://www.hp.com/go/proliant

HP ProLiant servers VMware support matrix,

http://h20219.www2.hp.com/enterprise/cache/505363-0-0-0-121.html

VMware SiteSurvey and CPU Identification Utility,

http://www.vmware.com/download/shared_utilities.html

VMware Hardware Compatibility Guide search,

http://www.vmware.com/resources/compatibility/search.php?

VMotion Compatibility Info Guide, http://www.vmware.com/files/pdf/vmotion_info_guide.pdf

Basic System Administration Guide for ESX 3.5 U2,

http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_admin_guide.pdf

vSphere Resource Management Guide,

http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf

VMotion and CPU Compatibility FAQ, http://kb.vmware.com/kb/1005764

Enhanced VMotion Compatibility (EVC) processor support, http://kb.vmware.com/kb/1003212

Detecting and Using CPU Features in Applications, http://kb.vmware.com/kb/1005763

General VMotion Intel processor compatibility information, http://kb.vmware.com/kb/1991

General VMotion AMD processor compatibility information, http://kb.vmware.com/kb/1992

To help us improve our documents, please provide feedback at

http://h20219.www2.hp.com/ActiveAnswers/us/en/solutions/technical_tools_feedback.html.

© Copyright 2009-2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

AMD and AMD Opteron are trademarks of Advanced Micro Devices, Inc. Intel, Core and Xeon are trademarks of Intel Corporation in the U.S. and other countries.

4AA2-6016ENW, Created April 2009; Updated, May 2010, Rev. 1