7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
1/57
vSphere HA and Datastore
Access Outages Current-Capabilities Deep-Dive andTech Preview
Smriti Desai, VMware, Inc.
Keith Farkas, VMware, Inc.
INF-BCO2807
#vmworldinf
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
2/57
2
Disclaimer
This session may contain product features that are
currently under development.
This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Technical feasibi lity and market demand will affect final delivery.
Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
3/57
3
VMware Business Continuity Solutions
Local Availability
vSphere High Availability
vSphere Fault Tolerance
vMotion and Storage vMotion
Data Protection
vSphere Data Protection
Storage APIs for Data Protection
Local Site Failover Site
vSphere vSpherevSphere vSphere vSphere
This talk
Disaster Recovery
vCenter Site Recovery Manager and
vSphere Replication
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
4/57
4
vSphere HA Recap
vSphere HA minimizes unplanned downtime
Provides automatic VM recovery in minutes
Protects against 3 types of failures
Does not require complex configuration
Is OS and application-independent
Infrastructure Connectivity Application
Host failures Host network isolated GuestOS hangs/crashes
VM crashes Datastore incurs PDL Application hangs/crashes
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
5/57
5
Talk Focus
Datastore accessibility outages occur infrequently but have a
large cost
Loss of accessibility is due to
Network or switch failure
Array, NFS sever, etc. misconfiguration
VM manageabil ity and availabil ity is affected
Applications with vdisks on inaccessible datastores hang, crash, or fail
May not be able to manage VMs on the affected hosts
vSphere HAs protection impacted
VMwareESX VMwareESX
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
6/57
6
Agenda and Objectives
Session has two major parts
1. Impact of datastore inaccessibi lity on HA failover workflows
2. Expanding HA protection against datastore inaccessibi lity
Objectives
Learn how HA workflows are impacted by datastore accessibility
Understand how vSphere 5.0/1 reduces the impact of inaccessibility
Preview the future - protecting VMs against datastore inaccessibility
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
7/57
7
Agenda: Part 1
1. Impact of datastore inaccessibil ity on HA failover workflows
Architecture overview
Datastore usage HA workflows and responses
2. Expanding HA protection against datastore inaccessibil ity
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
8/57
8
vSphere 5.0+ Architecture
HA Agent
Called the Fault Domain Manager (FDM)
Provides all the HA on-host functionality
Operation
vCenter Server (VC) manages the cluster Failover operations are independent of VC
Communicate over
Management network
Datastores
vCenter Server (VC)
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
9/57
9
FDM Master and Slave Roles
Any FDM can be master, selected by election
All others assume the role of FDM slaves
The FDM master
Monitors hosts and VMs
Manages VM restarts after failures
Reports cluster state to VC
The FDM slave
Forwards critical state changes to the master
Restart VMs when directed by the master
Elect new master
vCenter Server (VC)
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
10/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
11/57
11
Datastores Used for Communication
Datastores are used when management network is not available
Heartbeat datastores
Used by a master to monitor a partitioned/isolated slave
Enables a master to detect VM power state changes
VC chooses two (by default) for each host
Reselected after datastore accessibility changes
Home datastore of each VM
Used by isolated slaves to determine if a master owns the VM
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
12/57
12
Datastorecommunication HA host states Response
Persisted
Configuration
used in
triggers
used in
influence
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
13/57
13
Information Sources for HA Host States
The HA state reported for each host is derived using information f rom
VC
The master VC is communicating with
The FDM on the host
State Source
Election FDM on the host
Running (Master) VC
Connected (Slave) Master
Unreachable Master or VC
Isolated FDM on the host, reported by master
Partitioned Master
Dead Master
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
14/57
14
Information Sources for HA Host States
The HA state reported for each host is derived using information f rom
VC
The master VC is communicating with
The FDM on the host
State Source
Election FDM on the host
Running (Master) VC
Connected (Slave) Master
Unreachable Master or VC
Isolated FDM on the host, reported by master
Partitioned Master
Dead Master
Determined using
datastore
communication
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
15/57
15
How a Master Determines a Slaves State
Connected Slave
Unreachable
Partitioned / Isolated*
Dead
Not connected
No datastore heartbeats
No response to pings
Datastore heartbeats
No datastore heartbeats
Network available
* See slide notes
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
16/57
16
Impact of datastore accessibili ty on responses
Datastorecommunication HA host states Response
Persisted
Configuration
used in
triggers
used in
influence
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
17/57
17
Network Isolated FDM - VM Isolation Workflow
Apply isolationresponse
Homedatastore
accessible?
Master
owns VM?
Determine
VMs to
power off /
shutdown
No. Wait
Yes
Yes
No
Report VM poweroff
1 2 3
4 5
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
18/57
18
Network Isolated FDM - Home Datastore Inaccessibili ty
If all FDMs are isolated, all wil l apply isolation responses
VMs not restarted until master has access to VM datastores
Best practices
Redundant management networks
Reconfigure storage to reduce likelihood of inaccessible datastores
Use leave powered on isolation option
Apply isolationresponse
Homedatastore
accessible?
Master
owns VM?
No. Wait
Yes
Yes
No
Report VM poweroff
2 3
4 5
Determine
VMs to
power off /
shutdown
1
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
19/57
19
Network Isolated FDM - Heartbeat Datastore Accessibil ity
Isolated host and master have access to heartbeat datastores
Master will attempt failover on power off notification
Otherwise
Master will declare host dead and start failover immediately*
Same situation applies to partitioned hosts
Apply isolationresponse
Homedatastore
accessible?
Master
owns VM?
No. Wait
Yes
Yes
No
Report VM poweroff
* More info in backup slides
2 3
4 5
Determine
VMs to
power off /
shutdown
1
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
20/57
20
Host Declared DeadThe VM Failover Response
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
21/57
21
Retry
after adelay
Wait for
capacitychange
Host Dead: FDM Masters Workflow
Host declared dead
Determine VMs to be failed
over
Found
place?
No
Error encountered
Try to place each VMRestarted
?
Yes
No
End
Yes
1
2
3 4
5
6
7
Note: steps 2 to 7 apply anytimea VM is to be restarted
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
22/57
22
Impact of Datastore Accessibili ty: VMs to Failover
Case 1: home datastore of VM is not accessible on masters host
Master will proxy all accesses via a slave with access
Case 2: master may not know the VM is protected
Reason #1: VMs home datastore is inaccessible
VM cant be powered on in any case
Master will retry once datastore is accessible
Reason #2: partition, multiple masters, other master owns VM
But other master knows and will restart it if needed
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
23/57
23
Retry
after adelay
Wait for
capacitychange
Host Dead: FDM Masters Workflow
Host declared dead
Determine VMs to be failed
over
Found
place?
No
Error encountered
Try to place each VMRestarted
?
Yes
No
End
Yes
1
2
3 4
5
6
7
Note: steps 2 to 7 apply anytimea VM is to be restarted
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
24/57
24
Impact of Datastore Accessibil ity: VM Restart
Case 1: host manageabili ty impacted by datastore inaccessibi lity
Master will retry failovers on another host after timeout
Could take a long time to restart failed VMs
vSphere 5.0 and 5.1 enhancements significantly reduces impact
Case 2: one of VMs datastores is inaccessible on some/all hosts
Master will retry but could exhaust 5 retries before success
Future opportunity to enhance HA
Both are discussed next in part 2 of this session
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
25/57
25
Agenda: Part 2
1. Impact of datastore inaccessibility on HA failover workflows
2. Expanding HA protection against datastore inaccessibil ity
Technical direction
VM manageability and availability
Tech preview
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
26/57
26
Solution Approach for Inaccessible Datastores
Improve VM availabili ty by ensuring
1. VMs are manageable
2. VMs that use the datastore are moved to healthy hosts
Address #1 by enhancing ESX, #2 by enhancing HA
VMwareESX VMwareESXvCenter Server Manage
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
27/57
27
Types of Inaccessibil ity: PDL and APD
Are ESX storage-device states that indicate inaccessibility
PDL (Permanent Device Loss): device is permanently inaccessible
E.g., caused by removing a LUN using array management software
ESX infers state from
SCSI sense codes returned by an array
iSCSI login reject (target is gone or access not authorized)
Device must be recreated to restore normal operation
APD (All Paths Down): device is possibly temporarily inaccessible E.g., caused by unplugging a network cable
Device could become accessible at any time
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
28/57
28
T=140s (default)
APD timeout declared
I/O fast failing starts
Datastore reachable
APD cleared
Normal I/O behavior
ESX Enhancements for VM Manageabili ty
Idea: if a datastore is under APD/PDL, fail I/Os quickly
Impacted operations notified faster and allows others to proceed
T=0
APD detection
ESX PDL Support (vSphere 5.0) ESX APD Support (vSphere 5.1)
When under PDL, I/Os are failed
immediately
When under APD, non guest I/Os
are failed immediately after a delay
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
29/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
30/57
30
HA Enhancements for VM Availabil ity: The Future
We are exploring a signif icant extension to this mechanism
Design goals
Add support for APD
Triggered by PDL/APD declaration rather than guest I/Os
Full customization of responses (e.g., event only option)
Full user interface and detailed reporting
VM placement sensitive to accessibility
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
31/57
31
VM Component ProtectionCaveat: what follows is a prototype and a feature based on it
may look quite different, if and when we offer it
Protection Workflow:
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
32/57
Protection Workflow:
PDL
Datastore
inaccessible
Terminate andrestart VM
PDL
Determine
per VM
response
End
FailoverNo action
1
2
3
4
Protection Workflow: 2
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
33/57
Protection Workflow:
APD
Datastore
inaccessible
Terminate andrestart VM
End
No actionFailover
APD
No
Wait for APD
declaration
Wait for
optional delay
Determine
per VM
response
Could
reservecapacity?
Yes
2
3
4
56
7
1
Protection Workflow: 2
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
34/57
Protection Workflow:
APD
Datastore
inaccessible
Terminate andrestart VM
End
No actionFailover
APD
No
Wait for APD
declaration
Wait for
optional delay
Determine
per VM
response
Could
reservecapacity?
Yes
2
3
4
56
7
1
Reset guest if
requested
APDc
le
ared
8
Combined Workflow:
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
35/57
APD and PDL
Restart guest if
requested
Datastore
inaccessible
Terminate andrestart VM
APDc
le
ared
PDL
Determine
per VM
response
End
FailoverNo action No actionFailover
APD
No
Wait for APD
declaration
Wait for
optional delay
Determine
per VM
response
Could
reservecapacity?
Yes
D O i
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
36/57
36
Demo Overview
2 VMs on NFS
OracleDBServer
ExchangeServer
3 VMs on SAN
Webserver 1
Webserver 2
Ubuntu
APD impacts host A SAN
Converged NFS/iSCSI storage array
FC Switch Ethernet Switch
SAN NFS
WS1 WS2
OR EX
UB
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
37/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
38/57
Summary: Protection Against Datastore Inaccessibility
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
39/57
39
Summary: Protection Against Datastore Inaccessibility
Several platform enhancements in recent years
vSphere 5.0: PDL support
vSphere 5.1: APD support
vSphere 5.0U1: HA restarts VMs if they fail during a PDL/APD
The future: HA recovering VMs impacted by PDL/APD
Comprehensive: APD and PDL, and covers all VM I/Os
Configurable: Various levels of VM remediation
Usable: Enable with 1 click, detailed error reporting
Please send us your feedback on the proposed feature
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
40/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
41/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
42/57
42
Questions?
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
43/57
FILL OUT
A SURVEY
EVERY COMPLETE SURVEY
IS ENTERED INTO
DRAWING FOR A
$25 VMWARE COMPANY
STORE GIFT CERTIFICATE
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
44/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
45/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
46/57
46
VM Protection Workflow Example
2 VM Protection Workflow Power On
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
47/57
47
2. VM Protection Workflow Power On
6. Master updates protection list on disk
5. VC tells master to protect VM
4. VC reports VM is unprotected
2. User powers on VM
7. Master informs VC that it did
8. VC reports VM is protected
3. Host reports to VC VM powered on
1. VM is off. VM protection is N/A
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
48/57
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
49/57
Applying Concepts: Host Failures Host Dead
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
50/57
50
Applying Concepts: Host Failures Host Dead
Master declares a host dead when:
Master cant communicate with it over the network
Host is not connected to master Host does not respond to ICMP pings
Master observes no datastore heartbeats
Results in: Master attempts to restart all VMs from host
Restarts on network-reachable hosts andits own host
ESX 1 ESX 3
ESX 4ESX 2
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
51/57
Troubleshooting vSphere HA 5.0
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
52/57
52
Troubleshooting vSphere HA 5.0
HA issues proactive warning about possible future conditions
VMs not protected after powering on
Management network discontinuities Isolation addresses stop working
HA host states provide granularity in to error conditions
Al l HA conditions reported via events; config issues/alarms for some
Event descriptions describe problem and actions to take
All event messages contain vSphere HA so searching for HA issues easier
HA alarms are more fine grain and auto clearing (where appropriate)
5.0+ Troubleshooting guide which discusses likely top issues. E.g.,
Implications of each of the HA host states Topics on HB datastores, failovers, admission control
Will be updated periodically
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
53/57
Log File Format
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
54/57
54
og e o at
Log fi le contains time stamped rows
Many rows report the HA agent (FDM) module that logged the info
E.g.,
2011-06-01T05:48:00.945Z [FFFE2B90 info 'Invt' opID=SWI-a111addb] [InventoryManagerImpl::ProcessClusterChange] Cluster state
changed to Startup
Noteworthy modules are
Cluster module responsible for cluster functions Invt module responsible for caching key inventory details
Policy module responsible for deciding what to do on a failure
Placement module responsible for placing failed VMs
Execution module responsible for restarting VMs Monitor modules responsible for periodic health checks
FDM module responsible for communication with vCenter Server
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
55/57
Heartbeat Datastores(HB): Mechanisms
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
56/57
56
( )
Used by master for slaves not connected to it over network
Determine if a slave is alive
Rely on heartbeats issued to slaves HB datastores
Each FDM opens a file on each of its HB datastores for heartbeating purposes
Files contain no information. On VMFS datastores, file will have the minimum-allowedfile size
Files are named X-hb, where X is the (SDK API) moID of the host
Master periodically reads heartbeats of all partitioned / isolated slaves
Determine the set of VMs running on a slave
A FDM writes a list of powered on VMs into a file on each of its HB datastores
Master periodically reads the files of all partitioned/isolated slaves
Each poweron file contains at most 140 KB of info. On VMFS datastores, actual diskusage is determined by the file-sizes supported by the VMFS version
They are named X-powereon, where X is the (SDK API) moID of the host
7/27/2019 BCO2807-vSphere HA and Datastore Access Outages Current-Capabilities Deep-Dive and Tech Preview_Final_US.pdf
57/57