May 26, 2015
Sponsored by:
&
• Traditional tradeoff: Local vs. Central storage: – Central storage migrates very simply.
• Just complete last work from old location
–Then start at the new location.
• Extracts non-local latency tax on every access.
– Purely local storage makes migration a challenge.
• Must migrate VM image and local storage.
• And synchronize them.
2 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• SSD vs HDD: –SSDs provide most IOPS per $.
•SSDs need to be on host, or at least near the VM, or network latencies minimize SSD speed.
–HDDs provide most TBs per $.
•HDDs provide capacity, not processing speeds.
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
Host
Device Domain
DomU
VFS
File System
Block Driver
Network Driver
File System
Block Backend SSD Proxy
External NAS
Server(s)
App
NAS Client Block Driver
External SAN
Server(s)
Network Driver
Remote Block Access
Local Block Driver
Local Block Device
• A very common solution currently features a Block Backend that co-ordinates a local block device (typically SSD) with remote block device (still NAS or SAN).
• Problem:
– Any cache solution only stores a portion of a VM‟s storage locally.
– With NAS Proxy the “portion stored locally” can be understood as files and directories.
• A block-oriented solution never sees more than a collection of blocks.
• Many solutions are enhanced or made possible with File-orientation.
4
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• DomU can support both NAS
access and Virtual Disk Access – Most VMs will prefer directly-attached boot
drive.
• Backend can map virtual drive to
NAS.
• Meanwhile NAS Client can access
NAS Proxy via local internal
network.
• NAS Proxy can utilize local SSD
drive to optimize either.
5
Host
Device Domain
DomU
VFS
NAS Client
Network Driver
Network Driver
NAS Proxy
Network Backend
External NAS
Server(s)
App
Block Driver
Local Block Device
File System
Block Driver
BlockBackend
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• NAS Benefits
– Files enable understanding of context:
– Deduplication of files is far easier than deduplication of block.
– Hierarchical Storage Management works on files, not blocks.
• Traditional NAS Proxy Caveats
– Traditional NAS is unaware of VM-specific semantics.
– Excessive syncing – Historical assumptions are that files are
shared by default.
– Proxy overhead for non-cached files – Historically, the only easy
configuration was to proxy all NAS access.
• VM Storage Proxy provides full benefits of NAS without
limitations of traditional NAS Proxies.
6 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Distributed NAS protocols such as NFSv4 and
DFS/CIFS have features that make NAS proxies far
more efficient.
• They allow efficient caching via exclusive delegation
and oplocking (“opportunistic locking”).
– Proxy can temporarily claim a file.
• They have advanced session features to protect
against data loss from connection loss.
– File reverts to central NAS server if Proxy dies.
– System recovers from network partitions and temporary
outages of distributed servers and/or proxies as long as
they all have persistent redundant storage.
8 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Client can simply mount from NAS server.
• NAS server can refer select directories to a NAS “Server” that is actually the NAS Proxy.
– NAS Proxy/Server must act as a NAS Server for NAS replication procedures.
• Client does not need to identify itself.
– NAS server can infer identity from IP address.
• Advantage that Proxy/Server is enabled/disabled for each directory from the central NAS Server.
9
VM Client
NAS Server
NAS Proxy / Server
All Directories
Cached Files Directory access
Referral
Replication
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• NAS Proxy can act as a NAS
client and claim an exclusive
delegation on a file.
• It can then treat its cached copy
as the copy.
• It must replicate it back to the
central NAS server before
ending the delegation.
• Keep-alive messages track the
NAS client with the delegation.
• Protocol rules already cover
loss of contact/failure
scenarios.
10
VM Client
NAS Server
NAS Proxy / Server
All Directories
Cached Files
Claim Exclusive Delegation
DelegationGranted
Local copy canbe treated as
THE copy.
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Generic NAS Proxy does not understand that it is the sole writer for the VM Private Directory.
• NAS Proxy management is not integrated with VM Management. – The NAS system must detect that the VM using its VM Private
Directory has migrated.
• Proxies only optimize storage for selective files – but all traffic goes through them. – If there are directories/files that are not worth caching the clients
would ideally just go directly to the real server.
– NAS Referrals can be applied selectively, but Management Plane controls are awkward for dynamically controlling referrals.
• Optimized cache sharing across multiple clients is challenging without understanding of VM life-cycle.
12 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Allow VM‟s files to be stored and archived as
files.
• Allow storage associated with commodity servers
to be deployed effectively for VMs in combination
with NAS storage.
• Don‟t replicate to local storage those files that
probably won‟t be accessed anyway.
• Allow very thin provisioning of local storage.
– SSDs are designed to provide IOPs cost effectively,
not storage.
– If IOPS are not needed, avoid cost of redundant
storage. 13
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• The VM NAS Proxy is a specialized NAS Proxy that is optimized to support VMs and the Hypervisor Platform supporting them:
• A VM Private Directory is maintained for each VM:
– A per VM directory in the NAS global name space that holds files that are “local” for this VM.
– A “platform” sub-directory hold virtual disk images and other files used by the Hypervisor and/or Dom0.
– A “mount” sub-directory holds NAS mount points that are available only to this VM.
• These are implicitly named. The client does not know the path to its VM Private Directory because it does not know which VM it is.
• A “proxy” sub-directory holds files used by the VM NAS Proxy for this VM, such as the NAS Response Cache.
14 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
VM NAS Proxy as a VM. Virtualization Host
Virtualization Management
NAS VMS
Hypervisor Platform VM NAS ProxyVM
NAS
/vmexports/vm13478
/platform/mounts
/foo` /bar/vm13483
/platform/mounts
/foo/bar
VM13478
/foo/bar
VM NAS Proxy for a Rack
15
Rack (or other enclosure)
Virtualization Host
Virtualization Management
NAS VMS
Hypervisor Platform
VM NAS ProxyVM
NAS
/vmexports/vm13478
/platform/mounts
/foo` /bar/vm13483
/platform/mounts
/foo/bar
VM13478
/foo/bar
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Virtualization Management
– Unchanged, but new information is available to it.
• NAS VMS (Virtualization Management System)
– Extends Virtualization Management to assign VM NAS Proxies.
• Hypervisor Platform
– The Hypervisor and supporting domains.
• NAS system with NFSv4/v4.1 and/or DFS/CIFS.
– Unchanged, but optional enhancements are defined.
• VMs
– Unchanged, but optional enhancements for NAS Client are
defined.
• VM NAS Proxy
– A specialized NAS Proxy optimized for support of VMs
16 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• A VM NAS Proxy can control what is stored locally through several techniques:
– Which sub-directories of the VM‟s private directory are referred to it.
– Whether it provides local storage for specific files within those directories.
– Which blocks of a file that is stored locally are actually stored locally, vs. being accessed over the network.
– pNFS-aware clients can be enabled to access network block servers directly.
17 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• VM will be migrated from Current Host
to a new Host.
• VM has public IP address, which
Virtualization Management will migrate.
• VM has private IP Address which it will
use in new network without realizing that
it is on a new network.
• New VM NAS Proxy will have same
private IP address on the new network.
• New VM NAS Proxy does not respond
to the VM‟s private IP Address until it is
ready.
19
NAS
Current Home
Virtualization Host
10.1.1.1192.168.1.1
Hypervisor Platform10.1.3.115192.168.1.115
VM NAS Proxy
VM
10.1.1.2192.168.1.2
Planned Home
Virtualization Host
10.1.2.1192.168.1.1
Hypervisor Platform
10.1.3.115192.168.1.115
VM Storage Proxy
10.1.2.2192.168.1.2
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• By assigning VM NAS Proxies to an Internal Network migration of a VM can be made transparent to the VM. – All Internal Networks can look alike.
– Just as almost all home networks are 192.168.1.*
• NAS client does not need to know about the migration. – NAS clients will see a dropped connection to a NAS server.
– NAS clients will assume the NAS server is simply rebooting.
– Recovery will be complete and transparent to client layers.
– No change to the IP address of the VM NAS Proxy the Client uses.
• As far as the client is concerned it just reconnected to the same server – It even has the same response cache.
– The response cache is stored within the VM Private Directory.
20 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• VM will be migrated from Current Host to a new Host. – 10.1.3.115 is migrated.
• VM is still instructed to use the old VM NAS Proxy (10.1.1.2). – It will continue to use it until explicitly
redirected to the new VM NAS Proxy (10.1.2.2).
– This is very network inefficient.
– Current RTT measurements and congestion window will be wildly incorrect.
– VM NAS Proxy must accept clients from anywhere.
21
NAS
Current Home
Virtualization Host
10.1.1.1
Hypervisor Platform
10.1.3.115
VM NAS Proxy
VM
10.1.1.2
Planned Home
Virtualization Host
10.1.2.1
Hypervisor Platform10.1.3.115
VMNAS Proxy
10.1.2.2
Cont
inue
d ac
cess
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Distributed NAS features for Referral and Delegation have nothing to do with each other. – Referral/redirection is a Management Plane
feature.
• It is used to refer/redirect clients to other NAS servers.
– Delegation/oplocking is a Control Plane feature.
• It is used by NAS clients.
• A VM NAS Proxy integrates them in collaboration with NAS servers.
23 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• VM NAS Proxies use Delegation/Oplocking to
exclusively claim specific sub-directories
within a VM Private Directory.
– This is an enhancement to a NAS server.
• Supporting NAS servers will then
automatically direct referrals for the claimed
sub-directories to that VM NAS Proxy for the
duration of the delegation/oplock.
– Reverting is automatic, preferable to having to
reverse a management plane edit.
24
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Only the correct clients are referred,
• Other clients are refused access.
• Allowed access types:
– Hypervisor Platform access a „platform‟ sub-
directory for virtual disk images and snapshots.
– VM itself accesses NAS mounts in a „mount‟ sub-
directory.
– VM NAS Proxy keeps its own data in a „proxy‟
sub-directory.
• This is where the NAS Response Cache is kept.
25
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
VM NAS Proxy
NASServer
NASServer
13
2
Locally Cached
Files
• 1 – VM NAS Proxy takes exclusive delegation on a directory.
• 2 – Client referencing delegated directory is referred to VM NAS Proxy
• 3 – Client accesses directory on VM NAS Proxy
26
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• Progression of Solutions for Controlling File Locations:
– Without NFSv4 Referrals/DFS Redirections: • Load balancing requires editing each client‟s configuration.
• Or directing all traffic through a proxy.
– Using just standard Referrals/Redirections: • Central control of which directories are referred
• But only via Management Plane edits of exports.
– Integrating Delegation to allow Automatic Referrals: • Dynamic control over which directories are referred is
exercised by the VM NAS Proxy and/or VM NAS Management.
• Dynamic referral automatic ends when delegation ends.
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected] 27
• One of the features of the VM NAS Proxy is that it can dynamically control which of a VM‟s private directories are referred to it. – But standard granularity for referral is an entire directory.
• Partial Referrals is a new technique to publish alternate referrals for subsets of a directory: – Each partial referral identified by a regular expression.
– The alternate server is then specified for that regular expression, which can be “revert to default” to ignore the main referral.
– This information is published as named attributes for the directory.
• If a NAS client does not understand them then it will simply go to the default server for the directory
• The default server then acts as a proxy for the actual server.
29 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
VM NAS Proxy
NASServer
NASServer
1
2A
2B
Locally Cached
Files
• 1 – VM NAS Proxy advertises portions of directory that Client should access directly on the home NAS server.
• 2a – Client continues to access non-excluded portions of directory on the VM NAS Proxy, which caches them locally.
• 2b – But now accesses files covered by partial referral on the home NAS Server
30
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
VM NAS Proxy
NASServer
NASServer
1
2 3
Locally Cached
Files
• 1 – VM NAS Proxy advertises portions of directory that Client should access directly on the home NAS server.
• 2 – Client does not understand or ignores the partial referral, so all access for the directory goes through the VM NAS Proxy
• 3 – When files are not stored locally access is proxied via the NAS Server.
31
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• When the VM‟s NAS Client indicates that it understands pNFS, the VM NAS Proxy can supply pNFS layouts that mix references to its own storage and NAS storage.
• Referencing pNFS blocks directly is safe, because the VM NAS Proxy is the only client allowed to update the file while it is claimed.
• pNFS-aware clients can access unmodified and uncached blocks without overhead of going through the proxy.
Locally Cached Blocks
Logical Block #
Physical Block #
Logical Block #
Physical Block #
pNFS Layout
Logical Block #
Physical Block #
Logical Block #
Logical Block #
Volume ID
Physical Block #
Volume ID
Physical Block #
Volume ID
Physical Block #
Volume IDLogical Block #
Proxy pNFS Layout
Logical Block #
Physical Block #
Logical Block #
Logical Block #
Local Volume ID
Physical Block #
Volume ID
Physical Block #
Volume ID
Physical Block #
Local Volume ID
Logical Block #
Physical Block #
Volume IDLogical Block #
Physical Block #
Volume IDLogical Block #
Physical Block #
Volume IDLogical Block #
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected] 33
• Named attributes can identify files that are
to be stored only in the local cache.
– The file is never updated on the NAS server.
• Regular expression can easily identify
common temporary files.
• These files would be stored only on local
SDD
– never consuming any network bandwidth.
34 Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]
• VM NAS Proxy is only
on the path for a VM
accessing storage
when it wants to be.
– Control is fully dynamic.
• Access to Non-
optimized
directories/files can
bypass the Proxy.
• VM NAS Proxy is only
identified as referral
target for the correct
clients. 36
Access to referred directories, files
Default Access, may be referred to VM Storage ProxyAccess to non-referred directoriesAccess to partially un-referred files
Direct access to un-cached blocks
VM
VM NAS PRoxy
pNFS Block Server
NASServer
Xen Summit Aug 2, 2011 –
Caitlin.Bestler/[email protected]