Devconf.cz, February 2016 oVirt and Gluster Hyperconvergence Ramesh Nachimuthu Software Engineer Red Hat Devconf.cz, February 2016
Devconf.cz, February 2016
oVirt and Gluster Hyperconvergence
Ramesh NachimuthuSoftware EngineerRed HatDevconf.cz, February 2016
Devconf.cz, February 2016
Agenda
● Architecture of oVirt● Hosted Engine● Gluster in a nutshell● Hyper converged infrastructure
● How it looks● Setup● Management
● Recent enhancements● Conclusion
Devconf.cz, February 2016
Ovirt Architecture
oVirt is a virtualization platformto manage virtual machines,storage and networks
● Engine (ovirt-engine)Manages the oVirt hosts, and allows system administrators to create and deploy new Vms
● Host Agent (VDSM)oVirt engine communicates with VSDM to manage the Vms, storages and networks
Clu
ster
s
Storage ServerSto
rage
Eng
ine
Storage Server
Cluster Hosts
Devconf.cz, February 2016
Ovirt Storage Architecture
● Storage Domain● Centralized storage system
(images, templates, etc.)
● A standalone storage entity
● Stores the images and associated
Metadata
● Only real persistent storage for
VDSM
● Used for synchronization (sanlock)
● Storage Types● NFS, FCP, iSCSI
● Glusterfs
Clu
ster
s
Storage DomainSto
rage
Eng
ine
Storage Domain
VDSM Hosts
Devconf.cz, February 2016
Hosted Engine
● Ovirt Engine running inside a
VM
● Highly Available
● Can be migrated to a
different host
● Special Agent for Monitoring
● Bootstrap deployment
neededC
lust
ers
Storage Domain
Sto
rage
Poo
lE
ngin
e
Storage Domain
VDSM Hosts
Devconf.cz, February 2016
GlusterFS Architecture
● GlusterFS is a general purpose scale-out distributed file-system
supporting thousands of clients
● Aggregates storage exports over network interconnect to provide
a single unified namespace
● File-system completely in
userspace, runs on
commodity hardware
● Layered on disk file
systems that support
extended attributes
Devconf.cz, February 2016
GlusterFS Bricks
● A brick is an export directory located on a specific node
(e.g. host-01:/srv/fs1/brick1)
● Each brick inherits limits of the underlying file-system
● No limit on the number bricks per node (as best-practice each
brick in a cluster should be of the same size)
/srv/fs1/brick1 /srv/fs1/brick2/srv/fs1/brick2
File-System: /srv/fs1 /srv/fs2
Block Device: /dev/dm-1 /dev/dm-2
Devconf.cz, February 2016
GlusterFS Volumes
● A volume is a logical collection of bricks
● The mountable entity
● Different types of Volumes
● Distribute, Stripe, Replicate (+ combinations), Quorum
● Type of a volume is specified at the time of volume creation
and determines how and where data is placed
● Accessible via FUSE, NFS, SMB, libgfapi
host01:/srv/fs1/brick1
Volume1
host02:/srv/fs1/brick1 host03:/srv/fs1/brick1
Distribute/Stripe/Replicate
Devconf.cz, February 2016
Fuse Access
QEMU
Kernel VFS /dev/fuse
Kernel
Userspace
GlusterFS
Net
File-System
Gluster Brick
NetKernel
Userspace
Client Server
FUSE Access
● More context switches
Devconf.cz, February 2016
QEMU libgfapi Support
● GlusterFS exposes APIs for accessing Gluster volumes
● Reduces context switches
QEMU
Kernel
Userspace
Net
File-System
Gluster Brick
NetKernel
Userspace
Client Server
libgfapi Access
But see: https://bugzilla.redhat.com/show_bug.cgi?id=1247933
Devconf.cz, February 2016
Hyperconverged oVirt – GlusterFS
● The Data Center nodes are used both for virtualization and serving replicated images from the GlusterFS Bricks
● Engine runs inside a VM (Hosted Engine)
● The boxes can be standardized (hardware and deployment) for easy addition and replacement
● Support for both scaling up, adding more disks, and scaling out, adding more hosts
VM
s an
d S
tora
ge w
ith E
ngin
e in
side
a V
M
GlusterFS Volume
Bricks Bricks Bricks
Experimental?
Devconf.cz, February 2016
Improvements needed for HC
● GlusterFS support re-added to hosted-engine setup● With libgfapi and replica 3 volume
● Ovirt-Engine Appliance● Preconfigured VM with oVirt Engine● Cloud-init based customization
● Shared configuration● All nodes see the same configuration data● Upgrade path from oVirt 3.5
● Management GUI for the oVirt-engine VM and HE
Devconf.cz, February 2016
HC ingredients
● at least 3 virtualization capable hosts (CentOS 7.1+)● 10 GB of temporary space on the primary host● two separate partitions for data (20GB+) on all hosts● DHCP configured to map a MAC address to a fixed IP● DNS configured with A and PTR names for the IP● oVirt release package installed on all hosts
http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm
● Physical console on the primary host or network access and screen package installed
Devconf.cz, February 2016
HC Setup - Gluster Volume
● 3 Node Gluster cluster● 2 Volumes with 3 way replication
● One for Hosted Engine VM and other for regular VMs● Enable Cluster Quorum● Sharding for better self-heal and geo-rep performace● Few more tunings
● network.ping-timeout = 10● cluster.data-self-heal-algorithm = full● performance.low-prio-threads = 32● Group = virt● storage.owner-uid, storage.owner-uid = 36
Devconf.cz, February 2016
HC Setup – Hosted Engine
Packages Required● ovirt-engine-appliance● ovirt-hosted-engine-setup● Vdsm-gluster
hosted-engine –-deploy● 'glusterfs' storage type● OVA image for hosted-engine VM● Clound-init to customize the VM
● FQDN and Mac for hosted-engine VM
More details at http://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hyperconvergence.html
Devconf.cz, February 2016
HC Setup – Final Touch
● Hosted Engine running with single node Cluster● Enable Gluster service● Add remaining nodes● Create and add the main storage domain
Devconf.cz, February 2016
Enabling Gluster
● Gluster Service support is located in the Cluster properties
● Deploy Hosts with GlusterFS Server support
● Enable Bricks and Volume Management from oVirt WebAdmin and REST-API
● Engine is not taking in consideration GlusterFS on Virtualization Power-Saving policies and Fencing yet
Devconf.cz, February 2016
Adding addtional Nodes
● Simple checkbox during in the Add host dialog
● Host deploy script does everything else auto-magically
Removed at the last moment because of unresolved issues. It can be done through “hosted-engine –deploy” script
Devconf.cz, February 2016
Adding Gluster storage
● Gluster volume management using WebAdmin Portal and REST-API
● Volume Pofiling and Capacity monitoring
● Geo Replication
● Volume Snapshot
Devconf.cz, February 2016
Add Storage Domain
● Master Storage Domain using gluster volume
● Imports Hosted Engine Storage Domain and Hosted Engine VM autmotically.
Devconf.cz, February 2016
Dashboard
Devconf.cz, February 2016
Hosted engine VM management
● Support for editing the Hosted Engine VM
● Memory and CPU allocation, network configuration
● Work in progress..
● Distributed to all hosted engine nodes using OVF file on the storage domain
● Hosted engine daemons pick up the configuration when the management VM is restarted
Devconf.cz, February 2016
Hosted engine management
● Reporting configuration
– State transitions
– SMTP details● Timeout configuration
– Allowed downtime before forced recovery● Host scoring constants and rules
Devconf.cz, February 2016
Recent Enahacements
● Glusterfs Sharding
● Arbiter Volumes
● Replace Brick
● Backup vol file support in libvirt and QEMU-KVM
https://devconfcz2016.sched.org/event/5m1p/improvements-in-gluster-for-virtualization-usecase
Devconf.cz, February 2016
THANK YOU!
http://[email protected]@ovirt.org#ovirt irc.oftc.net
http://www.ovirt.org/Features/GlusterFS-Hyperconvergencehttp://blogs-ramesh.blogspot.in/2016/01/ovirt-and-gluster-hyperconvergence.html