Multiple Sites and Disaster Recovery with Ceph: Andrew Hatfield, Red Hat

Andrew HatfieldPractice Lead - Cloud Storage and Big Data

MULTIPLE SITES AND DISASTER RECOVERY WITH CEPH

OPENSTACK DAY AUSTRALIA (CANBERRA) NOVEMBER 2016

[email protected]@andrewhatfield

Today’s takeaways

● As OpenStack adoption matures and becomes more mission critical, Business Continuity increases in importance

● Storage is a key component of Business Continuity

● Ceph can deliver significant benefits already and it's getting even better

Please Note

RE: Block / RBD

This is about Business Continuity and Disaster Recovery

This is not yet for High Availability or Fault Tolerance

Ceph and OpenStack Overview

Ceph is tightly integrated into Openstack

Cinder, Glance, Swift, Nova and Manilla (Tech Preview)

Single storage platform for all OpenStack needs

Fast booting and cloning

Ceph and OpenStack Usage

OpenStack User Survey October 2016 https://www.openstack.org/analytics/

Expected capabilities

● Multiple isolated OpenStack environments

● Each site has in-live/in-sync backup of:

○ Glance images

○ Cinder block devices

○ Nova ephemeral disks

● In an event of a failure, any site can recover its data from another site

● Storage architecture based on Ceph

Properties:● Single OpenStack site● A data recovery site● Pool names & Cluster FSID match at each site

Challenge:● Failover procedure

How to recover?● Promote Secondary, demote Primary and reverse

replication● Recover data

Properties:● Keystone on the controllers (as usual)● Individual login on each region/site● Both sites have each other’s data● Both sites have the same cluster FSID

Challenge:● Replicate metadata for images and

volumes

How to recover?● Promote the secondary site● Import DB records in the survival site

Properties:● Shared Keystone● Federated Keystone● Both sites have each other's data● Works with 2 sites● Both sites have with the same cluster

FSID

Challenges:● Replicate UUID tokens● MySQL cross-replication over WAN● Requires low latency and high bandwidth● Fernet tokens are not ready yet

How to recover?● Promote the secondary site● Import DB records in the survival site

RBD mirroring

Available with Ceph Jewel and Red Hat Storage 2.0

● New daemon ‘rbd-mirror’ synchronises Ceph images from one cluster to another● Relies on two new RBD image features:

○ journaling: enables journaling for every transaction on the image○ mirroring: tells the rbd-mirror daemon to replicate images

● Images have states: primary and non-primary (promote and demote calls)

RBD mirroring

Features;

● Can replicate an individual Image or an entire Pool

● Integrates with cinder-replication configuration for OpenStack awareness

RBD mirroring write path

RBD Mirroring Setup

● Use different cluster names; routable connectivity● Deploy the rbd-mirror daemon on each cluster● Same pool name at both sites● Add peering pool● Add RBD image settings

○ Enable journaling on image○ Mirror pool or specific images

Challenges:

● No HA support for RBD-mirror yet● Two sites only● LibRBD-only, no current kRBD support

What’s Next For Block?

Today is shiny and the future is even brighter!

● Multiple node support

● Multiple site support

● Rbd-mirror proxy support

● Mirror QoS

● Optionally keep deleted images for configurable time

● Configurable replication delay

Global Single Object Namespace

Applications

Zone GroupAustralia

ZoneSouth

ZoneNorth

Applications

RGWS3 / Swift

RGWS3 / SwiftAsynchronous two-way replication

Try it out!

Ceph Test Drive

http://http://red.ht/2fMaWxr