BARNS: Backup and Recovery for NoSQL Databases Atish Kathpal, Priya Sehgal Advanced Technology Group, NetApp © 2017 NetApp, Inc. All rights reserved. 1
BARNS: Backup and Recovery for NoSQL DatabasesAtish Kathpal, Priya SehgalAdvanced Technology Group, NetApp
© 2017 NetApp, Inc. All rights reserved.1
Why Backup/Restore NoSQL DBs?
Customers are directly ingesting into NoSQL
Security breach are on the rise e.g. ransomware attackson MongoDB [1] and recent WannaCrypt exploits
“Fat-finger” errors eventually propagate to replicas
Sandbox deployments for test/dev
§ Bring up shadow clusters of different cardinality (from production cluster snapshots)
Compliance and regulatory requirements
IDC, 2016 report [2] lists data-protection and retention as one of the top infrastructural requirements for NoSQL
© 2017 NetApp, Inc. All rights reserved.2[1] http://thehackernews.com/2017/01/secure-mongodb-database.html [2] Nadkarni A., Polyglot Persistence: Insights on NoSQL Adoption and the Resulting Impact on Infrastructure. IDC. 2016 Feb.
Ransomware
From Backup/Restore Service PerspectiveNoSQL Database Classes
§ Master-slave§ Authoritative copy of each partition is contained in the master node
that we can backup.§ Loss of primary node leads to shard/partition-unavailability until new
leader is elected. § Example: MongoDB, Redis, Oracle NoSQL, MarkLogic
§ Master-less§ Data is scattered across nodes using consistent hashing techniques,
no single node has all data for a given partition§ Eventual consistency: Unavailability of a destination node does not
lead to write-failure, data is eventually replicated§ Example: Cassandra, Couchbase
© 2017 NetApp, Inc. All rights reserved.3
https://www.slideshare.net/mongodb/sharding-v-final, https://blog.imaginea.com/consistent-hashing-in-cassandra/
NoSQL DBs Hosted on Shared StorageHigh-level, conceptual deployment architecture
© 2017 NetApp, Inc. All rights reserved.4
NoSQL DB software(Node1)
Filesystem(DB Journal, Logs,
Data Files)
NoSQL DB software(Node2)
Filesystem(DB Journal, Logs,
Data Files)
NoSQL DB software(Node3)
Filesystem(DB Journal, Logs,
Data Files)
Shared Storage Array(Snapshots, Cloning, Compression, Deduplication, Encryption, Cloud Integrations)
LUN-1 LUN-2 LUN-3
Backup: Leverage storage snapshotsRestore: Leverage
cloning
Backup/Restore Challenges§ Cluster-consistency at scale
§ Cluster/App quiesce – significantly hampers application performance. Cross node consistency not guaranteed§ Take crash consistent snapshots§ Post process crash consistent snapshots (in a sand-box) using NoSQL DB stack
to reach an cluster-consistent state
§ Space Efficiency§ Replica set data copies do not de-duplicate – small row sizes, scattered
across nodes (Cassandra) and unique ids added by storage engines (MongoDB)
§ DB performs compression and encryption§ Remove replicas logically (application aware backup)
§ Topology Changes § Commodity nodes, at scale of 10-100s of nodes. E.g., Primary node
might be unreachable while taking backup in case of MongoDB§ Storage snapshots do not have context about cluster topology§ Use cases may require restore to a test/dev cluster of different cardinality§ Save Cluster topology and storage mapping as part of backup
© 2017 NetApp, Inc. All rights reserved.5
Cassandra N1
Cassandra N2
Cassandra N3
Write:(K1, V1), (K3, V3)
Write:(K1, V1), (K2, V2)
Write:(K2, V2), (K3, V3)
Shared Storage clusterLUN1: K1, K3LUN2: K1, K2LUN3: K2, K3
Insignificant block based
deduplication
Replication factor = 2
Existing open source utilities like Mongodump and Cassandra snapshots suffer from above challenges.
NoSQL Data Protection – Challenges
Challenges:
§ Fault tolerance§ Backup may capture stale data due
to eventual consistency§ Higher restore times, since
Cassandra will perform repairs during restore
Master-Less Databases
© 2017 NetApp, Inc. All rights reserved.6
Cassandra N1
Cassandra N2
Cassandra N3
Update:(K1, V11, Tnew)
Update:(K1, V11, Tnew)
Shared Storage clusterLUN1: (K1, V1, Told), …LUN2: (K1, V1, Told), …LUN3: …
Client, Update K1 (CL.ONE)
(K1, V11, Tnew)
Ack
Snapshot of LUN2 will point to stale
data
http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesReadRepair.html, https://wiki.apache.org/cassandra/HintedHandoff
BARNS Architecture
© 2017 NetApp, Inc. All rights reserved.7
Light Weight Backup Process(LWB)
Post-Process
Restore Process
Lib_DB
Lib_Storage
Lib_MongoDB Lib_Cassandra
Lib_Store1 Lib_Store2
Store1 Store2
APIs APIs
MongoDB Cluster
Cassandra Cluster
Mongo APIs Cassandra APIs
(PP)
Stop_balancer()Get_topology()Check_cluster_health()Prepare_Backup_Topo()Etc.
Get_lun_mappings()Take_snapshots()Lun_clone()Vault_backup()Etc.
Addresses challenges of:1. Taking cluster-consistent
backup at scale2. Taking storage efficient
backups (through replica removal)
3. Enables recovery/cloning to different cluster topologies
Phase 1: Light-weight Backup Phase
© 2017 NetApp, Inc. All rights reserved.9
C1
CL1D1
C4
CL4D4
C3
CL3D3
C2
CL2D2
L1 L2 L3 L4
BARNS:LWB
Production Cluster
CL1
D1
sn1
CL2
D2
sn2
CL3
D3
sn3
CL4
D4
sn4
1. Capture token assignment of each node 2. Store mapping of LUNs à Tokens
3. Take snapshots of L1 to L4
Token1 Token4Token3Token2
Backup Metadata example
Phase2: Post Process Phase Part1: Flush Commitlogs
© 2017 NetApp, Inc. All rights reserved.10
P1
CL1D1
P4
CL4D4
P3
CL3D3
P2
CL2D2
CL_sn1 CL_sn2 CL_sn3 CL_sn4
BARNS:PP-Phase
PP Node(s)
1. Clone the LWB Snapshot LUNs2. Mount on different PP processes or different nodes3. Start Cassandra processes P1 to P4
4. Flush CommitLogs
E.g. Data
K1, V1, T1
K2, V2, T1
K1,V11, T2
Phase2: Post Process Phase Part2: Compaction
© 2017 NetApp, Inc. All rights reserved.11
CL_sn1 CL_sn2 CL_sn3 CL_sn4
1. Using UnionFS mount all snapshot clones as read-only2. Create a new volume and LUN for full backup3. Mount it as RW through UnionFS4. Let PP node have all tokens of prod cluster5. Start Cassandra6. Start compaction process7. Final compacted files will be stored on
Fullbackup LUN
Data 1
K1, V1, T1
K2, V2, T1
K1,V11, T2
/cassandra
Data 2
K1, V1, T1
K1, V11, T2
K3,V3, T3
Data 3
K1, V1, T1
K2, V2, T1
Data 4
K3,V3, T3
K2, V22, T4
DataCompact
K1, V11, T2
K2, V22, T1
K3,V3, T3
Full_bk_lun
unionFS
PP Node
Keeping single copy of data in the backup => ~66% reduction in backup storage requirements
BARNS:PP-Phase
Cassandra RestoreThe post-process step enables cloning to different restore/clone topologies
© 2017 NetApp, Inc. All rights reserved.12
Full Backup
K1, V11, T2
K2, V22, T1
K3,V3, T3
Token1 Token2
N1 N2
Full_bk_lun
Clones
Token3 Token4
Evaluation - Cassandra Backup and Restore § Production cluster§ 4 nodes§ 4 iSCSI LUNs§ Commitlog and SSTables for a
node on same LUN§ Cassandra 4.0
§ Post Process Node§ 2 CPUs§ 8GB RAM
§ YCSB to ingest data
Full Backup
© 2017 NetApp, Inc. All rights reserved.13
• LWB – <10 secs• pp-flush - ~40 secs• pp-compact – time increases by 35-40% à incremental backup• Restore time – less than ~2 mins (irrespective of cluster size and data set size)
Master-Slave Distributed Database
Ø Check the paper or just attend the poster session J
BARNS Solution: MongoDB
© 2017 NetApp, Inc. All rights reserved.14
Summary
• Tracking replicas and cluster topologies is important for taking backups and performing flexible topology restores
• Existing open-source solutions have several inefficiencies like need for repairs after restore, lack of storage efficiency in backup and poor integrations with shared storage
• Opportunity to provide efficient backup and restore through light-weight snapshots and clones
© 2017 NetApp, Inc. All rights reserved.15