Mapreduce over snapshots

© Hortonworks Inc. 2011

MapReduce over snapshotsHBASE-8369

Enis Soztutar

Enis [at] apache [dot] org

@enissoz

Page 1


About Me

Page 2Architecting the Future of Big Data

• In the Hadoop space since 2007

• Committer and PMC Member in Apache HBase and Hadoop

• Working at Hortonworks as member of Technical Staff

• Twitter: @enissoz


Snapshots

• Currently a snapshot is a bunch of reference files together with some metadata

• A table’ snapshot can contain – Table descriptor– List of regions– References to files in the regions– References to WALs for regionservers

• Current snapshot impl is flush based– Forces flush to all regions, so that in-memory data is written to disk



MR over Snapshots

• Idea is do scan’s on the client side bypassing region servers

• Use snapshots since they are immutable

• Similar to short circuit hdfs reads

• TableSnapshotInputFormat works similar to TableInputFormat

• TableMapReduceUtil methods to configure the job



Deployment Options

HBase online

• Take snaphot while HBase is running• Run MR job over the snapshot

HBase offline• Take snapshot while HBase is running• Export Snapshot using ExportSnapshot to a different hdfs• Run MR job over snapshot with or without HBase running



TableSnapshotInputFormat

• Gets a Scan representing the query• Restore the snapshot to a temporary directory• For each region in the snapshot:

– Determine whether the region should be scanned (falls between scan start row and stop row)

– Create one split per region in the scan range ( # of map tasks)– Each RecordReader will open the region (Hregion) as in HRegionServer– An internal RegionScanner is used for running the scan



API



Timeline

• Will (hopefully) be committed to trunk next week or so

• Interest in bringing this to 0.94 and 0.96 bases as well

• Will come in HDP-2.1, which will be based on 0.96 line



Security Aspects

• HBase user owns the files in filesystem• Snapshot files are also owned by the HBase user• Mapreduce job should be able to read the files in the snapshot + actual

data files• HDFS only has posix-like perms based on user/group/other

– User running MR job has to be either the HBase user, or have group perms– HDFS does not have ACL’s, so there is no easy way to grant read access at

filesystem layer

• Idea: similar to current short circuit impl, we can implement a FD transfer– User will submit jobs under her own user credentials– Ask HBase daemons to open the files, and pass a handler / token



Performance

ScanTest:• Scan : open a scanner, do full table scan• SnapshotScan : open a client-side scanner, do full table scan• ScanMR : parallel full table scan from MR• SnapshotScanMR : do full table scan

• 8 Region servers, 6 disks each• HBase trunk• Hadoop-2.2 (HDP-2.0.7.0-12)• Load data with IntegrationTestBulkLoad

– Evenly distributed rows, created as bulk loaded hfiles. 3 column families

• # store files per region varies 3,6,9, and 12 (1,2,3,4 file per store)• Data sizes: 6.6G, 13.2G, 19.8G, 26.4G



Scan speed



API

• We do not want to limit snapshot scanning only to MapReduce

• Allow client side scanners over snapshot files



ResultScanner is main scan API



API (caution: not final yet)



To the future and beyond

• HBASE-8691 High-Throughput Streaming Scan API

• Can we bypass regionservers without taking snapshots?

• Bypass memstore data, or stream memstore data, but read directly from hfiles

• Secure reading from snapshots

• Keep up with the updates at – https://issues.apache.org/jira/browse/HBASE-8369



ThanksQuestions?

Architecting the Future of Big DataPage 16

Enis Söztutarenis [ at ] apache [dot] org@enissoz

Mapreduce over snapshots

Technology

future of big data hortonworks

table snapshot

snapshots hbase

apache hbase

load data

filesystem snapshot

hbase runningarchitecting

hbase daemons