1500 JIRAs in 20 minutes - HBaseCon 2013

1500 JIRAs in 20 MinutesThe Evolution of HBase, 2012-2013Ian Varley, Salesforce.com @thefutureian

It's been a year since the first HBaseCon.

What's changed?

It's been a year since the first HBaseCon.

What's changed?

(besides my beard length)

One lens on the evolution of HBase is through JIRA (issue tracking system).

HBase has a lot of activity.

HBase has a lot of activity.Total JIRAs, all time: ~8700

Opened in last year: ~2500Total JIRAs, all time: ~8700

Opened in last year: ~2500Fixed in last year: 1638

Total JIRAs, all time: ~8700

Opened in last year: ~2500Fixed in last year: 1638

Total JIRAs, all time: ~8700

resolved >= 2012-05-23AND resolved <= 2013-05-24AND resolution in (Fixed, Implemented)

So we're going to talk about them all. One by one.

We need to narrow it down.

First, let's get rid of the non-functional changes:

Test: 307

Test:Build:

Test:Build:Doc:

Test:Build:Doc:Ports:

503(some overlap)

Total:

503(some overlap)

"test", "junit", etc.

"pom", "classpath", "mvn", "build", etc.

"book", "[site]", "[refGuide]", "javadoc", etc.

"backport", "forward port", etc.

Total:

That leaves 1135 functional changes to go over.

(In 18 minutes.)

Break what's left into 2 parts:● Big Topics (20+ JIRAs on same issue)● Indie Hits (Cool for some other reason)

Top 10 "big topics":

Snapshots:Top 10 "big topics":

Snapshots:Replication:

Top 10 "big topics":8258

Snapshots:Replication:Compaction:

Snapshots:Replication:Compaction:Metrics:

Snapshots:Replication:Compaction:Metrics:Assignment:

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics:Assignment:Hadoop 2:

825854534437

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics:Assignment:Hadoop 2:Protobufs:

82585453443734

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics:Assignment:Hadoop 2:Protobufs:Security:

8258545344373428

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics:Assignment:Hadoop 2:Protobufs:Security:Bulk Loading:

825854534437342823

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics:Assignment:Hadoop 2:Protobufs:Security:Bulk Loading:Modularization:

82585453443734282321

416(some overlap)

(305 functional, 111 non-functional)

82585453443734282321

416(some overlap)

Let's dive in to the top 3.

SnapshotsThe gist: Take advantage of the fact that files in HDFS are already immutable to get fast "snapshots" of tables that you can roll back to. This is pretty tricky when you consider HBase is a distributed system and you want a point in time.

Main JIRAs:● HBASE-6055 - Offline Snapshots: Take a snapshot after first disabling

the table● HBASE-7290 - Online Snapshots: Take a snapshot of a live, running

table by splitting the memstore.● HBASE-7360 - Backport Snapshots to 0.94

Top contributors: Matteo B, Jonathan H, Ted Y, Jesse Y, Enis S

ReplicationThe gist: use asynchronous WAL shipping to replay all edits on a different (possibly remote) cluster, for Disaster Recovery or other operational purposes.

Main JIRAs: ● HBASE-1295 - Multi-data-center replication: Top level issue. Real meat

was actually implemented in 0.90 (Jan 2010), so not a new feature.● HBASE-8207 - Data loss when machine name contains "-". Doh.● HBASE-2611 - Handle RS failure while processing failure of another:

This was an ugly issue that took a while to fix. Corner cases matter!

Top contributors: J-D Cryans, Himanshu V, Chris T, Devaraj D, Lars H

Theme: corner cases!

Plug: stick around next while Chris Trezzo tweets about Replication!!

Theme: corner cases!

CompactionThe gist: In an LSM store, if you don't compact the store files, you end up with lots of 'em, which makes reads slower. Not a new feature, just improvements.

Main JIRAs: ● HBASE-7516 - Make compaction policy pluggable: allow users to

customize which files are included for compaction.● HBASE-2231 - Compaction events should be written to HLog: deal with

the case when regions have been reassigned since compaction started.

Look for cool stuff to come in the next year with tiered (aka "leveled") compaction policies, so you could do stuff like (e.g.) put "recent" data into smaller files that'll be hit frequently, and the older "long tail" data into bigger files that'll be hit less frequently.

Top contributors: Sergey S, Elliott C, Jimmy X, stack, Matteo B, Jesse Y

CompactionThe gist: In an LSM store, if you don't compact the store files, you end up with lots of 'em, which makes reads slower. Not a new feature, just improvements.

Main JIRAs: ● HBASE-7516 - Make compaction policy pluggable: allow users to

customize which files are included for compaction.● HBASE-2231 - Compaction events should be written to HLog: deal with

the case when regions have been reassigned since compaction started.

Look for cool stuff to come in the next year with tiered (aka "leveled") compaction policies, so you could do stuff like (e.g.) put "recent" data into smaller files that'll be hit frequently, and the older "long tail" data into bigger files that'll be hit less frequently.

Top contributors: Sergey S, Elliott C, Jimmy X, stack, Matteo B, Jesse Y

corner case!

Top 10 "big topics":Snapshots:Replication:Compaction:MetricsAssignmentHadoop 2ProtobufsSecurityBulk LoadingModularization

416(some overlap)

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics: move to metrics2.AssignmentHadoop 2ProtobufsSecurityBulk LoadingModularization

416(some overlap)

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics: move to metrics2.Assignment: it's tricky, yo.Hadoop 2ProtobufsSecurityBulk LoadingModularization

416(some overlap)

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics: move to metrics2.Assignment: it's tricky, yo.Hadoop 2: support it for HA NN.ProtobufsSecurityBulk LoadingModularization

416(some overlap)

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics: move to metrics2.Assignment: it's tricky, yo.Hadoop 2: support it for HA NN.Protobufs: wire compatibility!SecurityBulk LoadingModularization

416(some overlap)

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics: move to metrics2.Assignment: it's tricky, yo.Hadoop 2: support it for HA NN.Protobufs: wire compatibility!Security: kerberos, in the core.Bulk LoadingModularization

416(some overlap)

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics: move to metrics2.Assignment: it's tricky, yo.Hadoop 2: support it for HA NN.Protobufs: wire compatibility!Security: kerberos, in the core.Bulk Loading: pop in an HFile.Modularization

416(some overlap)

Top 10 "big topics":Snapshots:Replication:Compaction:Metrics: move to metrics2.Assignment: it's tricky, yo.Hadoop 2: support it for HA NN.Protobufs: wire compatibility!Security: kerberos, in the core.Bulk Loading: pop in an HFile.Modularization: break up the code.

416(some overlap)

Now on to the "Indie Hits JIRAs".

What's left? About half.Blocker:Critical:Major:Minor:

455206

830Trivial: 52

1638 total - (503 Non-Functional + 305 Categorized Functional) = 830 Remaining

Blocker:Critical:Major:Minor:

455206

573Trivial: 52

Let's cut out these:

What's left? About half.1638 total - (503 Non-Functional + 305 Categorized Functional) = 830 Remaining

We can't cover 573 issues.Let's just hit a few cool ones.

HBASE-5416HBASE-4676HBASE-7403HBASE-1212HBASE-7801HBASE-4072HBASE-3171HBASE-6868

HBASE-5416Interesting because: most commented JIRA (200+ human comments!)

Improve perf of scans with some kinds of filters

What? Avoid loading non-essential CFs until after filters run, big perf gain.

How?+++ Filter.java:+ abstract public boolean isFamilyEssential (byte[] name);

+++ HRegion.java: KeyValueScanner scanner = store.getScanner(scan, entry.getValue());- scanners.add(scanner);+ if (this.filter == null || !scan.doLoadColumnFamiliesOnDemand()+ || this.filter.isFamilyEssential(entry.getKey())) {+ scanners.add(scanner);+ } else {+ joinedScanners.add(scanner);+ }

By: Max Lapan for original idea & patch, Sergey Shelukhin for final impl

200 comments? Srsly?

From whom?

To save you some time, allow me to summarize.

Reenactment ...Feb 2012:

● Max Lapan: Hey guys, here's a cool patch!

● Max Lapan: Hey guys, here's a cool patch!● Nicolas S: This should be an app detail, not in core.

● Max Lapan: Hey guys, here's a cool patch!● Nicolas S: This should be an app detail, not in core.● Ted Yu: I fixed your typos while you were asleep!

● Max Lapan: Hey guys, here's a cool patch!● Nicolas S: This should be an app detail, not in core.● Ted Yu: I fixed your typos while you were asleep!● Nick: Not enough utest coverage to put this in core.● Max: Agree, but I can't find any other way to do this.

● Max Lapan: Hey guys, here's a cool patch!● Nicolas S: This should be an app detail, not in core.● Ted Yu: I fixed your typos while you were asleep!● Nick: Not enough utest coverage to put this in core.● Max: Agree, but I can't find any other way to do this.● Kannan: Why don't you try 2-phase w/ multiget?● Max: OK, ok, I'll try it.

Reenactment ...May 2012:

● Max: Ran in prod w/ 160-node 300TB cluster. Runs like a champ, 20x the 2-phase approach. Boom.

● Ted: Holy guacamole that's a big patch.

July 2012:● Max: Anybody there? Here's a perf test.● Ted: Cool!

Oct 2012:● Anoop: A coprocessor would make faster.● Max: We're on 0.90 and can't use CP.● Stack: -1, FB guys are right about needing more tests.

Reenactment ...Dec 2012:

● Sergey: I'm on it guys. Rebased on trunk, added the ability to configure, and integration tests.

● Stack: Still not enough tests. Some new code even when disabled? Who's reviewing? Go easy lads.

● Ram: I'm on it. Couple improvements, but looks good.

Reenactment ...Dec 31st, 2012 (while everyone else is partying):

● Lars: Ooh, let's pull this into 0.94! I made a patch.

● Lars: Ooh, let's pull this into 0.94! I made a patch.● Lars: ... hold the phone! This slows down a tight loop

case (even when disabled) by 10-20%.

case (even when disabled) by 10-20%.● Ted: I optimized the disabled path.● Lars: Sweet.

Reenactment ...Jan, 2013:

● Ram: +1, let's commit.● Ted: Committed to trunk● Lars: Committed to 0.94.

Reenactment ...Jan, 2013:

● Ram: +1, let's commit.● Ted: Committed to trunk● Lars: Committed to 0.94.

And there was much rejoi....

Reenactment ...Feb, 2013:

● Dave Latham: Stop the presses! This breaks rolling upgrade for me b/c I directly implement Filter.

● All: Crapface.

● All: Crapface.● Stack: We should back this out. SOMA pride!! Also,

Dave is running world's biggest HBase cluster, FYI.

Dave is running world's biggest HBase cluster, FYI.● Lars: Filter is internal. Extend FilterBase maybe?● Ted: If we take it OUT now, it's also a regression.

Dave is running world's biggest HBase cluster, FYI.● Lars: Filter is internal. Extend FilterBase maybe?● Ted: If we take it OUT now, it's also a regression.● Dave: Chill dudes, we can fix by changing our client.

Dave is running world's biggest HBase cluster, FYI.● Lars: Filter is internal. Extend FilterBase maybe?● Ted: If we take it OUT now, it's also a regression.● Dave: Chill dudes, we can fix by changing our client.● All: Uhh ... change it? Keep it? Change it?

Resolution: Change it (HBASE-7920)

Moral of the story? ● JIRA comments are a great way to learn.● Do the work to keep new features from

destabilizing core code paths.● Careful with changing interfaces.

HBASE-4676Interesting because: most watched (42 watchers), and biggest patch.

Prefix Compression - Trie data block encoding

What? An optimization to compress what we store for key/value prefixes.

How? ~8000 new lines added! (Originally written in git repo, here)

At SFDC, James Taylor reported seeing 5-15x improvement in Phoenix, with no degradation in scan performance. Woot!

By: Matt Corgan

HBASE-7403Interesting because: It's a cool feature. And went through 33 revisions!

Online Merge

What? The ability to merge regions online and transactionally, just like we do with splitting regions.How? The master moves the regions together (on the same regionserver) and send MERGE RPC to regionserver. Merge happens in a transaction.Example:RegionMergeTransaction mt = new RegionMergeTransaction(conf, parent, midKey)if (!mt.prepare(services)) return;try { mt.execute(server, services);} catch (IOException ioe) { try { mt.rollback(server, services); return; } catch (RuntimeException e) { myAbortable.abort("Failed merge, abort"); }}

By: Chunhui Shen

HBASE-1212Interesting because: Oldest issue (Feb, 2009) resolved w/ patch this year.

Merge tool expects regions to have diff seq ids

What? With aggregated hfile format, sequence id is written into file, not along side. In rare case where two store files have same sequence id and we want to merge the regions, it wouldn't work.How? In conjucntion with HBASE-7287, removes the code that did this:

--- HRegion.java List<StoreFile> srcFiles = es.getValue();- if (srcFiles.size() == 2) {- long seqA = srcFiles.get(0).getMaxSequenceId();- long seqB = srcFiles.get(1).getMaxSequenceId();- if (seqA == seqB) {- // Can't have same sequenceid since on open store, this is what- // distingushes the files (see the map of stores how its keyed by- // sequenceid).- throw new IOException("Files have same sequenceid: " + seqA);- }- }

By: Jean-Marc Spaggiari

HBASE-1212Interesting because: Oldest issue (Feb, 2009) resolved w/ patch this year.

Merge tool expects regions to have diff seq ids

What? With aggregated hfile format, sequence id is written into file, not along side. In rare case where two store files have same sequence id and we want to merge the regions, it wouldn't work.How? In conjucntion with HBASE-7287, removes the code that did this:

--- HRegion.java List<StoreFile> srcFiles = es.getValue();- if (srcFiles.size() == 2) {- long seqA = srcFiles.get(0).getMaxSequenceId();- long seqB = srcFiles.get(1).getMaxSequenceId();- if (seqA == seqB) {- // Can't have same sequenceid since on open store, this is what- // distingushes the files (see the map of stores how its keyed by- // sequenceid).- throw new IOException("Files have same sequenceid: " + seqA);- }- }

By: Jean-Marc Spaggiari

HBASE-7801Interesting because: has durability implications worth blogging about.

Allow a deferred sync option per Mutation

What? Previously, you could only turn WAL writing off completely, per table or edit. Now you can choose "none", "async", "sync" or "fsync".How?+++ Mutation.java+ public void setDurability(Durability d) {+ setAttribute(DURABILITY_ID_ATTR, Bytes.toBytes(d.ordinal()));+ this.writeToWAL = d != Durability.SKIP_WAL;+ }+++ HRegion.java+ private void syncOrDefer(long txid, Durability durability) {+ switch(durability) { ...+ case SKIP_WAL: // nothing to do+ break;+ case ASYNC_WAL: // defer the sync, unless we globally can't+ if (this.deferredLogSyncDisabled) { this.log.sync(txid); }+ break;+ case SYNC_WAL:+ case FSYNC_WAL:+ // sync the WAL edit (SYNC and FSYNC treated the same for now)+ this.log.sync(txid);+ break;+ }

By: Lars Hofhansl

HBASE-7801Interesting because: has durability implications worth blogging about.

Allow a deferred sync option per Mutation

What? Previously, you could only turn WAL writing off completely, per table or edit. Now you can choose "none", "async", "sync" or "fsync".How?+++ Mutation.java+ public void setDurability(Durability d) {+ setAttribute(DURABILITY_ID_ATTR, Bytes.toBytes(d.ordinal()));+ this.writeToWAL = d != Durability.SKIP_WAL;+ }+++ HRegion.java+ private void syncOrDefer(long txid, Durability durability) {+ switch(durability) { ...+ case SKIP_WAL: // nothing to do+ break;+ case ASYNC_WAL: // defer the sync, unless we globally can't+ if (this.deferredLogSyncDisabled) { this.log.sync(txid); }+ break;+ case SYNC_WAL:+ case FSYNC_WAL:+ // sync the WAL edit (SYNC and FSYNC treated the same for now)+ this.log.sync(txid);+ break;+ }

By: Lars Hofhansl

Wha ... ?Oh. See HADOOP-6313

HBASE-4072Interesting because: Biggest facepalm.

Disable reading zoo.cfg files

What? Used to be, if two system both use ZK and one needed to override values, the zoo.cfg values would always win. Caused a lot of goofy bugs in hbase utils like import/export, integration with other systems like flume. How? Put reading it behind a config that defaults to false.

+ if (conf.getBoolean(HBASE_CONFIG_READ_ZOOKEEPER_CONFIG, false)) {+ LOG.warn(+ "Parsing zoo.cfg is deprecated. Place all ZK related HBase " + + "configuration under the hbase-site.xml");

By: Harsh J

HBASE-4072Interesting because: Biggest facepalm.

Disable reading zoo.cfg files

What? Used to be, if two system both use ZK and one needed to override values, the zoo.cfg values would always win. Caused a lot of goofy bugs in hbase utils like import/export, integration with other systems like flume. How? Put reading it behind a config that defaults to false.

+ if (conf.getBoolean(HBASE_CONFIG_READ_ZOOKEEPER_CONFIG, false)) {+ LOG.warn(+ "Parsing zoo.cfg is deprecated. Place all ZK related HBase " + + "configuration under the hbase-site.xml");

By: Harsh J

HBASE-3171Interesting because: Only HBase JIRA with a downfall parody.

Drop ROOT, store META location in ZooKeeper

What? The ROOT just tells you where the META table is. That's silly.

How? Pretty big patch (59 files changed, 580 insertions(+), 1749 deletions(-))

By: J-D Cryans

http://www.youtube.com/watch?v=tuM9MYDssvg

HBASE-6868Interesting because: tiny fix, but marked as a blocker, and sunk 0.94.2 RC1.

Avoid double checksumming blocks

What? since HBASE-5074 (checksums), sometimes we double checksum.

How? 3 line patch to default to skip checksum if not local fs.

+++ HFileSystem.java // Incorrect data is read and HFileBlocks won't be able to read // their header magic numbers. See HBASE-5885 if (useHBaseChecksum && !(fs instanceof LocalFileSystem)) {+ conf = new Configuration(conf);+ conf.setBoolean("dfs.client.read.shortcircuit.skip.checksum", true); this.noChecksumFs = newInstanceFileSystem(conf);...+++ HRegionServer.java // If hbase checksum verification enabled, automatically //switch off hdfs checksum verification. this.useHBaseChecksum = conf.getBoolean(- HConstants.HBASE_CHECKSUM_VERIFICATION, true);+ HConstants.HBASE_CHECKSUM_VERIFICATION, false);

By: Lars Hofhansl

What's it all mean?Active codebase. Good!Complexity increasing. Bad!

credit: https://www.ohloh.net/p/hbase

One more interesting stat:

One more interesting stat:"Good on you"s

One more interesting stat:

"Good on you"s

everyone else

Takeaways?Busy community.New features!Fixing corner cases.

BTW: How did I do this?JIRA API +Phoenix on HBase +http://github.com/ivarley/jirachi

Thanks!@thefutureian

1500 JIRAs in 20 minutes - HBaseCon 2013

Technology

HBaseCon 2012 | Content Addressable Storages for Fun and...

HBaseCon 2013: Apache HBase on Flash

HBaseCon 2012 | HBase Security for the Enterprise - Andrew.....

HBaseCon 2013: Evolving a First-Generation Apache HBase...

TRANSPOCKET 1500 / 1500 RC / 1500 TIG

HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit...

HBaseCon 2012 | Orchestrating Clusters with Ironfan and Chef...

HBaseCon 2012 | Low Latency OLAP with HBase - Cosmin Lehene,...

HBaseCon 2012 | Building a Large Search Platform on a...

Список учатсников 20T1W3 · 6801077 1500...

Bonus 1: Digitale Tools¤chtiges Tool, mit dem auch agile...

TRANSPOCKET 1500 / 1500 RC / 1500 TIG · /...

Low Latency “OLAP” with HBase - HBaseCon 2012

HBaseCon 2012 | Scaling GIS In Three Acts

HBaseCon 2012 | Growing Your Inbox, HBase at Tumblr

Shivam P.G Greater...