1 HBase Tracing Elliott Clark
Feb 24, 2016
1
HBase TracingElliott Clark
2
Who Am I
• Elliott Clark• [email protected]
• HBase PMC• HBase Committer• HTrace Mantainer• Cloudera Engineer
• CDH Distribution
3
What’s Going on During Failovers
4
Motivation
• Want to improve MTTR• First need to know what’s going on• Google’s Dapper
5
Options
• Log all the things• We already do that.• Our logs are way too verbose as it is.• Hard to combine multiple logs
• HRegionServer• HMaster• Client
• Guess• Zipkin• Not Invented Here
6
Spoiler:Guessing Didn’t Win
7
About Zipkin
• Created by Twitter• Thanks
• Modeled after Dapper• Scala• Finagle
• Functional• Netty• Non-blocking
• Thrift• Cool UI• Pluggable Storage Backend
8
Dapper’s Terminology
• Span• Segment of a remote call• Contains Annotations• Has a parent trace• Can have multiple children spans• Can have multiple annotations
• Annotations• Trace
• Grouping of spans
9
Can’t use Cassandra and look at myself in the mirror
10
Hooking up Zipkin, HTrace, and HBase
Implementation
11
Zipkin HBase Storage
• Replaces need for Cassandra• Functional• Created a non-blocking client shim• Fixed Key Length
• ID’s are long• Mapping table
• ID -> Name• Name -> ID• Inspiration from OpenTSDB
• Open source on Github
12
HTrace
• Way to instrument Java Code• Used In HBase trunk• Coming to an install near you in 0.96.0• Pluggable where it emits Spans
• To a file• To sys.out
13
HTrace and Zipkin
• Modularize HTrace• Create htrace-zipkin• Create ZipkinSpanReceiver
• Receives spans from HTrace instrumented code• Converts to Thrift objects• Sends to Zipkin
• Background Flush• Batching
14
Let’s hope this works.
Demo
15