Appendix M. Enabling Dapper-like Tracing in HBase

Table of Contents

M.1. SpanReceivers
M.2. Client Modifications
M.3. Tracing from HBase Shell

HBASE-6449 added support for tracing requests through HBase, using the open source tracing library, HTrace. Setting up tracing is quite simple, however it currently requires some very minor changes to your client code (it would not be very difficult to remove this requirement).

M.1. SpanReceivers

The tracing system works by collecting information in structs called 'Spans'. It is up to you to choose how you want to receive this information by implementing the SpanReceiver interface, which defines one method:

public void receiveSpan(Span span);

This method serves as a callback whenever a span is completed. HTrace allows you to use as many SpanReceivers as you want so you can easily send trace information to multiple destinations.

Configure what SpanReceivers you'd like to us by putting a comma separated list of the fully-qualified class name of classes implementing SpanReceiver in hbase-site.xml property: hbase.trace.spanreceiver.classes.

HTrace includes a LocalFileSpanReceiver that writes all span information to local files in a JSON-based format. The LocalFileSpanReceiver looks in hbase-site.xml for a hbase.local-file-span-receiver.path property with a value describing the name of the file to which nodes should write their span information.

<property>
  <name>hbase.trace.spanreceiver.classes</name>
  <value>org.htrace.impl.LocalFileSpanReceiver</value>
</property>
<property>
  <name>hbase.local-file-span-receiver.path</name>
  <value>/var/log/hbase/htrace.out</value>
</property>

HTrace also provides ZipkinSpanReceiver which converts spans to Zipkin span format and send them to Zipkin server. In order to use this span receiver, you need to install the jar of htrace-zipkin to your HBase's classpath on all of the nodes in your cluster.

htrace-zipkin is published to the maven central repository. You could get the latest version from there or just build it locally and then copy it out to all nodes, change your config to use zipkin receiver, distribute the new configuration and then (rolling) restart.

Here is the example of manual setup procedure.

$ git clone https://github.com/cloudera/htrace
$ cd htrace/htrace-zipkin
$ mvn compile assembly:single
$ cp target/htrace-zipkin-*-jar-with-dependencies.jar $HBASE_HOME/lib/
  # copy jar to all nodes...

The ZipkinSpanReceiver looks in hbase-site.xml for a hbase.zipkin.collector-hostname and hbase.zipkin.collector-port property with a value describing the Zipkin collector server to which span information are sent.

<property>
  <name>hbase.trace.spanreceiver.classes</name>
  <value>org.htrace.impl.ZipkinSpanReceiver</value>
</property> 
<property>
  <name>hbase.zipkin.collector-hostname</name>
  <value>localhost</value>
</property> 
<property>
  <name>hbase.zipkin.collector-port</name>
  <value>9410</value>
</property> 

If you do not want to use the included span receivers, you are encouraged to write your own receiver (take a look at LocalFileSpanReceiver for an example). If you think others would benefit from your receiver, file a JIRA or send a pull request to HTrace.

comments powered by Disqus