E.2. Making use of Hadoop Native Libraries in HBase

E.2. Making use of Hadoop Native Libraries in HBase
Prev	Appendix E. Compression and Data Block Encoding In HBase	Next

The Hadoop shared library has a bunch of facility including compression libraries and fast crc'ing. To make this facility available to HBase, do the following. HBase/Hadoop will fall back to use alternatives if it cannot find the native library versions -- or fail outright if you asking for an explicit compressor and there is no alternative available.

If you see the following in your HBase logs, you know that HBase was unable to locate the Hadoop native libraries:

2014-08-07 09:26:20,139 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

If the libraries loaded successfully, the WARN message does not show.

Lets presume your Hadoop shipped with a native library that suits the platform you are running HBase on. To check if the Hadoop native library is available to HBase, run the following tool (available in Hadoop 2.1 and greater):

$ ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker
2014-08-26 13:15:38,717 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop: false
zlib:   false
snappy: false
lz4:    false
bzip2:  false
2014-08-26 13:15:38,863 INFO  [main] util.ExitUtil: Exiting with status 1

Above shows that the native hadoop library is not available in HBase context.

To fix the above, either copy the Hadoop native libraries local or symlink to them if the Hadoop and HBase stalls are adjacent in the filesystem. You could also point at their location by setting the LD_LIBRARY_PATH environment variable.

Where the JVM looks to find native librarys is "system dependent" (See java.lang.System#loadLibrary(name)). On linux, by default, is going to look in lib/native/PLATFORM where PLATFORM is the label for the platform your HBase is installed on. On a local linux machine, it seems to be the concatenation of the java properties os.name and os.arch followed by whether 32 or 64 bit. HBase on startup prints out all of the java system properties so find the os.name and os.arch in the log. For example:

....
      2014-08-06 15:27:22,853 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
      2014-08-06 15:27:22,853 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
      ...

So in this case, the PLATFORM string is Linux-amd64-64. Copying the Hadoop native libraries or symlinking at lib/native/Linux-amd64-64 will ensure they are found. Check with the Hadoop NativeLibraryChecker.

Here is example of how to point at the Hadoop libs with LD_LIBRARY_PATH environment variable:

$ LD_LIBRARY_PATH=~/hadoop-2.5.0-SNAPSHOT/lib/native ./bin/hbase --config ~/conf_hbase org.apache.hadoop.util.NativeLibraryChecker
2014-08-26 13:42:49,332 INFO  [main] bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
2014-08-26 13:42:49,337 INFO  [main] zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /home/stack/hadoop-2.5.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
zlib:   true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4:    true revision:99
bzip2:  true /lib64/libbz2.so.1

Set in hbase-env.sh the LD_LIBRARY_PATH environment variable when starting your HBase.