C.5.  SNAPPY

If snappy is installed, HBase can make use of it (courtesy of hadoop-snappy [32]).

  1. Build and install snappy on all nodes of your cluster (see below)

  2. Use CompressionTest to verify snappy support is enabled and the libs can be loaded ON ALL NODES of your cluster:

    $ hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://host/path/to/hbase snappy

  3. Create a column family with snappy compression and verify it in the hbase shell:

    $ hbase> create 't1', { NAME => 'cf1', COMPRESSION => 'SNAPPY' }
    hbase> describe 't1'

    In the output of the "describe" command, you need to ensure it lists "COMPRESSION => 'SNAPPY'"

C.5.1.  Installation

You will find the snappy library file under the .libs directory from your Snappy build (For example /home/hbase/snappy-1.0.5/.libs/). The file is called libsnappy.so.1.x.x where 1.x.x is the version of the snappy code you are building. You can either copy this file into your hbase directory under libsnappy.so name, or simply create a symbolic link to it.

The second file you need is the hadoop native library. You will find this file in your hadoop installation directory under lib/native/Linux-amd64-64/ or lib/native/Linux-i386-32/. The file you are looking for is libhadoop.so.1.x.x. Again, you can simply copy this file or link to it, under the name libhadoop.so.

At the end of the installation, you should have both libsnappy.so and libhadoop.so links or files present into lib/native/Linux-amd64-64 or into lib/native/Linux-i386-32

To point hbase at snappy support, in hbase-env.sh set

export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64

In /pathtoyourhadoop/lib/native/Linux-amd64-64 you should have something like:

        libsnappy.a
        libsnappy.so
        libsnappy.so.1
        libsnappy.so.1.1.2
    



[32] See Alejandro's note up on the list on difference between Snappy in Hadoop and Snappy in HBase

comments powered by Disqus