org.apache.hadoop.hive.hbase

Class HBaseSerDe

java.lang.Object

org.apache.hadoop.hive.serde2.AbstractSerDe

org.apache.hadoop.hive.hbase.HBaseSerDe

All Implemented Interfaces:

Deserializer, Serializer

public class HBaseSerDe
extends AbstractSerDe

HBaseSerDe can be used to serialize object into an HBase table and deserialize objects from an HBase table.

Field Summary

Fields

Modifier and Type	Field and Description
static String	HBASE_AUTOGENERATE_STRUCT
static String	HBASE_COLUMNS_MAPPING
static String	HBASE_COLUMNS_PREFIX_HIDE Defines if the prefix column from hbase should be hidden.
static String	HBASE_COLUMNS_REGEX_MATCHING Determines whether a regex matching should be done on the columns or not.
static String	HBASE_COMPOSITE_KEY_CLASS
static String	HBASE_COMPOSITE_KEY_FACTORY
static String	HBASE_COMPOSITE_KEY_TYPES
static String	HBASE_KEY_COL
static String	HBASE_PUT_TIMESTAMP
static String	HBASE_SCAN_BATCH
static String	HBASE_SCAN_CACHE
static String	HBASE_SCAN_CACHEBLOCKS
static String	HBASE_STRUCT_SERIALIZER_CLASS
static String	HBASE_TABLE_DEFAULT_STORAGE_TYPE
static String	HBASE_TABLE_NAME
static String	HBASE_TIMESTAMP_COL
static org.slf4j.Logger	LOG
static String	SERIALIZATION_TYPE Defines the type for a column.

Fields inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

configErrors

Constructor Summary

Constructors

Constructor and Description
HBaseSerDe()

Method Summary

Modifier and Type	Method and Description
static void	configureJobConf(TableDesc tableDesc, org.apache.hadoop.mapred.JobConf jobConf)
Object	deserialize(org.apache.hadoop.io.Writable result) Deserialize a row from the HBase Result writable to a LazyObject
HBaseSerDeParameters	getHBaseSerdeParam()
HBaseKeyFactory	getKeyFactory()
ObjectInspector	getObjectInspector() Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).
LazySerDeParameters	getSerdeParams()
SerDeStats	getSerDeStats() Returns statistics collected when serializing
Class<? extends org.apache.hadoop.io.Writable>	getSerializedClass() Returns the Writable class that would be returned by the serialize method.
void	initialize(org.apache.hadoop.conf.Configuration conf, Properties tbl) Initialize the SerDe given parameters.
static ColumnMappings	parseColumnsMapping(String columnsMappingSpec)
static ColumnMappings	parseColumnsMapping(String columnsMappingSpec, boolean doColumnRegexMatching)
static ColumnMappings	parseColumnsMapping(String columnsMappingSpec, boolean doColumnRegexMatching, boolean hideColumnPrefix) Parses the HBase columns mapping specifier to identify the column families, qualifiers and also caches the byte arrays corresponding to them.
org.apache.hadoop.io.Writable	serialize(Object obj, ObjectInspector objInspector) Serialize an object by navigating inside the Object with the ObjectInspector.
String	toString()

Methods inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

getConfigurationErrors, initialize, shouldStoreFieldsInMetastore

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

HBASE_COLUMNS_MAPPING

public static final String HBASE_COLUMNS_MAPPING

See Also:

Constant Field Values

HBASE_TABLE_NAME

public static final String HBASE_TABLE_NAME

See Also:

Constant Field Values

HBASE_TABLE_DEFAULT_STORAGE_TYPE

public static final String HBASE_TABLE_DEFAULT_STORAGE_TYPE

See Also:

Constant Field Values

HBASE_KEY_COL

public static final String HBASE_KEY_COL

See Also:

Constant Field Values

HBASE_TIMESTAMP_COL

public static final String HBASE_TIMESTAMP_COL

See Also:

Constant Field Values

HBASE_PUT_TIMESTAMP

public static final String HBASE_PUT_TIMESTAMP

See Also:

Constant Field Values

HBASE_COMPOSITE_KEY_CLASS

public static final String HBASE_COMPOSITE_KEY_CLASS

See Also:

Constant Field Values

HBASE_COMPOSITE_KEY_TYPES

public static final String HBASE_COMPOSITE_KEY_TYPES

See Also:

Constant Field Values

HBASE_COMPOSITE_KEY_FACTORY

public static final String HBASE_COMPOSITE_KEY_FACTORY

See Also:

Constant Field Values

HBASE_STRUCT_SERIALIZER_CLASS

public static final String HBASE_STRUCT_SERIALIZER_CLASS

See Also:

Constant Field Values

HBASE_SCAN_CACHE

public static final String HBASE_SCAN_CACHE

See Also:

Constant Field Values

HBASE_SCAN_CACHEBLOCKS

public static final String HBASE_SCAN_CACHEBLOCKS

See Also:

Constant Field Values

HBASE_SCAN_BATCH

public static final String HBASE_SCAN_BATCH

See Also:

Constant Field Values

HBASE_AUTOGENERATE_STRUCT

public static final String HBASE_AUTOGENERATE_STRUCT

See Also:

Constant Field Values

HBASE_COLUMNS_REGEX_MATCHING

public static final String HBASE_COLUMNS_REGEX_MATCHING

Determines whether a regex matching should be done on the columns or not. Defaults to true. WARNING: Note that currently this only supports the suffix wildcard .*

See Also:

Constant Field Values

SERIALIZATION_TYPE

public static final String SERIALIZATION_TYPE

Defines the type for a column.

See Also:

Constant Field Values

HBASE_COLUMNS_PREFIX_HIDE

public static final String HBASE_COLUMNS_PREFIX_HIDE

Defines if the prefix column from hbase should be hidden. It works only when @HBASE_COLUMNS_REGEX_MATCHING is true. Default value of this parameter is false

See Also:

Constant Field Values

Constructor Detail

HBaseSerDe

public HBaseSerDe()
           throws SerDeException

Throws:

SerDeException

Method Detail

toString

public String toString()

Overrides:

toString in class Object

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf,
                       Properties tbl)
                throws SerDeException

Initialize the SerDe given parameters.

Specified by:

initialize in interface Deserializer

Specified by:

initialize in interface Serializer

Specified by:

initialize in class AbstractSerDe

Parameters:

conf - System properties. Can be null in compile time

tbl - table properties

Throws:

SerDeException

See Also:

AbstractSerDe.initialize(Configuration, Properties)

parseColumnsMapping

public static ColumnMappings parseColumnsMapping(String columnsMappingSpec)
                                          throws SerDeException

Throws:

SerDeException

parseColumnsMapping

public static ColumnMappings parseColumnsMapping(String columnsMappingSpec,
                                                 boolean doColumnRegexMatching)
                                          throws SerDeException

Throws:

SerDeException

parseColumnsMapping

public static ColumnMappings parseColumnsMapping(String columnsMappingSpec,
                                                 boolean doColumnRegexMatching,
                                                 boolean hideColumnPrefix)
                                          throws SerDeException

Parses the HBase columns mapping specifier to identify the column families, qualifiers and also caches the byte arrays corresponding to them. One of the Hive table columns maps to the HBase row key, by default the first column.

Parameters:

columnsMappingSpec - string hbase.columns.mapping specified when creating table

doColumnRegexMatching - whether to do a regex matching on the columns or not

hideColumnPrefix - whether to hide a prefix of column mapping in key name in a map (works only if @doColumnRegexMatching is true)

Returns:

List which contains the column mapping information by position

Throws:

SerDeException

getSerdeParams

public LazySerDeParameters getSerdeParams()

getHBaseSerdeParam

public HBaseSerDeParameters getHBaseSerdeParam()

deserialize

public Object deserialize(org.apache.hadoop.io.Writable result)
                   throws SerDeException

Deserialize a row from the HBase Result writable to a LazyObject

Specified by:

deserialize in interface Deserializer

Specified by:

deserialize in class AbstractSerDe

Parameters:

result - the HBase Result Writable containing the row

Returns:

the deserialized object

Throws:

SerDeException

See Also:

AbstractSerDe.deserialize(Writable)

getObjectInspector

public ObjectInspector getObjectInspector()
                                   throws SerDeException

Description copied from class: AbstractSerDe

Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).

Specified by:

getObjectInspector in interface Deserializer

Specified by:

getObjectInspector in class AbstractSerDe

Throws:

SerDeException

getSerializedClass

public Class<? extends org.apache.hadoop.io.Writable> getSerializedClass()

Description copied from class: AbstractSerDe

Returns the Writable class that would be returned by the serialize method. This is used to initialize SequenceFile header.

Specified by:

getSerializedClass in interface Serializer

Specified by:

getSerializedClass in class AbstractSerDe

serialize

public org.apache.hadoop.io.Writable serialize(Object obj,
                                               ObjectInspector objInspector)
                                        throws SerDeException

Description copied from class: AbstractSerDe

Serialize an object by navigating inside the Object with the ObjectInspector. In most cases, the return value of this function will be constant since the function will reuse the Writable object. If the client wants to keep a copy of the Writable, the client needs to clone the returned value.

Specified by:

serialize in interface Serializer

Specified by:

serialize in class AbstractSerDe

Throws:

SerDeException

getSerDeStats

public SerDeStats getSerDeStats()

Description copied from class: AbstractSerDe

Returns statistics collected when serializing

Specified by:

getSerDeStats in interface Deserializer

Specified by:

getSerDeStats in interface Serializer

Specified by:

getSerDeStats in class AbstractSerDe

getKeyFactory

public HBaseKeyFactory getKeyFactory()

configureJobConf

public static void configureJobConf(TableDesc tableDesc,
                                    org.apache.hadoop.mapred.JobConf jobConf)
                             throws Exception

Throws:

Exception