org.apache.hadoop.hive.hbase

Class HBaseStorageHandler

java.lang.Object

org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler

org.apache.hadoop.hive.hbase.HBaseStorageHandler

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, HiveMetaHook, HiveStorageHandler, HiveStoragePredicateHandler

public class HBaseStorageHandler
extends DefaultStorageHandler
implements HiveMetaHook, HiveStoragePredicateHandler

HBaseStorageHandler provides a HiveStorageHandler implementation for HBase.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.ql.metadata.HiveStoragePredicateHandler

HiveStoragePredicateHandler.DecomposedPredicate

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_PREFIX

Constructor Summary

Constructors

Constructor and Description
HBaseStorageHandler()

Method Summary

Methods

Modifier and Type	Method and Description
void	commitCreateTable(Table table) Called after successfully adding a new table definition to the metastore during CREATE TABLE.
void	commitDropTable(Table tbl, boolean deleteData) Called after successfully removing a table definition from the metastore during DROP TABLE.
void	configureInputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureJobConf(TableDesc tableDesc, org.apache.hadoop.mapred.JobConf jobConf) Called just before submitting MapReduce job.
void	configureOutputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureTableJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) Deprecated use configureInputJobProperties/configureOutputJobProperties methods instead.
HiveStoragePredicateHandler.DecomposedPredicate	decomposePredicate(org.apache.hadoop.mapred.JobConf jobConf, Deserializer deserializer, ExprNodeDesc predicate) Gives the storage handler a chance to decompose a predicate.
static HiveStoragePredicateHandler.DecomposedPredicate	decomposePredicate(org.apache.hadoop.mapred.JobConf jobConf, HBaseSerDe hBaseSerDe, ExprNodeDesc predicate)
org.apache.hadoop.conf.Configuration	getConf()
Class<? extends org.apache.hadoop.mapred.InputFormat>	getInputFormatClass()
org.apache.hadoop.conf.Configuration	getJobConf()
HiveMetaHook	getMetaHook()
Class<? extends org.apache.hadoop.mapred.OutputFormat>	getOutputFormatClass()
Class<? extends SerDe>	getSerDeClass()
static boolean	isHBaseGenerateHFiles(org.apache.hadoop.conf.Configuration conf) Return true when HBaseStorageHandler should generate hfiles instead of operate against the online table.
void	preCreateTable(Table tbl) Called before a new table definition is added to the metastore during CREATE TABLE.
void	preDropTable(Table table) Called before a table definition is removed from the metastore during DROP TABLE.
void	rollbackCreateTable(Table table) Called after failure adding a new table definition to the metastore during CREATE TABLE.
void	rollbackDropTable(Table table) Called after failure removing a table definition from the metastore during DROP TABLE.
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler

getAuthorizationProvider, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_PREFIX

public static final String DEFAULT_PREFIX

See Also:

Constant Field Values

Constructor Detail

HBaseStorageHandler

public HBaseStorageHandler()

Method Detail

preDropTable

public void preDropTable(Table table)
                  throws MetaException

Description copied from interface: HiveMetaHook

Called before a table definition is removed from the metastore during DROP TABLE.

Specified by:

preDropTable in interface HiveMetaHook

Parameters:

table - table definition

Throws:

MetaException

rollbackDropTable

public void rollbackDropTable(Table table)
                       throws MetaException

Description copied from interface: HiveMetaHook

Called after failure removing a table definition from the metastore during DROP TABLE.

Specified by:

rollbackDropTable in interface HiveMetaHook

Parameters:

table - table definition

Throws:

MetaException

commitDropTable

public void commitDropTable(Table tbl,
                   boolean deleteData)
                     throws MetaException

Description copied from interface: HiveMetaHook

Called after successfully removing a table definition from the metastore during DROP TABLE.

Specified by:

commitDropTable in interface HiveMetaHook

Parameters:

tbl - table definition

deleteData - whether to delete data as well; this should typically be ignored in the case of an external table

Throws:

MetaException

preCreateTable

public void preCreateTable(Table tbl)
                    throws MetaException

Description copied from interface: HiveMetaHook

Called before a new table definition is added to the metastore during CREATE TABLE.

Specified by:

preCreateTable in interface HiveMetaHook

Parameters:

tbl - new table definition

Throws:

MetaException

rollbackCreateTable

public void rollbackCreateTable(Table table)
                         throws MetaException

Description copied from interface: HiveMetaHook

Called after failure adding a new table definition to the metastore during CREATE TABLE.

Specified by:

rollbackCreateTable in interface HiveMetaHook

Parameters:

table - new table definition

Throws:

MetaException

commitCreateTable

public void commitCreateTable(Table table)
                       throws MetaException

Description copied from interface: HiveMetaHook

Called after successfully adding a new table definition to the metastore during CREATE TABLE.

Specified by:

commitCreateTable in interface HiveMetaHook

Parameters:

table - new table definition

Throws:

MetaException

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

Overrides:

getConf in class DefaultStorageHandler

getJobConf

public org.apache.hadoop.conf.Configuration getJobConf()

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class DefaultStorageHandler

getInputFormatClass

public Class<? extends org.apache.hadoop.mapred.InputFormat> getInputFormatClass()

Specified by:

getInputFormatClass in interface HiveStorageHandler

Overrides:

getInputFormatClass in class DefaultStorageHandler

Returns:

Class providing an implementation of InputFormat

getOutputFormatClass

public Class<? extends org.apache.hadoop.mapred.OutputFormat> getOutputFormatClass()

Specified by:

getOutputFormatClass in interface HiveStorageHandler

Overrides:

getOutputFormatClass in class DefaultStorageHandler

Returns:

Class providing an implementation of OutputFormat

getSerDeClass

public Class<? extends SerDe> getSerDeClass()

Specified by:

getSerDeClass in interface HiveStorageHandler

Overrides:

getSerDeClass in class DefaultStorageHandler

Returns:

Class providing an implementation of SerDe

getMetaHook

public HiveMetaHook getMetaHook()

Specified by:

getMetaHook in interface HiveStorageHandler

Overrides:

getMetaHook in class DefaultStorageHandler

Returns:

metadata hook implementation, or null if this storage handler does not need any metadata notifications

configureInputJobProperties

public void configureInputJobProperties(TableDesc tableDesc,
                               Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for input.

Specified by:

configureInputJobProperties in interface HiveStorageHandler

Overrides:

configureInputJobProperties in class DefaultStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureOutputJobProperties

public void configureOutputJobProperties(TableDesc tableDesc,
                                Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for output.

Specified by:

configureOutputJobProperties in interface HiveStorageHandler

Overrides:

configureOutputJobProperties in class DefaultStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureTableJobProperties

public void configureTableJobProperties(TableDesc tableDesc,
                               Map<String,String> jobProperties)

Description copied from interface: HiveStorageHandler

Deprecated use configureInputJobProperties/configureOutputJobProperties methods instead. Configures properties for a job based on the definition of the source or target table it accesses.

Specified by:

configureTableJobProperties in interface HiveStorageHandler

Overrides:

configureTableJobProperties in class DefaultStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

isHBaseGenerateHFiles

public static boolean isHBaseGenerateHFiles(org.apache.hadoop.conf.Configuration conf)

Return true when HBaseStorageHandler should generate hfiles instead of operate against the online table. This mode is implicitly applied when "hive.hbase.generatehfiles" is true.

configureJobConf

public void configureJobConf(TableDesc tableDesc,
                    org.apache.hadoop.mapred.JobConf jobConf)

Description copied from interface: HiveStorageHandler

Called just before submitting MapReduce job.

Specified by:

configureJobConf in interface HiveStorageHandler

Overrides:

configureJobConf in class DefaultStorageHandler

Parameters:

tableDesc - descriptor for the table being accessed

decomposePredicate

public HiveStoragePredicateHandler.DecomposedPredicate decomposePredicate(org.apache.hadoop.mapred.JobConf jobConf,
                                                                 Deserializer deserializer,
                                                                 ExprNodeDesc predicate)

Description copied from interface: HiveStoragePredicateHandler

Gives the storage handler a chance to decompose a predicate. The storage handler should analyze the predicate and return the portion of it which cannot be evaluated during table access. For example, if the original predicate is x = 2 AND upper(y)='YUM', the storage handler might be able to handle x = 2 but leave the "residual" upper(y)='YUM' for Hive to deal with. The breakdown need not be non-overlapping; for example, given the predicate x LIKE 'a%b', the storage handler might be able to evaluate the prefix search x LIKE 'a%', leaving x LIKE '%b' as the residual.

Specified by:

decomposePredicate in interface HiveStoragePredicateHandler

Parameters:

jobConf - contains a job configuration matching the one that will later be passed to getRecordReader and getSplits

deserializer - deserializer which will be used when fetching rows

predicate - predicate to be decomposed

Returns:

decomposed form of predicate, or null if no pushdown is possible at all

decomposePredicate

public static HiveStoragePredicateHandler.DecomposedPredicate decomposePredicate(org.apache.hadoop.mapred.JobConf jobConf,
                                                                 HBaseSerDe hBaseSerDe,
                                                                 ExprNodeDesc predicate)