org.apache.hadoop.hive.ql.io

Class CombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.hadoop.hive.ql.io.HiveInputFormat<K,V>

org.apache.hadoop.hive.ql.io.CombineHiveInputFormat<K,V>

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<K,V>, org.apache.hadoop.mapred.JobConfigurable

public class CombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
extends HiveInputFormat<K,V>

CombineHiveInputFormat is a parameterized InputFormat which looks at the path name and determine the correct InputFormat for that path name from mapredPlan.pathToPartitionInfo(). It can be used to read files with different input format in the same map-reduce job.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	CombineHiveInputFormat.AvoidSplitCombination This is a marker interface that is used to identify the formats where combine split generation is not applicable
static class	CombineHiveInputFormat.CombineHiveInputSplit CombineHiveInputSplit encapsulates an InputSplit with its corresponding inputFormatClassName.

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

HiveInputFormat.HiveInputSplit

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG

Fields inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

mrwork, pathToPartitionInfo

Constructor Summary

Constructors

Constructor and Description
CombineHiveInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapred.RecordReader	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter) Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit.
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits) Create Hive splits based on CombineFileSplit.

Methods inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

configure, getInputFormatFromCache, getPartitionDescFromPath, init, pushFilters, pushProjectionsAndFilters, pushProjectionsAndFilters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

Constructor Detail

CombineHiveInputFormat

public CombineHiveInputFormat()

Method Detail

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                              int numSplits)
                                                throws IOException

Create Hive splits based on CombineFileSplit.

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Overrides:

getSplits in class HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException

getRecordReader

public org.apache.hadoop.mapred.RecordReader getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                    org.apache.hadoop.mapred.JobConf job,
                                                    org.apache.hadoop.mapred.Reporter reporter)
                                                      throws IOException

Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit.

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Overrides:

getRecordReader in class HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException