org.apache.hive.hcatalog.mapreduce

Class HCatBaseInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.WritableComparable,HCatRecord>

org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat

Direct Known Subclasses:

HCatInputFormat

public abstract class HCatBaseInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.WritableComparable,HCatRecord>

Constructor Summary

Constructors

Constructor and Description
HCatBaseInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.WritableComparable,HCatRecord>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) Create the RecordReader for the given InputSplit.
protected static org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable>	getMapRedInputFormat(org.apache.hadoop.mapred.JobConf job, Class inputFormatClass)
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext jobContext) Logically split the set of input files for the job.
static HCatSchema	getTableSchema(org.apache.hadoop.conf.Configuration conf) Gets the HCatTable schema for the table specified in the HCatInputFormat.setInput call on the specified job context.
static void	setOutputSchema(org.apache.hadoop.mapreduce.Job job, HCatSchema hcatSchema) Set the schema for the HCatRecord data returned by HCatInputFormat.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HCatBaseInputFormat

public HCatBaseInputFormat()

Method Detail

setOutputSchema

public static void setOutputSchema(org.apache.hadoop.mapreduce.Job job,
                   HCatSchema hcatSchema)
                            throws IOException

Set the schema for the HCatRecord data returned by HCatInputFormat.

Parameters:

job - the job object

hcatSchema - the schema to use as the consolidated schema

Throws:

IOException

getMapRedInputFormat

protected static org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable> getMapRedInputFormat(org.apache.hadoop.mapred.JobConf job,
                                                                                                                               Class inputFormatClass)
                                                                                                                                           throws IOException

Throws:

IOException

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext jobContext)
                                                       throws IOException,
                                                              InterruptedException

Logically split the set of input files for the job. Returns the underlying InputFormat's splits

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.WritableComparable,HCatRecord>

Parameters:

jobContext - the job context object

Returns:

the splits, an HCatInputSplit wrapper over the storage handler InputSplits

Throws:

IOException - or InterruptedException

InterruptedException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.WritableComparable,HCatRecord> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                              org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
                                                                                                                throws IOException,
                                                                                                                       InterruptedException

Create the RecordReader for the given InputSplit. Returns the underlying RecordReader if the required operations are supported and schema matches with HCatTable schema. Returns an HCatRecordReader if operations need to be implemented in HCat.

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.WritableComparable,HCatRecord>

Parameters:

split - the split

taskContext - the task attempt context

Returns:

the record reader instance, either an HCatRecordReader(later) or the underlying storage handler's RecordReader

Throws:

IOException - or InterruptedException

InterruptedException

getTableSchema

public static HCatSchema getTableSchema(org.apache.hadoop.conf.Configuration conf)
                                 throws IOException

Gets the HCatTable schema for the table specified in the HCatInputFormat.setInput call on the specified job context. This information is available only after HCatInputFormat.setInput has been called for a JobContext.

Parameters:

conf - the Configuration object

Returns:

the table schema

Throws:

IOException - if HCatInputFormat.setInput has not been called for the current context