org.apache.tez.mapreduce.input

Class MRInput

java.lang.Object

org.apache.tez.runtime.api.AbstractLogicalInput

org.apache.tez.mapreduce.input.base.MRInputBase

org.apache.tez.mapreduce.input.MRInput

All Implemented Interfaces:

org.apache.tez.runtime.api.Input, org.apache.tez.runtime.api.InputFrameworkInterface, org.apache.tez.runtime.api.LogicalInput, org.apache.tez.runtime.api.LogicalInputFrameworkInterface

@InterfaceAudience.Public
public class MRInput
extends org.apache.tez.mapreduce.input.base.MRInputBase

MRInput is an Input which provides key/values pairs for the consumer. It is compatible with all standard Apache Hadoop MapReduce InputFormat implementations. This class is not meant to be extended by external projects.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.tez.mapreduce.lib.MRReader	mrReader
protected org.apache.hadoop.mapreduce.split.JobSplit.TaskSplitIndex	splitMetaInfo

Fields inherited from class org.apache.tez.mapreduce.input.base.MRInputBase

inputRecordCounter, jobConf

Constructor Summary

Constructors

Constructor and Description
MRInput(org.apache.tez.runtime.api.InputContext inputContext, int numPhysicalInputs)

Method Summary

Modifier and Type	Method and Description
List<org.apache.tez.runtime.api.Event>	close()
static org.apache.tez.mapreduce.input.MRInput.MRInputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> inputFormat) Create an MRInput.MRInputConfigBuilder
static org.apache.tez.mapreduce.input.MRInput.MRInputConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf, Class<?> inputFormat, String inputPaths) Create an MRInput.MRInputConfigBuilder for FileInputFormat or FileInputFormat format based InputFormats.
org.apache.hadoop.conf.Configuration	getConfigUpdates() MRInput sets some additional parameters like split location when using the new API.
float	getProgress()
org.apache.tez.runtime.library.api.KeyValueReader	getReader() Returns a KeyValueReader that can be used to read Map Reduce compatible key value data.
void	handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)
List<org.apache.tez.runtime.api.Event>	initialize()
void	start()

Methods inherited from class org.apache.tez.runtime.api.AbstractLogicalInput

getContext, getNumPhysicalInputs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mrReader

protected org.apache.tez.mapreduce.lib.MRReader mrReader

splitMetaInfo

protected org.apache.hadoop.mapreduce.split.JobSplit.TaskSplitIndex splitMetaInfo

Constructor Detail

MRInput

public MRInput(org.apache.tez.runtime.api.InputContext inputContext,
               int numPhysicalInputs)

Method Detail

createConfigBuilder

public static org.apache.tez.mapreduce.input.MRInput.MRInputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                              @Nullable
                                                                                              Class<?> inputFormat)

Create an MRInput.MRInputConfigBuilder

The preferred usage model is to provide all of the parameters, and use methods to configure the Input.

For legacy applications, which may already have a fully configured Configuration instance, the inputFormat can be specified as null

Parameters:

conf - Configuration for the MRInput. This configuration instance will be modified in place

inputFormat - InputFormat derived class. This can be null. If the InputFormat specified is null, the provided configuration should be complete.

Returns:

MRInput.MRInputConfigBuilder

createConfigBuilder

public static org.apache.tez.mapreduce.input.MRInput.MRInputConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                              @Nullable
                                                                                              Class<?> inputFormat,
                                                                                              @Nullable
                                                                                              String inputPaths)

Create an MRInput.MRInputConfigBuilder for FileInputFormat or FileInputFormat format based InputFormats.

The preferred usage model is to provide all of the parameters, and use methods to configure the Input.

For legacy applications, which may already have a fully configured Configuration instance, the inputFormat and inputPath can be specified as null

Parameters:

conf - Configuration for the MRInput. This configuration instance will be modified in place

inputFormat - InputFormat derived class. This can be null. If the InputFormat specified is null, the provided configuration should be complete.

inputPaths - Comma separated input paths

Returns:

MRInput.MRInputConfigBuilder

initialize

public List<org.apache.tez.runtime.api.Event> initialize()
                                                  throws IOException

Specified by:

initialize in interface org.apache.tez.runtime.api.InputFrameworkInterface

Overrides:

initialize in class org.apache.tez.mapreduce.input.base.MRInputBase

Throws:

IOException

start

public void start()

getReader

public org.apache.tez.runtime.library.api.KeyValueReader getReader()
                                                            throws IOException

Returns a KeyValueReader that can be used to read Map Reduce compatible key value data. An exception will be thrown if next() is invoked after false, either from the framework or from the underlying InputFormat

Specified by:

getReader in interface org.apache.tez.runtime.api.Input

Overrides:

getReader in class org.apache.tez.mapreduce.input.base.MRInputBase

Throws:

IOException

handleEvents

public void handleEvents(List<org.apache.tez.runtime.api.Event> inputEvents)
                  throws Exception

Throws:

Exception

close

public List<org.apache.tez.runtime.api.Event> close()
                                             throws IOException

Throws:

IOException

getConfigUpdates

public org.apache.hadoop.conf.Configuration getConfigUpdates()

MRInput sets some additional parameters like split location when using the new API. This methods returns the list of additional updates, and should be used by Processors using the old MapReduce API with MRInput.

Returns:

the additional fields set by MRInput

getProgress

public float getProgress()
                  throws org.apache.tez.runtime.api.ProgressFailedException,
                         InterruptedException

Overrides:

getProgress in class org.apache.tez.runtime.api.AbstractLogicalInput

Throws:

org.apache.tez.runtime.api.ProgressFailedException

InterruptedException