org.apache.hadoop.mapreduce.split

Class TezMapReduceSplitsGrouper

java.lang.Object

org.apache.tez.mapreduce.grouper.TezSplitGrouper

org.apache.hadoop.mapreduce.split.TezMapReduceSplitsGrouper

@InterfaceAudience.Public
 @InterfaceStability.Evolving
public class TezMapReduceSplitsGrouper
extends TezSplitGrouper

Helper that provides a grouping of input splits based on multiple parameters. It creates TezGroupedSplit to wrap the each group of real InputSplits

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder Deprecated. See TezSplitGrouper.TezMRSplitsGrouperConfigBuilder

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION Deprecated. See equivalent in TezSplitGrouper
static float	TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT Deprecated. See equivalent in TezSplitGrouper
static String	TEZ_GROUPING_REPEATABLE Deprecated. See equivalent in TezSplitGrouper
static boolean	TEZ_GROUPING_REPEATABLE_DEFAULT Deprecated. See equivalent in TezSplitGrouper
static String	TEZ_GROUPING_SPLIT_BY_COUNT Deprecated. See equivalent in TezSplitGrouper
static boolean	TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT Deprecated. See equivalent in TezSplitGrouper
static String	TEZ_GROUPING_SPLIT_BY_LENGTH Deprecated. See equivalent in TezSplitGrouper
static boolean	TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT Deprecated. See equivalent in TezSplitGrouper
static String	TEZ_GROUPING_SPLIT_COUNT Deprecated. See equivalent in TezSplitGrouper
static String	TEZ_GROUPING_SPLIT_MAX_SIZE Deprecated. See equivalent in TezSplitGrouper
static long	TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT Deprecated. See equivalent in TezSplitGrouper
static String	TEZ_GROUPING_SPLIT_MIN_SIZE Deprecated. See equivalent in TezSplitGrouper
static long	TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT Deprecated. See equivalent in TezSplitGrouper
static String	TEZ_GROUPING_SPLIT_WAVES Deprecated. See equivalent in TezSplitGrouper
static float	TEZ_GROUPING_SPLIT_WAVES_DEFAULT Deprecated. See equivalent in TezSplitGrouper

Fields inherited from class org.apache.tez.mapreduce.grouper.TezSplitGrouper

TEZ_GROUPING_NODE_LOCAL_ONLY, TEZ_GROUPING_NODE_LOCAL_ONLY_DEFAULT

Constructor Summary

Constructors

Constructor and Description
TezMapReduceSplitsGrouper()

Method Summary

Modifier and Type	Method and Description
static TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder	createConfigBuilder(org.apache.hadoop.conf.Configuration conf) Deprecated. See TezSplitGrouper.newConfigBuilder(Configuration)
List<org.apache.hadoop.mapreduce.InputSplit>	getGroupedSplits(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.mapreduce.InputSplit> originalSplits, int desiredNumSplits, String wrappedInputFormatName)
List<org.apache.hadoop.mapreduce.InputSplit>	getGroupedSplits(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.mapreduce.InputSplit> originalSplits, int desiredNumSplits, String wrappedInputFormatName, SplitSizeEstimator estimator)
List<org.apache.hadoop.mapreduce.InputSplit>	getGroupedSplits(org.apache.hadoop.conf.Configuration conf, List<org.apache.hadoop.mapreduce.InputSplit> originalSplits, int desiredNumSplits, String wrappedInputFormatName, SplitSizeEstimator estimator, SplitLocationProvider locationProvider)

Methods inherited from class org.apache.tez.mapreduce.grouper.TezSplitGrouper

getGroupedSplits, newConfigBuilder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_GROUPING_SPLIT_COUNT

@Deprecated
public static final String TEZ_GROUPING_SPLIT_COUNT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_LENGTH

@Deprecated
public static final String TEZ_GROUPING_SPLIT_BY_LENGTH

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT

@Deprecated
public static final boolean TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_COUNT

@Deprecated
public static final String TEZ_GROUPING_SPLIT_BY_COUNT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT

@Deprecated
public static final boolean TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_WAVES

@Deprecated
public static final String TEZ_GROUPING_SPLIT_WAVES

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_WAVES_DEFAULT

@Deprecated
public static final float TEZ_GROUPING_SPLIT_WAVES_DEFAULT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MAX_SIZE

@Deprecated
public static final String TEZ_GROUPING_SPLIT_MAX_SIZE

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT

@Deprecated
public static final long TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MIN_SIZE

@Deprecated
public static final String TEZ_GROUPING_SPLIT_MIN_SIZE

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT

@Deprecated
public static final long TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION

@Deprecated
public static final String TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT

@Deprecated
public static final float TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_REPEATABLE

@Deprecated
public static final String TEZ_GROUPING_REPEATABLE

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

TEZ_GROUPING_REPEATABLE_DEFAULT

@Deprecated
public static final boolean TEZ_GROUPING_REPEATABLE_DEFAULT

Deprecated. See equivalent in TezSplitGrouper

See Also:

Constant Field Values

Constructor Detail

TezMapReduceSplitsGrouper

public TezMapReduceSplitsGrouper()

Method Detail

getGroupedSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getGroupedSplits(org.apache.hadoop.conf.Configuration conf,
                                                                     List<org.apache.hadoop.mapreduce.InputSplit> originalSplits,
                                                                     int desiredNumSplits,
                                                                     String wrappedInputFormatName)
                                                              throws IOException,
                                                                     InterruptedException

Throws:

IOException

InterruptedException

getGroupedSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getGroupedSplits(org.apache.hadoop.conf.Configuration conf,
                                                                     List<org.apache.hadoop.mapreduce.InputSplit> originalSplits,
                                                                     int desiredNumSplits,
                                                                     String wrappedInputFormatName,
                                                                     SplitSizeEstimator estimator)
                                                              throws IOException,
                                                                     InterruptedException

Throws:

IOException

InterruptedException

getGroupedSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getGroupedSplits(org.apache.hadoop.conf.Configuration conf,
                                                                     List<org.apache.hadoop.mapreduce.InputSplit> originalSplits,
                                                                     int desiredNumSplits,
                                                                     String wrappedInputFormatName,
                                                                     SplitSizeEstimator estimator,
                                                                     SplitLocationProvider locationProvider)
                                                              throws IOException,
                                                                     InterruptedException

Throws:

IOException

InterruptedException

createConfigBuilder

@Deprecated
public static TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder createConfigBuilder(org.apache.hadoop.conf.Configuration conf)

Deprecated. See TezSplitGrouper.newConfigBuilder(Configuration)

Builder that can be used to configure grouping in Tez

Parameters:

conf - Configuration This will be modified in place. If configuration values may be changed at runtime via a config file then pass in a Configuration that is initialized from a config file. The parameters that are not overridden in code will be derived from the Configuration object.

Returns:

TezMapReduceSplitsGrouper.TezMRSplitsGrouperConfigBuilder