org.apache.tez.mapreduce.grouper

Class TezSplitGrouper

java.lang.Object

org.apache.tez.mapreduce.grouper.TezSplitGrouper

Direct Known Subclasses:

TezMapredSplitsGrouper, TezMapReduceSplitsGrouper

public abstract class TezSplitGrouper
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TezSplitGrouper.TezMRSplitsGrouperConfigBuilder

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEZ_GROUPING_NODE_LOCAL_ONLY Generate node local splits only.
static boolean	TEZ_GROUPING_NODE_LOCAL_ONLY_DEFAULT
static String	TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION This factor is used to decrease the per group desired (length and count) limits for groups created by combining splits within a rack.
static float	TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT
static String	TEZ_GROUPING_REPEATABLE Repeated invocations of grouping on the same splits with the same parameters will produce the same groups.
static boolean	TEZ_GROUPING_REPEATABLE_DEFAULT
static String	TEZ_GROUPING_SPLIT_BY_COUNT Limit the number of splits in a group by the number of splits in the group
static boolean	TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT
static String	TEZ_GROUPING_SPLIT_BY_LENGTH Limit the number of splits in a group by the total length of the splits in the group
static boolean	TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT
static String	TEZ_GROUPING_SPLIT_COUNT Specify the number of splits desired to be created
static String	TEZ_GROUPING_SPLIT_MAX_SIZE Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large splits.
static long	TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT
static String	TEZ_GROUPING_SPLIT_MIN_SIZE Lower bound on the size (in bytes) of a grouped split, to avoid generating too many small splits.
static long	TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT
static String	TEZ_GROUPING_SPLIT_WAVES The multiplier for available queue capacity when determining number of tasks for a Vertex.
static float	TEZ_GROUPING_SPLIT_WAVES_DEFAULT

Constructor Summary

Constructors

Constructor and Description
TezSplitGrouper()

Method Summary

Modifier and Type	Method and Description
List<GroupedSplitContainer>	getGroupedSplits(org.apache.hadoop.conf.Configuration conf, List<SplitContainer> originalSplits, int desiredNumSplits, String wrappedInputFormatName, SplitSizeEstimatorWrapper estimator, SplitLocationProviderWrapper locationProvider)
static TezSplitGrouper.TezMRSplitsGrouperConfigBuilder	newConfigBuilder(org.apache.hadoop.conf.Configuration conf) Builder that can be used to configure grouping in Tez

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEZ_GROUPING_SPLIT_COUNT

public static final String TEZ_GROUPING_SPLIT_COUNT

Specify the number of splits desired to be created

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_LENGTH

public static final String TEZ_GROUPING_SPLIT_BY_LENGTH

Limit the number of splits in a group by the total length of the splits in the group

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT

public static final boolean TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_COUNT

public static final String TEZ_GROUPING_SPLIT_BY_COUNT

Limit the number of splits in a group by the number of splits in the group

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT

public static final boolean TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_WAVES

public static final String TEZ_GROUPING_SPLIT_WAVES

The multiplier for available queue capacity when determining number of tasks for a Vertex. 1.7 with 100% queue available implies generating a number of tasks roughly equal to 170% of the available containers on the queue. This enables multiple waves of mappers where the final wave is slightly smaller than the remaining waves. The gap helps overlap the final wave with any slower tasks from previous waves and tries to hide the delays from the slower tasks. Good values for this are 1.7, 2.7, 3.7 etc. Increase the number of waves to make the tasks smaller or shorter.

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_WAVES_DEFAULT

public static final float TEZ_GROUPING_SPLIT_WAVES_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MAX_SIZE

public static final String TEZ_GROUPING_SPLIT_MAX_SIZE

Upper bound on the size (in bytes) of a grouped split, to avoid generating excessively large splits.

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT

public static final long TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MIN_SIZE

public static final String TEZ_GROUPING_SPLIT_MIN_SIZE

Lower bound on the size (in bytes) of a grouped split, to avoid generating too many small splits.

See Also:

Constant Field Values

TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT

public static final long TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION

public static final String TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION

This factor is used to decrease the per group desired (length and count) limits for groups created by combining splits within a rack. Since reading this split involves reading data intra rack, the group is made smaller to cover up for the increased latencies in doing intra rack reads. The value should be a fraction <= 1.

See Also:

Constant Field Values

TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT

public static final float TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_REPEATABLE

public static final String TEZ_GROUPING_REPEATABLE

Repeated invocations of grouping on the same splits with the same parameters will produce the same groups. This may help in cache reuse but may cause hot-spotting on nodes when there are a large number of jobs reading the same hot data. True by default.

See Also:

Constant Field Values

TEZ_GROUPING_REPEATABLE_DEFAULT

public static final boolean TEZ_GROUPING_REPEATABLE_DEFAULT

See Also:

Constant Field Values

TEZ_GROUPING_NODE_LOCAL_ONLY

public static final String TEZ_GROUPING_NODE_LOCAL_ONLY

Generate node local splits only. This prevents fallback to rack locality etc, and overrides the target size for small splits.

See Also:

Constant Field Values

TEZ_GROUPING_NODE_LOCAL_ONLY_DEFAULT

public static final boolean TEZ_GROUPING_NODE_LOCAL_ONLY_DEFAULT

See Also:

Constant Field Values

Constructor Detail

TezSplitGrouper

public TezSplitGrouper()

Method Detail

getGroupedSplits

public List<GroupedSplitContainer> getGroupedSplits(org.apache.hadoop.conf.Configuration conf,
                                                    List<SplitContainer> originalSplits,
                                                    int desiredNumSplits,
                                                    String wrappedInputFormatName,
                                                    SplitSizeEstimatorWrapper estimator,
                                                    SplitLocationProviderWrapper locationProvider)
                                             throws IOException,
                                                    InterruptedException

Throws:

IOException

InterruptedException

newConfigBuilder

public static TezSplitGrouper.TezMRSplitsGrouperConfigBuilder newConfigBuilder(org.apache.hadoop.conf.Configuration conf)

Builder that can be used to configure grouping in Tez

Parameters:

conf - Configuration This will be modified in place. If configuration values may be changed at runtime via a config file then pass in a Configuration that is initialized from a config file. The parameters that are not overridden in code will be derived from the Configuration object.

Returns:

TezSplitGrouper.TezMRSplitsGrouperConfigBuilder