org.apache.hadoop.hive.ql.exec.tez

Class SplitGrouper

java.lang.Object

org.apache.hadoop.hive.ql.exec.tez.SplitGrouper

public class SplitGrouper
extends Object

SplitGrouper is used to combine splits based on head room and locality. It also enforces restrictions around schema, file format and bucketing.

Constructor Summary

Constructors

Constructor and Description
SplitGrouper()

Method Summary

Methods

Modifier and Type	Method and Description
List<org.apache.tez.dag.api.TaskLocationHint>	createTaskLocationHints(org.apache.hadoop.mapred.InputSplit[] splits) Create task location hints from a set of input splits
com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit>	generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapred.InputSplit[] splits, float waves, int availableSlots) Generate groups of splits, separated by schema evolution boundaries
com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit>	generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.mapred.InputSplit[] splits, float waves, int availableSlots, String inputName, boolean groupAcrossFiles) Generate groups of splits, separated by schema evolution boundaries
com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit>	group(org.apache.hadoop.conf.Configuration conf, com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> bucketSplitMultimap, int availableSlots, float waves) group splits for each bucket separately - while evenly filling all the available slots with tasks

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SplitGrouper

public SplitGrouper()

Method Detail

group

public com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> group(org.apache.hadoop.conf.Configuration conf,
                                                                                    com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> bucketSplitMultimap,
                                                                                    int availableSlots,
                                                                                    float waves)
                                                                                      throws IOException

group splits for each bucket separately - while evenly filling all the available slots with tasks

Throws:

IOException

createTaskLocationHints

public List<org.apache.tez.dag.api.TaskLocationHint> createTaskLocationHints(org.apache.hadoop.mapred.InputSplit[] splits)
                                                                      throws IOException

Create task location hints from a set of input splits

Parameters:

splits - the actual splits

Returns:

taskLocationHints - 1 per input split specified

Throws:

IOException

generateGroupedSplits

public com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                    org.apache.hadoop.conf.Configuration conf,
                                                                                                    org.apache.hadoop.mapred.InputSplit[] splits,
                                                                                                    float waves,
                                                                                                    int availableSlots)
                                                                                                      throws Exception

Generate groups of splits, separated by schema evolution boundaries

Throws:

Exception

generateGroupedSplits

public com.google.common.collect.Multimap<Integer,org.apache.hadoop.mapred.InputSplit> generateGroupedSplits(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                    org.apache.hadoop.conf.Configuration conf,
                                                                                                    org.apache.hadoop.mapred.InputSplit[] splits,
                                                                                                    float waves,
                                                                                                    int availableSlots,
                                                                                                    String inputName,
                                                                                                    boolean groupAcrossFiles)
                                                                                                      throws Exception

Generate groups of splits, separated by schema evolution boundaries

Throws:

Exception