org.apache.hadoop.hive.ql.exec

Class GroupByOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<GroupByDesc>

org.apache.hadoop.hive.ql.exec.GroupByOperator

All Implemented Interfaces:

Serializable, Cloneable, Node

public class GroupByOperator
extends Operator<GroupByDesc>

GroupBy operator implementation.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
static int	javaHashEntryOverHead
static int	javaObjectOverHead
static int	javaSizePrimitiveType
static int	javaSizeUnknownType
protected long	maxHashTblMemory Max memory usable by the hashtable before it should flush.
protected long	maxMemory Total amount of memory allowed for JVM heap.
protected float	memoryThreshold configure percent of memory threshold usable by QP.
protected int	numEntriesHashTable Current number of entries in the hash table.

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

alias, childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVECOUNTERCREATEDFILES, HIVECOUNTERFATAL, id, inputObjInspectors, isLogDebugEnabled, isLogInfoEnabled, isLogTraceEnabled, LOG, operatorId, out, outputObjInspector, parentOperators, PLOG, reporter, state, statsMap

Constructor Summary

Constructors

Constructor and Description
GroupByOperator()

Method Summary

Methods

Modifier and Type	Method and Description
boolean	acceptLimitPushdown() we can push the limit above GBY (running in Reducer), since that will generate single row for each group.
void	closeOp(boolean abort) We need to forward all the aggregations to children.
void	flush() Forward all aggregations to children.
List<String>	genColLists(HashMap<Operator<? extends OperatorDesc>,OpParseContext> opParseCtx)
String	getName() Implements the getName function for the Node Interface.
static String	getOperatorName()
OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
static javolution.util.FastBitSet	groupingSet2BitSet(int value)
protected Collection<Future<?>>	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
protected GenericUDAFEvaluator.AggregationBuffer[]	newAggregations()
void	process(Object row, int tag) Process the row.
protected void	resetAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs)
protected void	updateAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs, Object row, ObjectInspector rowInspector, boolean hashAggr, boolean newEntryForHashAggr, Object[][] lastInvoke)

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, completeInitializationOp, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, endGroup, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logStats, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setDone, setExecContext, setGroupKeyObject, setId, setInputContext, setInputObjInspectors, setOperatorId, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, startGroup, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

maxMemory

protected transient long maxMemory

Total amount of memory allowed for JVM heap.

maxHashTblMemory

protected transient long maxHashTblMemory

Max memory usable by the hashtable before it should flush.

memoryThreshold

protected transient float memoryThreshold

configure percent of memory threshold usable by QP.

numEntriesHashTable

protected transient int numEntriesHashTable

Current number of entries in the hash table.

javaObjectOverHead

public static final int javaObjectOverHead

See Also:

Constant Field Values

javaHashEntryOverHead

public static final int javaHashEntryOverHead

See Also:

Constant Field Values

javaSizePrimitiveType

public static final int javaSizePrimitiveType

See Also:

Constant Field Values

javaSizeUnknownType

public static final int javaSizeUnknownType

See Also:

Constant Field Values

Constructor Detail

GroupByOperator

public GroupByOperator()

Method Detail

groupingSet2BitSet

public static javolution.util.FastBitSet groupingSet2BitSet(int value)

initializeOp

protected Collection<Future<?>> initializeOp(org.apache.hadoop.conf.Configuration hconf)
                                      throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<GroupByDesc>

Throws:

HiveException

newAggregations

protected GenericUDAFEvaluator.AggregationBuffer[] newAggregations()
                                                            throws HiveException

Throws:

HiveException

resetAggregations

protected void resetAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs)
                          throws HiveException

Throws:

HiveException

updateAggregations

protected void updateAggregations(GenericUDAFEvaluator.AggregationBuffer[] aggs,
                      Object row,
                      ObjectInspector rowInspector,
                      boolean hashAggr,
                      boolean newEntryForHashAggr,
                      Object[][] lastInvoke)
                           throws HiveException

Throws:

HiveException

process

public void process(Object row,
           int tag)
             throws HiveException

Description copied from class: Operator

Process the row.

Specified by:

process in class Operator<GroupByDesc>

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

flush

public void flush()
           throws HiveException

Forward all aggregations to children. It is only used by DemuxOperator.

Overrides:

flush in class Operator<GroupByDesc>

Throws:

HiveException

closeOp

public void closeOp(boolean abort)
             throws HiveException

We need to forward all the aggregations to children.

Overrides:

closeOp in class Operator<GroupByDesc>

Throws:

HiveException

genColLists

public List<String> genColLists(HashMap<Operator<? extends OperatorDesc>,OpParseContext> opParseCtx)

getName

public String getName()

Description copied from class: Operator

Implements the getName function for the Node Interface.

Specified by:

getName in interface Node

Overrides:

getName in class Operator<GroupByDesc>

Returns:

the name of the operator

getOperatorName

public static String getOperatorName()

getType

public OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Specified by:

getType in class Operator<GroupByDesc>

Returns:

OperatorType.*

acceptLimitPushdown

public boolean acceptLimitPushdown()

we can push the limit above GBY (running in Reducer), since that will generate single row for each group. This doesn't necessarily hold for GBY (running in Mappers), so we don't push limit above it.

Overrides:

acceptLimitPushdown in class Operator<GroupByDesc>