org.apache.hadoop.hive.ql.exec

Class Operator<T extends OperatorDesc>

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

All Implemented Interfaces:

Serializable, Cloneable, Node

Direct Known Subclasses:

AbstractFileMergeOperator, AppMasterEventOperator, CollectOperator, CommonJoinOperator, DemuxOperator, DummyStoreOperator, FilterOperator, ForwardOperator, GroupByOperator, HashTableDummyOperator, LateralViewForwardOperator, LateralViewJoinOperator, LimitOperator, ListSinkOperator, MapOperator, MuxOperator, PTFOperator, ScriptOperator, SelectOperator, TableScanOperator, TerminalOperator, UDTFOperator, UnionOperator, VectorGroupByOperator, VectorSelectOperator

public abstract class Operator<T extends OperatorDesc>
extends Object
implements Serializable, Cloneable, Node

Base operator implementation.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	Operator.OperatorFunc OperatorFunc.
static class	Operator.State State.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	alias
protected List<Operator<? extends OperatorDesc>>	childOperators
protected Operator<? extends OperatorDesc>[]	childOperatorsArray Cache childOperators in an array for faster access.
protected int[]	childOperatorsTag
protected Map<String,ExprNodeDesc>	colExprMap A map of output column name to input expression map.
protected T	conf
static String	CONTEXT_NAME_KEY
protected boolean	done
protected Object	groupKeyObject
static String	HIVECOUNTERCREATEDFILES
static String	HIVECOUNTERFATAL
protected String	id
protected ObjectInspector[]	inputObjInspectors
protected boolean	isLogDebugEnabled
protected boolean	isLogInfoEnabled
protected boolean	isLogTraceEnabled
protected org.apache.commons.logging.Log	LOG
protected String	operatorId
protected org.apache.hadoop.mapred.OutputCollector	out
protected ObjectInspector	outputObjInspector
protected List<Operator<? extends OperatorDesc>>	parentOperators
protected org.apache.commons.logging.Log	PLOG
protected org.apache.hadoop.mapred.Reporter	reporter
protected Operator.State	state
protected Map<String,org.apache.hadoop.io.LongWritable>	statsMap

Constructor Summary

Constructors

Constructor and Description
Operator()
Operator(org.apache.hadoop.mapred.Reporter reporter) Create an operator with a reporter.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	acceptLimitPushdown() used for LimitPushdownOptimizer if all of the operators between limit and reduce-sink does not remove any input rows in the range of limit count, limit can be pushed down to reduce-sink operator.
protected boolean	allInitializedParentsAreClosed()
protected boolean	areAllParentsInitialized() checks whether all parent operators are initialized or not.
void	augmentPlan() Called during semantic analysis as operators are being added in order to give them a chance to compute any additional plan information needed.
void	cleanUpInputFileChanged()
void	cleanUpInputFileChangedOp()
Operator<? extends OperatorDesc>	clone()
Operator<? extends OperatorDesc>	cloneOp() Clones only the operator.
Operator<? extends OperatorDesc>	cloneRecursiveChildren() Recursively clones all the children of the tree, Fixes the pointers to children, parents and the pointers to itself coming from the children.
void	close(boolean abort)
protected void	closeOp(boolean abort) Operator specific close routine.
boolean	columnNamesRowResolvedCanBeObtained()
protected void	completeInitializationOp(Object[] os) This metod can be used to retrieve the results from async operations started at init time - before the operator pipeline is started.
static Operator	createDummy()
protected void	defaultEndGroup()
protected void	defaultStartGroup()
String	dump(int level)
String	dump(int level, HashSet<Integer> seenOpts)
void	endGroup()
void	flush()
protected void	forward(Object row, ObjectInspector rowInspector)
protected List<String>	getAdditionalCounters()
List<Operator<? extends OperatorDesc>>	getChildOperators()
ArrayList<Node>	getChildren() Implements the getChildren function for the Node Interface.
Map<String,ExprNodeDesc>	getColumnExprMap() Returns a map of output column name to input expression map Note that currently it returns only key columns for ReduceSink and GroupBy operators.
T	getConf()
org.apache.hadoop.conf.Configuration	getConfiguration()
boolean	getDone()
ExecMapperContext	getExecContext()
Object	getGroupKeyObject()
String	getIdentifier() This function is not named getId(), to make sure java serialization does NOT serialize it.
ObjectInspector[]	getInputObjInspectors()
String	getName() Implements the getName function for the Node Interface.
protected long	getNextCntr(long cntr)
int	getNumChild()
int	getNumParent()
String	getOperatorId()
static String	getOperatorName()
OpTraits	getOpTraits()
ObjectInspector	getOutputObjInspector()
List<Operator<? extends OperatorDesc>>	getParentOperators()
RowSchema	getSchema()
Statistics	getStatistics()
Map<String,Long>	getStats()
abstract OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
protected static ObjectInspector[]	initEvaluators(ExprNodeEvaluator<?>[] evals, int start, int length, ObjectInspector rowInspector) Initialize an array of ExprNodeEvaluator from start, for specified length and return the result ObjectInspectors.
protected static ObjectInspector[]	initEvaluators(ExprNodeEvaluator<?>[] evals, ObjectInspector rowInspector) Initialize an array of ExprNodeEvaluator and return the result ObjectInspectors.
protected static StructObjectInspector	initEvaluatorsAndReturnStruct(ExprNodeEvaluator<?>[] evals, List<String> outputColName, ObjectInspector rowInspector) Initialize an array of ExprNodeEvaluator and put the return values into a StructObjectInspector with integer field names.
void	initialize(org.apache.hadoop.conf.Configuration hconf, ObjectInspector[] inputOIs) Initializes operators only if all parents have been initialized.
protected void	initialize(org.apache.hadoop.conf.Configuration hconf, ObjectInspector inputOI, int parentId) Collects all the parent's output object inspectors and calls actual initialization method.
protected void	initializeChildren(org.apache.hadoop.conf.Configuration hconf) Calls initialize on each of the children with outputObjetInspector as the output row format.
void	initializeLocalWork(org.apache.hadoop.conf.Configuration hconf)
protected Collection<Future<?>>	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
void	initOperatorId()
boolean	isUseBucketizedHiveInputFormat()
void	jobClose(org.apache.hadoop.conf.Configuration conf, boolean success) Unlike other operator interfaces which are called from map or reduce task, jobClose is called from the jobclient side once the job has completed.
void	jobCloseOp(org.apache.hadoop.conf.Configuration conf, boolean success)
void	logStats()
boolean	opAllowedAfterMapJoin()
boolean	opAllowedBeforeMapJoin()
boolean	opAllowedBeforeSortMergeJoin()
boolean	opAllowedConvertMapJoin()
void	passExecContext(ExecMapperContext execContext) Pass the execContext reference to every child operator
void	preorderMap(Operator.OperatorFunc opFunc)
abstract void	process(Object row, int tag) Process the row.
void	processGroup(int tag)
void	removeChild(Operator<? extends OperatorDesc> child)
void	removeChildAndAdoptItsChildren(Operator<? extends OperatorDesc> child) Remove a child and add all of the child's children to the location of the child
boolean	removeChildren(int depth)
void	removeParent(Operator<? extends OperatorDesc> parent)
void	replaceChild(Operator<? extends OperatorDesc> child, Operator<? extends OperatorDesc> newChild) Replace one child with another at the same position.
void	replaceParent(Operator<? extends OperatorDesc> parent, Operator<? extends OperatorDesc> newParent) Replace one parent with another at the same position.
void	reset()
static void	resetId()
void	resetStats()
void	setAlias(String alias) Store the alias this operator is working on behalf of.
void	setChildOperators(List<Operator<? extends OperatorDesc>> childOperators)
void	setColumnExprMap(Map<String,ExprNodeDesc> colExprMap)
void	setConf(T conf)
protected void	setDone(boolean done)
void	setExecContext(ExecMapperContext execContext)
void	setGroupKeyObject(Object keyObject)
void	setId(String id)
void	setInputContext(String inputPath, String tableName, String partitionName)
void	setInputObjInspectors(ObjectInspector[] inputObjInspectors)
void	setOperatorId(String operatorId)
void	setOpTraits(OpTraits metaInfo)
void	setOutputCollector(org.apache.hadoop.mapred.OutputCollector out)
void	setParentOperators(List<Operator<? extends OperatorDesc>> parentOperators)
void	setReporter(org.apache.hadoop.mapred.Reporter rep)
void	setSchema(RowSchema rowSchema)
void	setStatistics(Statistics stats)
void	setUseBucketizedHiveInputFormat(boolean useBucketizedHiveInputFormat) Before setting this to true make sure it's not reading ACID tables
void	startGroup()
boolean	supportAutomaticSortMergeJoin() Whether this operator supports automatic sort merge join.
boolean	supportSkewJoinOptimization()
boolean	supportUnionRemoveOptimization()
String	toString()
static String	toString(Collection<Operator<? extends OperatorDesc>> top)

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

HIVECOUNTERCREATEDFILES

public static final String HIVECOUNTERCREATEDFILES

See Also:

Constant Field Values

HIVECOUNTERFATAL

public static final String HIVECOUNTERFATAL

See Also:

Constant Field Values

CONTEXT_NAME_KEY

public static final String CONTEXT_NAME_KEY

See Also:

Constant Field Values

childOperators

protected List<Operator<? extends OperatorDesc>> childOperators

parentOperators

protected List<Operator<? extends OperatorDesc>> parentOperators

operatorId

protected String operatorId

state

protected transient Operator.State state

conf

protected T extends OperatorDesc conf

done

protected boolean done

statsMap

protected transient Map<String,org.apache.hadoop.io.LongWritable> statsMap

out

protected transient org.apache.hadoop.mapred.OutputCollector out

LOG

protected final transient org.apache.commons.logging.Log LOG

PLOG

protected final transient org.apache.commons.logging.Log PLOG

isLogInfoEnabled

protected final transient boolean isLogInfoEnabled

isLogDebugEnabled

protected final transient boolean isLogDebugEnabled

isLogTraceEnabled

protected final transient boolean isLogTraceEnabled

alias

protected transient String alias

reporter

protected transient org.apache.hadoop.mapred.Reporter reporter

id

protected transient String id

inputObjInspectors

protected transient ObjectInspector[] inputObjInspectors

outputObjInspector

protected transient ObjectInspector outputObjInspector

colExprMap

protected Map<String,ExprNodeDesc> colExprMap

A map of output column name to input expression map. This is used by optimizer and built during semantic analysis contains only key elements for reduce sink and group by op

childOperatorsArray

protected transient Operator<? extends OperatorDesc>[] childOperatorsArray

Cache childOperators in an array for faster access. childOperatorsArray is accessed per row, so it's important to make the access efficient.

childOperatorsTag

protected transient int[] childOperatorsTag

groupKeyObject

protected transient Object groupKeyObject

Constructor Detail

Operator

public Operator()

Operator

public Operator(org.apache.hadoop.mapred.Reporter reporter)

Create an operator with a reporter.

Parameters:

reporter - Used to report progress of certain operators.

Method Detail

resetId

public static void resetId()

setChildOperators

public void setChildOperators(List<Operator<? extends OperatorDesc>> childOperators)

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

getChildOperators

public List<Operator<? extends OperatorDesc>> getChildOperators()

getNumChild

public int getNumChild()

getChildren

public ArrayList<Node> getChildren()

Implements the getChildren function for the Node Interface.

Specified by:

getChildren in interface Node

Returns:

List

setParentOperators

public void setParentOperators(List<Operator<? extends OperatorDesc>> parentOperators)

getParentOperators

public List<Operator<? extends OperatorDesc>> getParentOperators()

getNumParent

public int getNumParent()

setConf

public void setConf(T conf)

getConf

public T getConf()

getDone

public boolean getDone()

setDone

protected final void setDone(boolean done)

setSchema

public void setSchema(RowSchema rowSchema)

getSchema

public RowSchema getSchema()

setId

public void setId(String id)

getIdentifier

public String getIdentifier()

This function is not named getId(), to make sure java serialization does NOT serialize it. Some TestParse tests will fail if we serialize this field, since the Operator ID will change based on the number of query tests.

setReporter

public void setReporter(org.apache.hadoop.mapred.Reporter rep)

setOutputCollector

public void setOutputCollector(org.apache.hadoop.mapred.OutputCollector out)

setAlias

public void setAlias(String alias)

Store the alias this operator is working on behalf of.

getStats

public Map<String,Long> getStats()

areAllParentsInitialized

protected boolean areAllParentsInitialized()

checks whether all parent operators are initialized or not.

Returns:

true if there are no parents or all parents are initialized. false otherwise

initialize

public final void initialize(org.apache.hadoop.conf.Configuration hconf,
              ObjectInspector[] inputOIs)
                      throws HiveException

Initializes operators only if all parents have been initialized. Calls operator specific initializer which then initializes child ops.

Parameters:

hconf -

inputOIs - input object inspector array indexes by tag id. null value is ignored.

Throws:

HiveException

completeInitializationOp

protected void completeInitializationOp(Object[] os)
                                 throws HiveException

This metod can be used to retrieve the results from async operations started at init time - before the operator pipeline is started.

Parameters:

os -

Throws:

HiveException

initializeLocalWork

public void initializeLocalWork(org.apache.hadoop.conf.Configuration hconf)
                         throws HiveException

Throws:

HiveException

initializeOp

protected Collection<Future<?>> initializeOp(org.apache.hadoop.conf.Configuration hconf)
                                      throws HiveException

Operator specific initialization.

Throws:

HiveException

initializeChildren

protected void initializeChildren(org.apache.hadoop.conf.Configuration hconf)
                           throws HiveException

Calls initialize on each of the children with outputObjetInspector as the output row format.

Throws:

HiveException

passExecContext

public void passExecContext(ExecMapperContext execContext)

Pass the execContext reference to every child operator

initialize

protected void initialize(org.apache.hadoop.conf.Configuration hconf,
              ObjectInspector inputOI,
              int parentId)
                   throws HiveException

Collects all the parent's output object inspectors and calls actual initialization method.

Parameters:

hconf -

inputOI - OI of the row that this parent will pass to this op

parentId - parent operator id

Throws:

HiveException

getInputObjInspectors

public ObjectInspector[] getInputObjInspectors()

setInputObjInspectors

public void setInputObjInspectors(ObjectInspector[] inputObjInspectors)

getOutputObjInspector

public ObjectInspector getOutputObjInspector()

process

public abstract void process(Object row,
           int tag)
                      throws HiveException

Process the row.

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

defaultStartGroup

protected final void defaultStartGroup()
                                throws HiveException

Throws:

HiveException

defaultEndGroup

protected final void defaultEndGroup()
                              throws HiveException

Throws:

HiveException

startGroup

public void startGroup()
                throws HiveException

Throws:

HiveException

endGroup

public void endGroup()
              throws HiveException

Throws:

HiveException

flush

public void flush()
           throws HiveException

Throws:

HiveException

processGroup

public void processGroup(int tag)
                  throws HiveException

Throws:

HiveException

allInitializedParentsAreClosed

protected boolean allInitializedParentsAreClosed()

close

public void close(boolean abort)
           throws HiveException

Throws:

HiveException

closeOp

protected void closeOp(boolean abort)
                throws HiveException

Operator specific close routine. Operators which inherents this class should overwrite this funtion for their specific cleanup routine.

Throws:

HiveException

jobCloseOp

public void jobCloseOp(org.apache.hadoop.conf.Configuration conf,
              boolean success)
                throws HiveException

Throws:

HiveException

jobClose

public void jobClose(org.apache.hadoop.conf.Configuration conf,
            boolean success)
              throws HiveException

Unlike other operator interfaces which are called from map or reduce task, jobClose is called from the jobclient side once the job has completed.

Parameters:

conf - Configuration with with which job was submitted

success - whether the job was completed successfully or not

Throws:

HiveException

replaceChild

public void replaceChild(Operator<? extends OperatorDesc> child,
                Operator<? extends OperatorDesc> newChild)

Replace one child with another at the same position. The parent of the child is not changed

Parameters:

child - the old child

newChild - the new child

removeChild

public void removeChild(Operator<? extends OperatorDesc> child)

removeChildAndAdoptItsChildren

public void removeChildAndAdoptItsChildren(Operator<? extends OperatorDesc> child)
                                    throws SemanticException

Remove a child and add all of the child's children to the location of the child

Parameters:

child - If this operator is not the only parent of the child. There can be unpredictable result.

Throws:

SemanticException

removeParent

public void removeParent(Operator<? extends OperatorDesc> parent)

removeChildren

public boolean removeChildren(int depth)

replaceParent

public void replaceParent(Operator<? extends OperatorDesc> parent,
                 Operator<? extends OperatorDesc> newParent)

Replace one parent with another at the same position. Chilren of the new parent are not updated

Parameters:

parent - the old parent

newParent - the new parent

getNextCntr

protected long getNextCntr(long cntr)

forward

protected void forward(Object row,
           ObjectInspector rowInspector)
                throws HiveException

Throws:

HiveException

resetStats

public void resetStats()

reset

public void reset()

preorderMap

public void preorderMap(Operator.OperatorFunc opFunc)

logStats

public void logStats()

getName

public String getName()

Implements the getName function for the Node Interface.

Specified by:

getName in interface Node

Returns:

the name of the operator

getOperatorName

public static String getOperatorName()

getColumnExprMap

public Map<String,ExprNodeDesc> getColumnExprMap()

Returns a map of output column name to input expression map Note that currently it returns only key columns for ReduceSink and GroupBy operators.

Returns:

null if the operator doesn't change columns

setColumnExprMap

public void setColumnExprMap(Map<String,ExprNodeDesc> colExprMap)

dump

public String dump(int level)

dump

public String dump(int level,
          HashSet<Integer> seenOpts)

initEvaluators

protected static ObjectInspector[] initEvaluators(ExprNodeEvaluator<?>[] evals,
                               ObjectInspector rowInspector)
                                           throws HiveException

Initialize an array of ExprNodeEvaluator and return the result ObjectInspectors.

Throws:

HiveException

initEvaluators

protected static ObjectInspector[] initEvaluators(ExprNodeEvaluator<?>[] evals,
                               int start,
                               int length,
                               ObjectInspector rowInspector)
                                           throws HiveException

Initialize an array of ExprNodeEvaluator from start, for specified length and return the result ObjectInspectors.

Throws:

HiveException

initEvaluatorsAndReturnStruct

protected static StructObjectInspector initEvaluatorsAndReturnStruct(ExprNodeEvaluator<?>[] evals,
                                                  List<String> outputColName,
                                                  ObjectInspector rowInspector)
                                                              throws HiveException

Initialize an array of ExprNodeEvaluator and put the return values into a StructObjectInspector with integer field names.

Throws:

HiveException

getOperatorId

public String getOperatorId()

initOperatorId

public void initOperatorId()

setOperatorId

public void setOperatorId(String operatorId)

getAdditionalCounters

protected List<String> getAdditionalCounters()

getType

public abstract OperatorType getType()

Return the type of the specific operator among the types in OperatorType.

Returns:

OperatorType.*

setGroupKeyObject

public void setGroupKeyObject(Object keyObject)

getGroupKeyObject

public Object getGroupKeyObject()

augmentPlan

public void augmentPlan()

Called during semantic analysis as operators are being added in order to give them a chance to compute any additional plan information needed. Does nothing by default.

getExecContext

public ExecMapperContext getExecContext()

setExecContext

public void setExecContext(ExecMapperContext execContext)

cleanUpInputFileChanged

public void cleanUpInputFileChanged()
                             throws HiveException

Throws:

HiveException

cleanUpInputFileChangedOp

public void cleanUpInputFileChangedOp()
                               throws HiveException

Throws:

HiveException

setInputContext

public void setInputContext(String inputPath,
                   String tableName,
                   String partitionName)

supportSkewJoinOptimization

public boolean supportSkewJoinOptimization()

clone

public Operator<? extends OperatorDesc> clone()
                                       throws CloneNotSupportedException

Overrides:

clone in class Object

Throws:

CloneNotSupportedException

cloneOp

public Operator<? extends OperatorDesc> cloneOp()
                                         throws CloneNotSupportedException

Clones only the operator. The children and parent are set to null.

Returns:

Cloned operator

Throws:

CloneNotSupportedException

cloneRecursiveChildren

public Operator<? extends OperatorDesc> cloneRecursiveChildren()
                                                        throws CloneNotSupportedException

Recursively clones all the children of the tree, Fixes the pointers to children, parents and the pointers to itself coming from the children. It does not fix the pointers to itself coming from parents, parents continue to point to the original child.

Returns:

Cloned operator

Throws:

CloneNotSupportedException

columnNamesRowResolvedCanBeObtained

public boolean columnNamesRowResolvedCanBeObtained()

isUseBucketizedHiveInputFormat

public boolean isUseBucketizedHiveInputFormat()

setUseBucketizedHiveInputFormat

public void setUseBucketizedHiveInputFormat(boolean useBucketizedHiveInputFormat)

Before setting this to true make sure it's not reading ACID tables

Parameters:

useBucketizedHiveInputFormat -

supportAutomaticSortMergeJoin

public boolean supportAutomaticSortMergeJoin()

Whether this operator supports automatic sort merge join. The stack is traversed, and this method is invoked for all the operators.

Returns:

TRUE if yes, FALSE otherwise.

supportUnionRemoveOptimization

public boolean supportUnionRemoveOptimization()

opAllowedBeforeMapJoin

public boolean opAllowedBeforeMapJoin()

opAllowedAfterMapJoin

public boolean opAllowedAfterMapJoin()

opAllowedConvertMapJoin

public boolean opAllowedConvertMapJoin()

opAllowedBeforeSortMergeJoin

public boolean opAllowedBeforeSortMergeJoin()

acceptLimitPushdown

public boolean acceptLimitPushdown()

used for LimitPushdownOptimizer if all of the operators between limit and reduce-sink does not remove any input rows in the range of limit count, limit can be pushed down to reduce-sink operator. forward, select, etc.

toString

public String toString()

Overrides:

toString in class Object

toString

public static String toString(Collection<Operator<? extends OperatorDesc>> top)

getStatistics

public Statistics getStatistics()

getOpTraits

public OpTraits getOpTraits()

setOpTraits

public void setOpTraits(OpTraits metaInfo)

setStatistics

public void setStatistics(Statistics stats)

createDummy

public static Operator createDummy()