org.apache.hadoop.hive.ql.optimizer.correlation

Class ReduceSinkDeDuplication.AbsctractReducerReducerProc

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.correlation.ReduceSinkDeDuplication.AbsctractReducerReducerProc

All Implemented Interfaces:

NodeProcessor

Enclosing class:

ReduceSinkDeDuplication

public abstract static class ReduceSinkDeDuplication.AbsctractReducerReducerProc
extends Object
implements NodeProcessor

Constructor Summary

Constructors

Constructor and Description
ReduceSinkDeDuplication.AbsctractReducerReducerProc()

Method Summary

Methods

Modifier and Type	Method and Description
protected Integer	checkNumReducer(int creduce, int preduce) If number of reducers for RS is -1, the RS can have any number of reducers.
protected Integer	checkOrder(String corder, String porder)
protected boolean	merge(ReduceSinkOperator cRS, JoinOperator pJoin, int minReducer)
protected boolean	merge(ReduceSinkOperator cRS, ReduceSinkOperator pRS, int minReducer) Current RSDedup remove/replace child RS.
Object	process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) Generic process for all ops that don't have specific implementations.
protected abstract Object	process(ReduceSinkOperator cRS, GroupByOperator cGBY, ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
protected abstract Object	process(ReduceSinkOperator cRS, ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
protected Integer	sameKeys(List<ExprNodeDesc> cexprs, List<ExprNodeDesc> pexprs, Operator<?> child, Operator<?> parent)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ReduceSinkDeDuplication.AbsctractReducerReducerProc

public ReduceSinkDeDuplication.AbsctractReducerReducerProc()

Method Detail

process

public Object process(Node nd,
             Stack<Node> stack,
             NodeProcessorCtx procCtx,
             Object... nodeOutputs)
               throws SemanticException

Description copied from interface: NodeProcessor

Generic process for all ops that don't have specific implementations.

Specified by:

process in interface NodeProcessor

Parameters:

nd - operator to process

procCtx - operator processor context

nodeOutputs - A variable argument list of outputs from other nodes in the walk

Returns:

Object to be returned by the process call

Throws:

SemanticException

process

protected abstract Object process(ReduceSinkOperator cRS,
             ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
                           throws SemanticException

Throws:

SemanticException

process

protected abstract Object process(ReduceSinkOperator cRS,
             GroupByOperator cGBY,
             ReduceSinkDeDuplication.ReduceSinkDeduplicateProcCtx dedupCtx)
                           throws SemanticException

Throws:

SemanticException

merge

protected boolean merge(ReduceSinkOperator cRS,
            JoinOperator pJoin,
            int minReducer)
                 throws SemanticException

Throws:

SemanticException

merge

protected boolean merge(ReduceSinkOperator cRS,
            ReduceSinkOperator pRS,
            int minReducer)
                 throws SemanticException

Current RSDedup remove/replace child RS. For key columns, sorting order, and the number of reducers, copy more specific part of configurations of child RS to that of parent RS. For partitioning columns, if both child RS and parent RS have been assigned partitioning columns, we will choose the more general partitioning columns. If parent RS has not been assigned any partitioning column, we will use partitioning columns (if exist) of child RS.

Throws:

SemanticException

sameKeys

protected Integer sameKeys(List<ExprNodeDesc> cexprs,
               List<ExprNodeDesc> pexprs,
               Operator<?> child,
               Operator<?> parent)
                    throws SemanticException

Throws:

SemanticException

checkOrder

protected Integer checkOrder(String corder,
                 String porder)

checkNumReducer

protected Integer checkNumReducer(int creduce,
                      int preduce)

If number of reducers for RS is -1, the RS can have any number of reducers. It's generally true except for order-by or forced bucketing cases. if both of num-reducers are not -1, those number should be the same.