org.apache.hadoop.hive.ql.optimizer.spark

Class SparkMapJoinOptimizer

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.spark.SparkMapJoinOptimizer

All Implemented Interfaces:

NodeProcessor

public class SparkMapJoinOptimizer
extends Object
implements NodeProcessor

SparkMapJoinOptimizer cloned from ConvertJoinMapJoin is an optimization that replaces a common join (aka shuffle join) with a map join (aka broadcast or fragment replicate join when possible. Map joins have restrictions on which joins can be converted (e.g.: full outer joins cannot be handled as map joins) as well as memory restrictions (one side of the join has to fit into memory).

Constructor Summary

Constructors

Constructor and Description
SparkMapJoinOptimizer()

Method Summary

Methods

Modifier and Type	Method and Description
MapJoinOperator	convertJoinMapJoin(JoinOperator joinOp, OptimizeSparkProcContext context, int bigTablePosition)
Object	process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) Generic process for all ops that don't have specific implementations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkMapJoinOptimizer

public SparkMapJoinOptimizer()

Method Detail

process

public Object process(Node nd,
             Stack<Node> stack,
             NodeProcessorCtx procCtx,
             Object... nodeOutputs)
               throws SemanticException

Description copied from interface: NodeProcessor

Generic process for all ops that don't have specific implementations.

Specified by:

process in interface NodeProcessor

Parameters:

nd - operator to process

procCtx - operator processor context

nodeOutputs - A variable argument list of outputs from other nodes in the walk

Returns:

Object to be returned by the process call

Throws:

SemanticException

convertJoinMapJoin

public MapJoinOperator convertJoinMapJoin(JoinOperator joinOp,
                                 OptimizeSparkProcContext context,
                                 int bigTablePosition)
                                   throws SemanticException

Throws:

SemanticException