org.apache.hadoop.hive.ql.parse.spark

Class GenSparkUtils

java.lang.Object

org.apache.hadoop.hive.ql.parse.spark.GenSparkUtils

public class GenSparkUtils
extends Object

GenSparkUtils is a collection of shared helper methods to produce SparkWork Cloned from GenTezUtils.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	GenSparkUtils()

Method Summary

Methods

Modifier and Type	Method and Description
void	annotateMapWork(GenSparkProcContext context) Fill MapWork with 'local' work and bucket information for SMB Join.
MapWork	createMapWork(GenSparkProcContext context, Operator<?> root, SparkWork sparkWork, PrunedPartitionList partitions)
MapWork	createMapWork(GenSparkProcContext context, Operator<?> root, SparkWork sparkWork, PrunedPartitionList partitions, boolean deferSetup)
ReduceWork	createReduceWork(GenSparkProcContext context, Operator<?> root, SparkWork sparkWork)
static <T> T	getChildOperator(Operator<?> op, Class<T> klazz)
static SparkEdgeProperty	getEdgeProperty(ReduceSinkOperator reduceSink, ReduceWork reduceWork)
int	getNextSeqNumber()
static GenSparkUtils	getUtils()
void	processFileSink(GenSparkProcContext context, FileSinkOperator fileSink)
void	removeUnionOperators(org.apache.hadoop.conf.Configuration conf, GenSparkProcContext context, BaseWork work)
void	resetSequenceNumber()
protected void	setupMapWork(MapWork mapWork, GenSparkProcContext context, PrunedPartitionList partitions, Operator<? extends OperatorDesc> root, String alias)
protected void	setupReduceSink(GenSparkProcContext context, ReduceWork reduceWork, ReduceSinkOperator reduceSink)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GenSparkUtils

protected GenSparkUtils()

Method Detail

getUtils

public static GenSparkUtils getUtils()

resetSequenceNumber

public void resetSequenceNumber()

createReduceWork

public ReduceWork createReduceWork(GenSparkProcContext context,
                          Operator<?> root,
                          SparkWork sparkWork)
                            throws SemanticException

Throws:

SemanticException

setupReduceSink

protected void setupReduceSink(GenSparkProcContext context,
                   ReduceWork reduceWork,
                   ReduceSinkOperator reduceSink)

createMapWork

public MapWork createMapWork(GenSparkProcContext context,
                    Operator<?> root,
                    SparkWork sparkWork,
                    PrunedPartitionList partitions)
                      throws SemanticException

Throws:

SemanticException

createMapWork

public MapWork createMapWork(GenSparkProcContext context,
                    Operator<?> root,
                    SparkWork sparkWork,
                    PrunedPartitionList partitions,
                    boolean deferSetup)
                      throws SemanticException

Throws:

SemanticException

setupMapWork

protected void setupMapWork(MapWork mapWork,
                GenSparkProcContext context,
                PrunedPartitionList partitions,
                Operator<? extends OperatorDesc> root,
                String alias)
                     throws SemanticException

Throws:

SemanticException

removeUnionOperators

public void removeUnionOperators(org.apache.hadoop.conf.Configuration conf,
                        GenSparkProcContext context,
                        BaseWork work)
                          throws SemanticException

Throws:

SemanticException

processFileSink

public void processFileSink(GenSparkProcContext context,
                   FileSinkOperator fileSink)
                     throws SemanticException

Throws:

SemanticException

getEdgeProperty

public static SparkEdgeProperty getEdgeProperty(ReduceSinkOperator reduceSink,
                                ReduceWork reduceWork)
                                         throws SemanticException

Throws:

SemanticException

getChildOperator

public static <T> T getChildOperator(Operator<?> op,
                     Class<T> klazz)
                          throws SemanticException

Throws:

SemanticException

annotateMapWork

public void annotateMapWork(GenSparkProcContext context)
                     throws SemanticException

Fill MapWork with 'local' work and bucket information for SMB Join.

Parameters:

context - context, containing references to MapWorks and their SMB information.

Throws:

SemanticException

getNextSeqNumber

public int getNextSeqNumber()