org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinInnerGenerateResultOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, Node

Direct Known Subclasses:

VectorMapJoinInnerLongOperator, VectorMapJoinInnerMultiKeyOperator, VectorMapJoinInnerStringOperator

public abstract class VectorMapJoinInnerGenerateResultOperator
extends VectorMapJoinGenerateResultOperator

This class has methods for generating vectorized join results for inner joins. Inner joins use a hash map to lookup the 1 or more small table values. One vector inner join optimization is projecting inner keys. When a key appears in the small table results area, instead of copying or referencing key we just include that key again in the output projection. Another optimization is when an inner join does not have any small table columns in the join result, we use a different variation call inner big only. That variation uses a hash multi-set instead of hash map since there are no values (just a count).

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	allMatchs
protected int[]	equalKeySeriesAllMatchIndices
protected int[]	equalKeySeriesDuplicateCounts
protected int[]	equalKeySeriesHashMapResultIndices
protected boolean[]	equalKeySeriesIsSingleValue
protected VectorMapJoinHashMapResult[]	hashMapResults
protected int[]	spillHashMapResultIndices
protected int[]	spills

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

batchCounter

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

bigTableByteColumnVectorColumns, bigTableFilterExpressions, bigTableKeyColumnMap, bigTableKeyExpressions, bigTableKeyTypeNames, bigTableOuterKeyMapping, bigTableOuterKeyOutputVectorColumns, bigTableRetainedMapping, bigTableRetainedVectorCopy, bigTableValueColumnMap, bigTableValueExpressions, bigTableValueTypeNames, bigTableVectorCopyOuterKeys, isOuterJoin, needCommonSetup, needHashTableSetup, outputProjection, outputTypeNames, overflowBatch, overflowRepeatedThreshold, posSingleVectorMapJoinSmallTable, smallTableByteColumnVectorColumns, smallTableMapping, smallTableOutputVectorColumns, smallTableVectorDeserializeRow, spillReplayBatch, taskName, useOverflowRepeatedThreshold, vContext, vectorMapJoinHashTable, vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, hashMapRowGetters, loader, mapJoinTables, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVECOUNTERCREATEDFILES, HIVECOUNTERFATAL, id, inputObjInspectors, isLogDebugEnabled, isLogInfoEnabled, isLogTraceEnabled, operatorId, out, outputObjInspector, parentOperators, PLOG, reporter, state, statsMap

Constructor Summary

Constructors

Constructor and Description
VectorMapJoinInnerGenerateResultOperator()
VectorMapJoinInnerGenerateResultOperator(VectorizationContext vContext, OperatorDesc conf)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	commonSetup(VectorizedRowBatch batch)
protected void	finishInner(VectorizedRowBatch batch, int allMatchCount, int equalKeySeriesCount, int spillCount, int hashMapResultCount) Generate the inner join output results for one vectorized row batch.
protected void	finishInnerRepeated(VectorizedRowBatch batch, JoinUtil.JoinResult joinResult, VectorMapJoinHashTableResult hashMapResult)
protected void	innerPerBatchSetup(VectorizedRowBatch batch) Do the per-batch setup for an inner join.

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

closeOp, forwardBigTableBatch, forwardOverflow, generateHashMapResultMultiValue, generateHashMapResultRepeatedAll, generateHashMapResultSingleValue, intArrayToRangesString, longArrayToRangesString, performValueExpressions, reloadHashTable, reProcessBigTable, spillBatchRepeated, spillHashMapBatch, verifyMonotonicallyIncreasing

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

determineCommonInfo, displayBatchColumns, getHashTableLoader, getOuputVectorizationContext, getType, initializeOp, loadHashTable, setupOverflowBatch, setupVOutContext

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

cleanUpInputFileChangedOp, completeInitializationOp, endGroup, generateMapMetaData, getName, getOperatorName, getRefKey, getValueObjectInspectors, process, setMapJoinKey, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setDone, setExecContext, setGroupKeyObject, setId, setInputContext, setInputObjInspectors, setOperatorId, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

hashMapResults

protected transient VectorMapJoinHashMapResult[] hashMapResults

allMatchs

protected transient int[] allMatchs

equalKeySeriesHashMapResultIndices

protected transient int[] equalKeySeriesHashMapResultIndices

equalKeySeriesAllMatchIndices

protected transient int[] equalKeySeriesAllMatchIndices

equalKeySeriesIsSingleValue

protected transient boolean[] equalKeySeriesIsSingleValue

equalKeySeriesDuplicateCounts

protected transient int[] equalKeySeriesDuplicateCounts

spills

protected transient int[] spills

spillHashMapResultIndices

protected transient int[] spillHashMapResultIndices

Constructor Detail

VectorMapJoinInnerGenerateResultOperator

public VectorMapJoinInnerGenerateResultOperator()

VectorMapJoinInnerGenerateResultOperator

public VectorMapJoinInnerGenerateResultOperator(VectorizationContext vContext,
                                        OperatorDesc conf)
                                         throws HiveException

Throws:

HiveException

Method Detail

commonSetup

protected void commonSetup(VectorizedRowBatch batch)
                    throws HiveException

Overrides:

commonSetup in class VectorMapJoinGenerateResultOperator

Throws:

HiveException

innerPerBatchSetup

protected void innerPerBatchSetup(VectorizedRowBatch batch)

Do the per-batch setup for an inner join.

finishInner

protected void finishInner(VectorizedRowBatch batch,
               int allMatchCount,
               int equalKeySeriesCount,
               int spillCount,
               int hashMapResultCount)
                    throws HiveException,
                           IOException

Generate the inner join output results for one vectorized row batch.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

allMatchCount - Number of matches in allMatchs.

equalKeySeriesCount - Number of single value matches.

spillCount - Number of spills in spills.

hashMapResultCount - Number of entries in hashMapResults.

Throws:

HiveException

IOException

finishInnerRepeated

protected void finishInnerRepeated(VectorizedRowBatch batch,
                       JoinUtil.JoinResult joinResult,
                       VectorMapJoinHashTableResult hashMapResult)
                            throws HiveException,
                                   IOException

Throws:

HiveException

IOException