org.apache.hadoop.hive.ql.parse

Class SemanticAnalyzer

java.lang.Object

org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

Direct Known Subclasses:

CalcitePlanner, ColumnStatsSemanticAnalyzer, UpdateDeleteSemanticAnalyzer

public class SemanticAnalyzer
extends BaseSemanticAnalyzer

Implementation of the semantic analyzer. It generates the query plan. There are other specific semantic analyzers for some hive operations such as DDLSemanticAnalyzer for ddl operations.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SemanticAnalyzer.GenericUDAFInfo Class to store GenericUDAF related information.

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

BaseSemanticAnalyzer.AnalyzeRewriteContext, BaseSemanticAnalyzer.TableSpec

Field Summary

Fields

Modifier and Type	Field and Description
protected BaseSemanticAnalyzer.AnalyzeRewriteContext	analyzeRewrite
protected boolean	disableJoinMerge
static String	DUMMY_DATABASE
static String	DUMMY_TABLE
protected boolean	noscan
protected LinkedHashMap<Operator<? extends OperatorDesc>,OpParseContext>	opParseCtx
protected boolean	partialscan
static String	SUBQUERY_TAG_1
static String	SUBQUERY_TAG_2
protected HashMap<String,Operator<? extends OperatorDesc>>	topOps
protected UnparseTranslator	unparseTranslator

Fields inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

acidFileSinks, columnAccessInfo, conf, console, ctx, db, fetchTask, HIVE_COLUMN_ORDER_ASC, HIVE_COLUMN_ORDER_DESC, idToTableNameMap, inputs, linfo, LOG, outputs, queryProperties, rootTasks, STATIC_LOG, tableAccessInfo, updateColumnAccessInfo

Constructor Summary

Constructors

Constructor and Description
SemanticAnalyzer(HiveConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	analyzeInternal(ASTNode ast)
protected boolean	deleting()
protected static boolean	distinctExprsExists(QB qb)
boolean	doPhase1(ASTNode ast, QB qb, org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.Phase1Ctx ctx_1, org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.PlannerContext plannerCtx) Phase 1: (including, but not limited to): 1.
void	doPhase1QBExpr(ASTNode ast, QBExpr qbexpr, String id, String alias)
Map<ASTNode,ExprNodeDesc>	genAllExprNodeDesc(ASTNode expr, RowResolver input) Generates an expression node descriptors for the expression and children of it with default TypeCheckCtx.
Map<ASTNode,ExprNodeDesc>	genAllExprNodeDesc(ASTNode expr, RowResolver input, TypeCheckCtx tcCtx) Generates all of the expression node descriptors for the expression and children of it passed in the arguments.
static String	generateErrorMessage(ASTNode ast, String message)
ExprNodeDesc	genExprNodeDesc(ASTNode expr, RowResolver input) Generates an expression node descriptor for the expression with TypeCheckCtx.
ExprNodeDesc	genExprNodeDesc(ASTNode expr, RowResolver input, TypeCheckCtx tcCtx) Returns expression node descriptor for the expression.
protected Operator	genFileSinkPlan(String dest, QB qb, Operator input)
Operator	genPlan(QB qb)
Operator	genPlan(QB qb, boolean skipAmbiguityCheck)
static ASTNode	genSelectDIAST(RowResolver rr)
protected String	getAliasId(String alias, QB qb)
static String	getColumnInternalName(int pos)
static GenericUDAFEvaluator	getGenericUDAFEvaluator(String aggName, ArrayList<ExprNodeDesc> aggParameters, ASTNode aggTree, boolean isDistinct, boolean isAllColumns) Returns the GenericUDAFEvaluator for the aggregation.
static SemanticAnalyzer.GenericUDAFInfo	getGenericUDAFInfo(GenericUDAFEvaluator evaluator, GenericUDAFEvaluator.Mode emode, ArrayList<ExprNodeDesc> aggParameters) Returns the GenericUDAFInfo struct for the aggregation.
protected List<Integer>	getGroupingSets(List<ASTNode> groupByExpr, QBParseInfo parseInfo, String dest)
protected List<Integer>	getGroupingSetsForCube(int size)
protected List<Integer>	getGroupingSetsForRollup(int size)
void	getMetaData(QB qb)
void	getMetaData(QB qb, ReadEntity parentInput)
ParseContext	getParseContext()
QB	getQB()
List<FieldSchema>	getResultSchema()
RowResolver	getRowResolver(Operator opt) Get the row resolver given an operator.
Table	getTable(TableScanOperator ts)
static GenericUDAFEvaluator.Mode	groupByDescModeToUDAFMode(GroupByDesc.Mode mode, boolean isDistinct)
void	init(boolean clearPartsCache)
void	initParseCtx(ParseContext pctx)
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.Phase1Ctx	initPhase1Ctx()
static boolean	isAcidTable(Table tab)
static ExprNodeDesc	isConstantParameterInAggregationParameters(String internalName, List<ExprNodeDesc> reduceValues) Check if the given internalName represents a constant parameter in aggregation parameters of an aggregation tree.
static ArrayList<WindowingSpec.WindowExpressionSpec>	parseSelect(String selectExprStr)
protected void	processNoScanCommand(ASTNode tree) process analyze ...
protected void	processPartialScanCommand(ASTNode tree) process analyze ...
<T extends OperatorDesc> Operator<T>	putOpInsertMap(Operator<T> op, RowResolver rr)
protected void	reset(boolean clearPartsCache)
protected void	setAST(ASTNode newAST)
static int	setBit(int bitmap, int bitIdx)
void	setQB(QB qb)
protected boolean	updating()
void	validate()

Methods inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

analyze, analyzeDDLSkewedValues, analyzeSkewedTablDDLColNames, analyzeStoredAdDirs, charSetString, constructListBucketingCtx, createHiveDB, escapeSQLString, getAcidFileSinks, getColumnAccessInfo, getColumnNames, getColumnNamesOrder, getColumns, getColumns, getDatabase, getDatabase, getDb, getDbTableNamePair, getDotName, getFetchTask, getIdToTableNameMap, getInputs, getLineageInfo, getOutputs, getPartition, getPartitions, getQualifiedTableName, getQueryProperties, getRootTasks, getSkewedValueFromASTNode, getSkewedValuesFromASTNode, getTable, getTable, getTable, getTable, getTable, getTableAccessInfo, getTypeStringFromAST, getUnescapedName, getUnescapedName, getUnescapedUnqualifiedTableName, getUpdateColumnAccessInfo, initCtx, isValidPrefixSpec, readProps, setColumnAccessInfo, setFetchTask, setLineageInfo, setTableAccessInfo, setUpdateColumnAccessInfo, skipAuthorization, stripIdentifierQuotes, stripQuotes, toMessage, toReadEntity, toReadEntity, toWriteEntity, toWriteEntity, unescapeIdentifier, unescapeSQLString, validatePartColumnType, validatePartSpec

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DUMMY_DATABASE

public static final String DUMMY_DATABASE

See Also:

Constant Field Values

DUMMY_TABLE

public static final String DUMMY_TABLE

See Also:

Constant Field Values

SUBQUERY_TAG_1

public static final String SUBQUERY_TAG_1

See Also:

Constant Field Values

SUBQUERY_TAG_2

public static final String SUBQUERY_TAG_2

See Also:

Constant Field Values

topOps

protected HashMap<String,Operator<? extends OperatorDesc>> topOps

opParseCtx

protected LinkedHashMap<Operator<? extends OperatorDesc>,OpParseContext> opParseCtx

unparseTranslator

protected final UnparseTranslator unparseTranslator

noscan

protected boolean noscan

partialscan

protected boolean partialscan

disableJoinMerge

protected volatile boolean disableJoinMerge

analyzeRewrite

protected BaseSemanticAnalyzer.AnalyzeRewriteContext analyzeRewrite

Constructor Detail

SemanticAnalyzer

public SemanticAnalyzer(HiveConf conf)
                 throws SemanticException

Throws:

SemanticException

Method Detail

reset

protected void reset(boolean clearPartsCache)

Overrides:

reset in class BaseSemanticAnalyzer

initParseCtx

public void initParseCtx(ParseContext pctx)

getParseContext

public ParseContext getParseContext()

doPhase1QBExpr

public void doPhase1QBExpr(ASTNode ast,
                  QBExpr qbexpr,
                  String id,
                  String alias)
                    throws SemanticException

Throws:

SemanticException

generateErrorMessage

public static String generateErrorMessage(ASTNode ast,
                          String message)

setAST

protected void setAST(ASTNode newAST)

doPhase1

public boolean doPhase1(ASTNode ast,
               QB qb,
               org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.Phase1Ctx ctx_1,
               org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.PlannerContext plannerCtx)
                 throws SemanticException

Phase 1: (including, but not limited to): 1. Gets all the aliases for all the tables / subqueries and makes the appropriate mapping in aliasToTabs, aliasToSubq 2. Gets the location of the destination and names the clause "inclause" + i 3. Creates a map from a string representation of an aggregation tree to the actual aggregation AST 4. Creates a mapping from the clause name to the select expression AST in destToSelExpr 5. Creates a mapping from a table alias to the lateral view AST's in aliasToLateralViews

Parameters:

ast -

qb -

ctx_1 -

Throws:

SemanticException

getTable

public Table getTable(TableScanOperator ts)

getMetaData

public void getMetaData(QB qb)
                 throws SemanticException

Throws:

SemanticException

getMetaData

public void getMetaData(QB qb,
               ReadEntity parentInput)
                 throws SemanticException

Throws:

SemanticException

putOpInsertMap

public <T extends OperatorDesc> Operator<T> putOpInsertMap(Operator<T> op,
                                                  RowResolver rr)

getColumnInternalName

public static String getColumnInternalName(int pos)

getGroupingSetsForRollup

protected List<Integer> getGroupingSetsForRollup(int size)

getGroupingSetsForCube

protected List<Integer> getGroupingSetsForCube(int size)

getGroupingSets

protected List<Integer> getGroupingSets(List<ASTNode> groupByExpr,
                            QBParseInfo parseInfo,
                            String dest)
                                 throws SemanticException

Throws:

SemanticException

setBit

public static int setBit(int bitmap,
         int bitIdx)

getGenericUDAFEvaluator

public static GenericUDAFEvaluator getGenericUDAFEvaluator(String aggName,
                                           ArrayList<ExprNodeDesc> aggParameters,
                                           ASTNode aggTree,
                                           boolean isDistinct,
                                           boolean isAllColumns)
                                                    throws SemanticException

Returns the GenericUDAFEvaluator for the aggregation. This is called once for each GroupBy aggregation.

Throws:

SemanticException

getGenericUDAFInfo

public static SemanticAnalyzer.GenericUDAFInfo getGenericUDAFInfo(GenericUDAFEvaluator evaluator,
                                                  GenericUDAFEvaluator.Mode emode,
                                                  ArrayList<ExprNodeDesc> aggParameters)
                                                           throws SemanticException

Returns the GenericUDAFInfo struct for the aggregation.

Parameters:

aggName - The name of the UDAF.

aggParameters - The exprNodeDesc of the original parameters

aggTree - The ASTNode node of the UDAF in the query.

Returns:

GenericUDAFInfo

Throws:

SemanticException - when the UDAF is not found or has problems.

groupByDescModeToUDAFMode

public static GenericUDAFEvaluator.Mode groupByDescModeToUDAFMode(GroupByDesc.Mode mode,
                                                  boolean isDistinct)

isConstantParameterInAggregationParameters

public static ExprNodeDesc isConstantParameterInAggregationParameters(String internalName,
                                                      List<ExprNodeDesc> reduceValues)

Check if the given internalName represents a constant parameter in aggregation parameters of an aggregation tree. This method is only invoked when map-side aggregation is not involved. In this case, every parameter in every aggregation tree should already have a corresponding ColumnInfo, which is generated when the corresponding ReduceSinkOperator of the GroupByOperator being generating is generated. If we find that this parameter is a constant parameter, we will return the corresponding ExprNodeDesc in reduceValues, and we will not need to use a new ExprNodeColumnDesc, which can not be treated as a constant parameter, for this parameter (since the writableObjectInspector of a ExprNodeColumnDesc will not be a instance of ConstantObjectInspector).

Parameters:

reduceValues - value columns of the corresponding ReduceSinkOperator

internalName - the internal name of this parameter

Returns:

the ExprNodeDesc of the constant parameter if the given internalName represents a constant parameter; otherwise, return null

genFileSinkPlan

protected Operator genFileSinkPlan(String dest,
                       QB qb,
                       Operator input)
                            throws SemanticException

Throws:

SemanticException

distinctExprsExists

protected static boolean distinctExprsExists(QB qb)

getAliasId

protected String getAliasId(String alias,
                QB qb)

genPlan

public Operator genPlan(QB qb)
                 throws SemanticException

Throws:

SemanticException

genPlan

public Operator genPlan(QB qb,
               boolean skipAmbiguityCheck)
                 throws SemanticException

Throws:

SemanticException

initPhase1Ctx

public org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.Phase1Ctx initPhase1Ctx()

init

public void init(boolean clearPartsCache)

Overrides:

init in class BaseSemanticAnalyzer

analyzeInternal

public void analyzeInternal(ASTNode ast)
                     throws SemanticException

Specified by:

analyzeInternal in class BaseSemanticAnalyzer

Throws:

SemanticException

getResultSchema

public List<FieldSchema> getResultSchema()

Overrides:

getResultSchema in class BaseSemanticAnalyzer

Returns:

the schema for the fields which will be produced when the statement is executed, or null if not known

genExprNodeDesc

public ExprNodeDesc genExprNodeDesc(ASTNode expr,
                           RowResolver input)
                             throws SemanticException

Generates an expression node descriptor for the expression with TypeCheckCtx.

Throws:

SemanticException

genAllExprNodeDesc

public Map<ASTNode,ExprNodeDesc> genAllExprNodeDesc(ASTNode expr,
                                           RowResolver input)
                                             throws SemanticException

Generates an expression node descriptors for the expression and children of it with default TypeCheckCtx.

Throws:

SemanticException

genExprNodeDesc

public ExprNodeDesc genExprNodeDesc(ASTNode expr,
                           RowResolver input,
                           TypeCheckCtx tcCtx)
                             throws SemanticException

Returns expression node descriptor for the expression. If it's evaluated already in previous operator, it can be retrieved from cache.

Throws:

SemanticException

genAllExprNodeDesc

public Map<ASTNode,ExprNodeDesc> genAllExprNodeDesc(ASTNode expr,
                                           RowResolver input,
                                           TypeCheckCtx tcCtx)
                                             throws SemanticException

Generates all of the expression node descriptors for the expression and children of it passed in the arguments. This function uses the row resolver and the metadata information that are passed as arguments to resolve the column names to internal names.

Parameters:

expr - The expression

input - The row resolver

tcCtx - Customized type-checking context

Returns:

expression to exprNodeDesc mapping

Throws:

SemanticException - Failed to evaluate expression

validate

public void validate()
              throws SemanticException

Overrides:

validate in class BaseSemanticAnalyzer

Throws:

SemanticException

getRowResolver

public RowResolver getRowResolver(Operator opt)

Get the row resolver given an operator.

processPartialScanCommand

protected void processPartialScanCommand(ASTNode tree)
                                  throws SemanticException

process analyze ... partial command separate it from noscan command process so that it provides us flexibility

Parameters:

tree -

Throws:

SemanticException

processNoScanCommand

protected void processNoScanCommand(ASTNode tree)
                             throws SemanticException

process analyze ... noscan command

Parameters:

tree -

Throws:

SemanticException

getQB

public QB getQB()

setQB

public void setQB(QB qb)

parseSelect

public static ArrayList<WindowingSpec.WindowExpressionSpec> parseSelect(String selectExprStr)
                                                                 throws SemanticException

Throws:

SemanticException

isAcidTable

public static boolean isAcidTable(Table tab)

updating

protected boolean updating()

deleting

protected boolean deleting()

genSelectDIAST

public static ASTNode genSelectDIAST(RowResolver rr)