org.apache.hadoop.hive.ql.plan

Class PlanUtils

java.lang.Object

org.apache.hadoop.hive.ql.plan.PlanUtils

public final class PlanUtils
extends Object

PlanUtils.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PlanUtils.ExpressionTypes ExpressionTypes.

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.apache.commons.logging.Log	LOG

Method Summary

Methods

Modifier and Type	Method and Description
static void	addExprToStringBuffer(ExprNodeDesc expr, StringBuffer sb)
static ReadEntity	addInput(Set<ReadEntity> inputs, ReadEntity newInput)
static void	addInputsForView(ParseContext parseCtx)
static void	configureInputJobPropertiesForStorageHandler(TableDesc tableDesc) Loads the storage handler (if one exists) for the given table and invokes HiveStorageHandler.configureInputJobProperties(TableDesc, java.util.Map).
static void	configureJobConf(TableDesc tableDesc, org.apache.hadoop.mapred.JobConf jobConf)
static void	configureOutputJobPropertiesForStorageHandler(TableDesc tableDesc) Loads the storage handler (if one exists) for the given table and invokes HiveStorageHandler.configureOutputJobProperties(TableDesc, java.util.Map).
static long	getCountForMapJoinDumpFilePrefix()
static TableDesc	getDefaultQueryOutputTableDesc(String cols, String colTypes, String fileFormat)
static TableDesc	getDefaultTableDesc(CreateTableDesc directoryDesc, String cols, String colTypes)
static TableDesc	getDefaultTableDesc(String separatorCode) Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode.
static TableDesc	getDefaultTableDesc(String separatorCode, String columns) Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode and column names (comma separated string).
static TableDesc	getDefaultTableDesc(String separatorCode, String columns, boolean lastColumnTakesRestOfTheLine) Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode and column names (comma separated string), and whether the last column should take the rest of the line.
static TableDesc	getDefaultTableDesc(String separatorCode, String columns, String columnTypes, boolean lastColumnTakesRestOfTheLine) Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode and column names (comma separated string), and whether the last column should take the rest of the line.
static String	getExprListString(Collection<? extends ExprNodeDesc> exprs)
static List<FieldSchema>	getFieldSchemasFromColumnInfo(ArrayList<ColumnInfo> cols, String fieldPrefix) Convert the ColumnInfo to FieldSchema.
static List<FieldSchema>	getFieldSchemasFromColumnList(List<ExprNodeDesc> cols, List<String> outputColumnNames, int start, String fieldPrefix) Convert the ColumnList to FieldSchema list.
static List<FieldSchema>	getFieldSchemasFromColumnList(List<ExprNodeDesc> cols, String fieldPrefix) Convert the ColumnList to FieldSchema list.
static List<FieldSchema>	getFieldSchemasFromColumnListWithLength(List<ExprNodeDesc> cols, List<List<Integer>> distinctColIndices, List<String> outputColumnNames, int length, String fieldPrefix) Convert the ColumnList to FieldSchema list.
static List<FieldSchema>	getFieldSchemasFromRowSchema(RowSchema row, String fieldPrefix) Convert the RowSchema to FieldSchema list.
static TableDesc	getIntermediateFileTableDesc(List<FieldSchema> fieldSchemas) Generate the table descriptor for intermediate files.
static TableDesc	getMapJoinKeyTableDesc(org.apache.hadoop.conf.Configuration conf, List<FieldSchema> fieldSchemas) Generate the table descriptor for Map-side join key.
static TableDesc	getMapJoinValueTableDesc(List<FieldSchema> fieldSchemas) Generate the table descriptor for Map-side join value.
static MapredWork	getMapRedWork()
static ReadEntity	getParentViewInfo(String alias_id, Map<String,ReadEntity> viewAliasToInput)
static TableDesc	getReduceKeyTableDesc(List<FieldSchema> fieldSchemas, String order) Generate the table descriptor for reduce key.
static ReduceSinkDesc	getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols, ArrayList<ExprNodeDesc> valueCols, List<String> outputColumnNames, boolean includeKeyCols, int tag, ArrayList<ExprNodeDesc> partitionCols, String order, int numReducers, AcidUtils.Operation writeType) Create the reduce sink descriptor.
static ReduceSinkDesc	getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols, ArrayList<ExprNodeDesc> valueCols, List<String> outputColumnNames, boolean includeKey, int tag, int numPartitionFields, int numReducers, AcidUtils.Operation writeType) Create the reduce sink descriptor.
static ReduceSinkDesc	getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols, int numKeys, ArrayList<ExprNodeDesc> valueCols, List<List<Integer>> distinctColIndices, List<String> outputKeyColumnNames, List<String> outputValueColumnNames, boolean includeKeyCols, int tag, ArrayList<ExprNodeDesc> partitionCols, String order, int numReducers, AcidUtils.Operation writeType) Create the reduce sink descriptor.
static ReduceSinkDesc	getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols, int numKeys, ArrayList<ExprNodeDesc> valueCols, List<List<Integer>> distinctColIndices, List<String> outputKeyColumnNames, List<String> outputValueColumnNames, boolean includeKey, int tag, int numPartitionFields, int numReducers, AcidUtils.Operation writeType) Create the reduce sink descriptor.
static TableDesc	getReduceValueTableDesc(List<FieldSchema> fieldSchemas) Generate the table descriptor for intermediate files.
static TableDesc	getTableDesc(Class<? extends Deserializer> serdeClass, String separatorCode, String columns) Generate the table descriptor of given serde with the separatorCode and column names (comma separated string).
static TableDesc	getTableDesc(Class<? extends Deserializer> serdeClass, String separatorCode, String columns, boolean lastColumnTakesRestOfTheLine) Generate the table descriptor of the serde specified with the separatorCode and column names (comma separated string), and whether the last column should take the rest of the line.
static TableDesc	getTableDesc(Class<? extends Deserializer> serdeClass, String separatorCode, String columns, String columnTypes, boolean lastColumnTakesRestOfTheLine)
static TableDesc	getTableDesc(Class<? extends Deserializer> serdeClass, String separatorCode, String columns, String columnTypes, boolean lastColumnTakesRestOfTheLine, boolean useDelimitedJSON)
static TableDesc	getTableDesc(Class<? extends Deserializer> serdeClass, String separatorCode, String columns, String columnTypes, boolean lastColumnTakesRestOfTheLine, boolean useDelimitedJSON, String fileFormat)
static TableDesc	getTableDesc(CreateTableDesc crtTblDesc, String cols, String colTypes) Generate a table descriptor from a createTableDesc.
static String	removePrefixFromWarehouseConfig(String origiKey) Remove prefix from "Path -> Alias" This is required for testing.
static List<FieldSchema>	sortFieldSchemas(List<FieldSchema> schema)
static String	stripQuotes(String val)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.apache.commons.logging.Log LOG

Method Detail

getCountForMapJoinDumpFilePrefix

public static long getCountForMapJoinDumpFilePrefix()

getMapRedWork

public static MapredWork getMapRedWork()

getDefaultTableDesc

public static TableDesc getDefaultTableDesc(CreateTableDesc directoryDesc,
                            String cols,
                            String colTypes)

getDefaultTableDesc

public static TableDesc getDefaultTableDesc(String separatorCode,
                            String columns)

Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode and column names (comma separated string).

getTableDesc

public static TableDesc getTableDesc(Class<? extends Deserializer> serdeClass,
                     String separatorCode,
                     String columns)

Generate the table descriptor of given serde with the separatorCode and column names (comma separated string).

getDefaultTableDesc

public static TableDesc getDefaultTableDesc(String separatorCode,
                            String columns,
                            boolean lastColumnTakesRestOfTheLine)

Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode and column names (comma separated string), and whether the last column should take the rest of the line.

getTableDesc

public static TableDesc getTableDesc(Class<? extends Deserializer> serdeClass,
                     String separatorCode,
                     String columns,
                     boolean lastColumnTakesRestOfTheLine)

Generate the table descriptor of the serde specified with the separatorCode and column names (comma separated string), and whether the last column should take the rest of the line.

getDefaultTableDesc

public static TableDesc getDefaultTableDesc(String separatorCode,
                            String columns,
                            String columnTypes,
                            boolean lastColumnTakesRestOfTheLine)

Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode and column names (comma separated string), and whether the last column should take the rest of the line.

getTableDesc

public static TableDesc getTableDesc(Class<? extends Deserializer> serdeClass,
                     String separatorCode,
                     String columns,
                     String columnTypes,
                     boolean lastColumnTakesRestOfTheLine)

getTableDesc

public static TableDesc getTableDesc(Class<? extends Deserializer> serdeClass,
                     String separatorCode,
                     String columns,
                     String columnTypes,
                     boolean lastColumnTakesRestOfTheLine,
                     boolean useDelimitedJSON)

getTableDesc

public static TableDesc getTableDesc(Class<? extends Deserializer> serdeClass,
                     String separatorCode,
                     String columns,
                     String columnTypes,
                     boolean lastColumnTakesRestOfTheLine,
                     boolean useDelimitedJSON,
                     String fileFormat)

getDefaultQueryOutputTableDesc

public static TableDesc getDefaultQueryOutputTableDesc(String cols,
                                       String colTypes,
                                       String fileFormat)

getTableDesc

public static TableDesc getTableDesc(CreateTableDesc crtTblDesc,
                     String cols,
                     String colTypes)

Generate a table descriptor from a createTableDesc.

getDefaultTableDesc

public static TableDesc getDefaultTableDesc(String separatorCode)

Generate the table descriptor of MetadataTypedColumnsetSerDe with the separatorCode. MetaDataTypedColumnsetSerDe is used because LazySimpleSerDe does not support a table with a single column "col" with type "array".

getReduceKeyTableDesc

public static TableDesc getReduceKeyTableDesc(List<FieldSchema> fieldSchemas,
                              String order)

Generate the table descriptor for reduce key.

getMapJoinKeyTableDesc

public static TableDesc getMapJoinKeyTableDesc(org.apache.hadoop.conf.Configuration conf,
                               List<FieldSchema> fieldSchemas)

Generate the table descriptor for Map-side join key.

getMapJoinValueTableDesc

public static TableDesc getMapJoinValueTableDesc(List<FieldSchema> fieldSchemas)

Generate the table descriptor for Map-side join value.

getIntermediateFileTableDesc

public static TableDesc getIntermediateFileTableDesc(List<FieldSchema> fieldSchemas)

Generate the table descriptor for intermediate files.

getReduceValueTableDesc

public static TableDesc getReduceValueTableDesc(List<FieldSchema> fieldSchemas)

Generate the table descriptor for intermediate files.

getFieldSchemasFromColumnListWithLength

public static List<FieldSchema> getFieldSchemasFromColumnListWithLength(List<ExprNodeDesc> cols,
                                                        List<List<Integer>> distinctColIndices,
                                                        List<String> outputColumnNames,
                                                        int length,
                                                        String fieldPrefix)

Convert the ColumnList to FieldSchema list. Adds union type for distinctColIndices.

getFieldSchemasFromColumnList

public static List<FieldSchema> getFieldSchemasFromColumnList(List<ExprNodeDesc> cols,
                                              List<String> outputColumnNames,
                                              int start,
                                              String fieldPrefix)

Convert the ColumnList to FieldSchema list.

getFieldSchemasFromColumnList

public static List<FieldSchema> getFieldSchemasFromColumnList(List<ExprNodeDesc> cols,
                                              String fieldPrefix)

Convert the ColumnList to FieldSchema list.

getFieldSchemasFromRowSchema

public static List<FieldSchema> getFieldSchemasFromRowSchema(RowSchema row,
                                             String fieldPrefix)

Convert the RowSchema to FieldSchema list.

getFieldSchemasFromColumnInfo

public static List<FieldSchema> getFieldSchemasFromColumnInfo(ArrayList<ColumnInfo> cols,
                                              String fieldPrefix)

Convert the ColumnInfo to FieldSchema.

sortFieldSchemas

public static List<FieldSchema> sortFieldSchemas(List<FieldSchema> schema)

getReduceSinkDesc

public static ReduceSinkDesc getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols,
                               ArrayList<ExprNodeDesc> valueCols,
                               List<String> outputColumnNames,
                               boolean includeKeyCols,
                               int tag,
                               ArrayList<ExprNodeDesc> partitionCols,
                               String order,
                               int numReducers,
                               AcidUtils.Operation writeType)

Create the reduce sink descriptor.

Parameters:

keyCols - The columns to be stored in the key

valueCols - The columns to be stored in the value

outputColumnNames - The output columns names

tag - The tag for this reducesink

partitionCols - The columns for partitioning.

numReducers - The number of reducers, set to -1 for automatic inference based on input data size.

writeType - Whether this is an Acid write, and if so whether it is insert, update, or delete.

Returns:

The reduceSinkDesc object.

getReduceSinkDesc

public static ReduceSinkDesc getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols,
                               int numKeys,
                               ArrayList<ExprNodeDesc> valueCols,
                               List<List<Integer>> distinctColIndices,
                               List<String> outputKeyColumnNames,
                               List<String> outputValueColumnNames,
                               boolean includeKeyCols,
                               int tag,
                               ArrayList<ExprNodeDesc> partitionCols,
                               String order,
                               int numReducers,
                               AcidUtils.Operation writeType)

Create the reduce sink descriptor.

Parameters:

keyCols - The columns to be stored in the key

numKeys - number of distribution key numbers. Equals to group-by-key numbers usually.

valueCols - The columns to be stored in the value

distinctColIndices - column indices for distinct aggregate parameters

outputKeyColumnNames - The output key columns names

outputValueColumnNames - The output value columns names

tag - The tag for this reducesink

partitionCols - The columns for partitioning.

numReducers - The number of reducers, set to -1 for automatic inference based on input data size.

writeType - Whether this is an Acid write, and if so whether it is insert, update, or delete.

Returns:

The reduceSinkDesc object.

getReduceSinkDesc

public static ReduceSinkDesc getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols,
                               ArrayList<ExprNodeDesc> valueCols,
                               List<String> outputColumnNames,
                               boolean includeKey,
                               int tag,
                               int numPartitionFields,
                               int numReducers,
                               AcidUtils.Operation writeType)
                                        throws SemanticException

Create the reduce sink descriptor.

Parameters:

keyCols - The columns to be stored in the key

valueCols - The columns to be stored in the value

outputColumnNames - The output columns names

tag - The tag for this reducesink

numPartitionFields - The first numPartitionFields of keyCols will be partition columns. If numPartitionFields=-1, then partition randomly.

numReducers - The number of reducers, set to -1 for automatic inference based on input data size.

writeType - Whether this is an Acid write, and if so whether it is insert, update, or delete.

Returns:

The reduceSinkDesc object.

Throws:

SemanticException

getReduceSinkDesc

public static ReduceSinkDesc getReduceSinkDesc(ArrayList<ExprNodeDesc> keyCols,
                               int numKeys,
                               ArrayList<ExprNodeDesc> valueCols,
                               List<List<Integer>> distinctColIndices,
                               List<String> outputKeyColumnNames,
                               List<String> outputValueColumnNames,
                               boolean includeKey,
                               int tag,
                               int numPartitionFields,
                               int numReducers,
                               AcidUtils.Operation writeType)
                                        throws SemanticException

Create the reduce sink descriptor.

Parameters:

keyCols - The columns to be stored in the key

numKeys - number of distribution keys. Equals to group-by-key numbers usually.

valueCols - The columns to be stored in the value

distinctColIndices - column indices for distinct aggregates

outputKeyColumnNames - The output key columns names

outputValueColumnNames - The output value columns names

tag - The tag for this reducesink

numPartitionFields - The first numPartitionFields of keyCols will be partition columns. If numPartitionFields=-1, then partition randomly.

numReducers - The number of reducers, set to -1 for automatic inference based on input data size.

writeType - Whether this is an Acid write, and if so whether it is insert, update, or delete.

Returns:

The reduceSinkDesc object.

Throws:

SemanticException

configureInputJobPropertiesForStorageHandler

public static void configureInputJobPropertiesForStorageHandler(TableDesc tableDesc)

Loads the storage handler (if one exists) for the given table and invokes HiveStorageHandler.configureInputJobProperties(TableDesc, java.util.Map).

Parameters:

tableDesc - table descriptor

configureOutputJobPropertiesForStorageHandler

public static void configureOutputJobPropertiesForStorageHandler(TableDesc tableDesc)

Loads the storage handler (if one exists) for the given table and invokes HiveStorageHandler.configureOutputJobProperties(TableDesc, java.util.Map).

Parameters:

tableDesc - table descriptor

configureJobConf

public static void configureJobConf(TableDesc tableDesc,
                    org.apache.hadoop.mapred.JobConf jobConf)

stripQuotes

public static String stripQuotes(String val)

removePrefixFromWarehouseConfig

public static String removePrefixFromWarehouseConfig(String origiKey)

Remove prefix from "Path -> Alias" This is required for testing. In order to verify that path is right, we need to display it in expected test result. But, mask pattern masks path with some patterns. So, we need to remove prefix from path which triggers mask pattern.

Parameters:

origiKey -

Returns:

addInput

public static ReadEntity addInput(Set<ReadEntity> inputs,
                  ReadEntity newInput)

getExprListString

public static String getExprListString(Collection<? extends ExprNodeDesc> exprs)

addExprToStringBuffer

public static void addExprToStringBuffer(ExprNodeDesc expr,
                         StringBuffer sb)

addInputsForView

public static void addInputsForView(ParseContext parseCtx)
                             throws HiveException

Throws:

HiveException

getParentViewInfo

public static ReadEntity getParentViewInfo(String alias_id,
                           Map<String,ReadEntity> viewAliasToInput)