org.apache.hadoop.hive.metastore

Class MetaStoreUtils

java.lang.Object

org.apache.hadoop.hive.metastore.MetaStoreUtils

public class MetaStoreUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static String	ARCHIVING_LEVEL
static String	DATABASE_WAREHOUSE_SUFFIX
static String	DEFAULT_DATABASE_COMMENT
static String	DEFAULT_DATABASE_NAME
protected static org.apache.commons.logging.Log	LOG

Constructor Summary

Constructors

Constructor and Description
MetaStoreUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static ClassLoader	addToClassPath(ClassLoader cloader, String[] newPaths) Add new elements to the classpath.
static boolean	canDropPartition(Table table, Partition partition)
static boolean	compareFieldColumns(List<FieldSchema> schema1, List<FieldSchema> schema2)
static boolean	containsAllFastStats(Map<String,String> partParams)
static Table	createColumnsetSchema(String name, List<String> columns, List<String> partCols, org.apache.hadoop.conf.Configuration conf)
static void	deleteWHDirectory(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, boolean use_trash)
static int	findFreePort() Finds a free port on the machine.
static int	getArchivingLevel(Partition part)
static Class<?>	getClass(String rawStoreClassName)
static String	getColumnCommentsFromFieldSchema(List<FieldSchema> fieldSchemas)
static String	getColumnNamesFromFieldSchema(List<FieldSchema> fieldSchemas) Convert FieldSchemas to columnNames.
static String	getColumnTypesFromFieldSchema(List<FieldSchema> fieldSchemas) Convert FieldSchemas to columnTypes.
static String	getDDLFromFieldSchema(String structName, List<FieldSchema> fieldSchemas) Convert FieldSchemas to Thrift DDL.
static Deserializer	getDeserializer(org.apache.hadoop.conf.Configuration conf, Partition part, Table table) getDeserializer Get the Deserializer for a partition.
static Deserializer	getDeserializer(org.apache.hadoop.conf.Configuration conf, Table table, boolean skipConfError) getDeserializer Get the Deserializer for a table.
static Class<? extends Deserializer>	getDeserializerClass(org.apache.hadoop.conf.Configuration conf, Table table)
static FieldSchema	getFieldSchemaFromTypeInfo(String fieldName, TypeInfo typeInfo) Convert TypeInfo to FieldSchema.
static List<FieldSchema>	getFieldsFromDeserializer(String tableName, Deserializer deserializer)
static String	getFullDDLFromFieldSchema(String structName, List<FieldSchema> fieldSchemas) Convert FieldSchemas to Thrift DDL + column names and column types
static String	getIndexTableName(String dbName, String baseTblName, String indexName)
static String	getListType(String t)
static String	getMapType(String k, String v)
static Map<String,String>	getMetaStoreSaslProperties(HiveConf conf) Read and return the meta store Sasl configuration.
static org.apache.hadoop.fs.Path	getOriginalLocation(Partition part)
static Properties	getPartitionMetadata(Partition partition, Table table)
static Properties	getPartSchemaFromTableSchema(StorageDescriptor sd, StorageDescriptor tblsd, Map<String,String> parameters, String databaseName, String tableName, List<FieldSchema> partitionKeys, Properties tblSchema) Get partition level schema from table level schema.
static ProtectMode	getProtectMode(Partition partition)
static ProtectMode	getProtectMode(Table table)
static List<String>	getPvals(List<FieldSchema> partCols, Map<String,String> partSpec) Given a list of partition columns and a partial mapping from some partition columns to values the function returns the values for the column.
static String[]	getQualifiedName(String defaultDbName, String tableName)
static Properties	getSchema(Partition part, Table table)
static Properties	getSchema(StorageDescriptor sd, StorageDescriptor tblsd, Map<String,String> parameters, String databaseName, String tableName, List<FieldSchema> partitionKeys)
static Properties	getTableMetadata(Table table)
static boolean	isArchived(Partition part)
static boolean	isDirEmpty(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Utility method that determines if a specified directory already has contents (non-hidden files) or not - useful to determine if an immutable table already has contents, for example.
static boolean	isExternalTable(Table table) Determines whether a table is an external table.
static boolean	isImmutableTable(Table table) Determines whether a table is an immutable table.
static boolean	isIndexTable(Table table)
static boolean	isNonNativeTable(Table table)
static boolean	isView(Table table)
static void	makeDir(org.apache.hadoop.fs.Path path, HiveConf hiveConf)
static String	makeFilterStringFromMap(Map<String,String> m) Given a map of partition column names to values, this creates a filter string that can be used to call the *byFilter methods
static <T> T	newInstance(Class<T> theClass, Class<?>[] parameterTypes, Object[] initargs) Create an object of the given class.
static boolean	partitionNameHasValidCharacters(List<String> partVals, Pattern partitionValidationPattern)
static void	populateQuickStats(org.apache.hadoop.fs.FileStatus[] fileStatus, Map<String,String> params)
static boolean	pvalMatches(List<String> partial, List<String> full) Returns true if partial has the same values as full for all values that aren't empty in partial.
static void	recursiveDelete(File f) recursiveDelete just recursively deletes a dir - you'd think Java would have something to do this??
static boolean	requireCalStats(org.apache.hadoop.conf.Configuration hiveConf, Partition oldPart, Partition newPart, Table tbl)
static void	setSerdeParam(SerDeInfo sdi, Properties schema, String param)
static void	startMetaStore(int port, HadoopThriftAuthBridge bridge)
static void	startMetaStore(int port, HadoopThriftAuthBridge bridge, HiveConf hiveConf)
static Map<String,String>	trimMapNulls(Map<String,String> dnMap, boolean retrieveMapNullsAsEmptyStrings) We have aneed to sanity-check the map before conversion from persisted objects to metadata thrift objects because null values in maps will cause a NPE if we send across thrift.
static String	typeToThriftType(String type) Convert type to ThriftType.
static boolean	updatePartitionStatsFast(PartitionSpecProxy.PartitionIterator part, Warehouse wh, boolean madeDir, boolean forceRecompute) Updates the numFiles and totalSize parameters for the passed Partition by querying the warehouse if the passed Partition does not already have values for these parameters.
static boolean	updatePartitionStatsFast(Partition part, Warehouse wh)
static boolean	updatePartitionStatsFast(Partition part, Warehouse wh, boolean madeDir)
static boolean	updatePartitionStatsFast(Partition part, Warehouse wh, boolean madeDir, boolean forceRecompute) Updates the numFiles and totalSize parameters for the passed Partition by querying the warehouse if the passed Partition does not already have values for these parameters.
static boolean	updateUnpartitionedTableStatsFast(Database db, Table tbl, Warehouse wh, boolean madeDir)
static boolean	updateUnpartitionedTableStatsFast(Database db, Table tbl, Warehouse wh, boolean madeDir, boolean forceRecompute)
static boolean	updateUnpartitionedTableStatsFast(Table tbl, org.apache.hadoop.fs.FileStatus[] fileStatus, boolean newDir, boolean forceRecompute) Updates the numFiles and totalSize parameters for the passed unpartitioned Table by querying the warehouse if the passed Table does not already have values for these parameters.
static boolean	validateColumnName(String name)
static boolean	validateColumnType(String type) validate column type if it is predefined, yes.
static boolean	validateName(String name) validateName Checks the name conforms to our standars which are: "[a-zA-z_0-9]+".
static void	validatePartitionNameCharacters(List<String> partVals, Pattern partitionValidationPattern)
static String	validateSkewedColNames(List<String> cols)
static String	validateSkewedColNamesSubsetCol(List<String> skewedColNames, List<FieldSchema> cols)
static String	validateTblColumns(List<FieldSchema> cols)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.apache.commons.logging.Log LOG

DEFAULT_DATABASE_NAME

public static final String DEFAULT_DATABASE_NAME

See Also:

Constant Field Values

DEFAULT_DATABASE_COMMENT

public static final String DEFAULT_DATABASE_COMMENT

See Also:

Constant Field Values

DATABASE_WAREHOUSE_SUFFIX

public static final String DATABASE_WAREHOUSE_SUFFIX

See Also:

Constant Field Values

ARCHIVING_LEVEL

public static String ARCHIVING_LEVEL

Constructor Detail

MetaStoreUtils

public MetaStoreUtils()

Method Detail

createColumnsetSchema

public static Table createColumnsetSchema(String name,
                          List<String> columns,
                          List<String> partCols,
                          org.apache.hadoop.conf.Configuration conf)
                                   throws MetaException

Throws:

MetaException

recursiveDelete

public static void recursiveDelete(File f)
                            throws IOException

recursiveDelete just recursively deletes a dir - you'd think Java would have something to do this??

Parameters:

f - - the file/dir to delete

Throws:

IOException - propogate f.delete() exceptions

containsAllFastStats

public static boolean containsAllFastStats(Map<String,String> partParams)

Parameters:

partParams -

Returns:

True if the passed Parameters Map contains values for all "Fast Stats".

updateUnpartitionedTableStatsFast

public static boolean updateUnpartitionedTableStatsFast(Database db,
                                        Table tbl,
                                        Warehouse wh,
                                        boolean madeDir)
                                                 throws MetaException

Throws:

MetaException

updateUnpartitionedTableStatsFast

public static boolean updateUnpartitionedTableStatsFast(Database db,
                                        Table tbl,
                                        Warehouse wh,
                                        boolean madeDir,
                                        boolean forceRecompute)
                                                 throws MetaException

Throws:

MetaException

updateUnpartitionedTableStatsFast

public static boolean updateUnpartitionedTableStatsFast(Table tbl,
                                        org.apache.hadoop.fs.FileStatus[] fileStatus,
                                        boolean newDir,
                                        boolean forceRecompute)
                                                 throws MetaException

Updates the numFiles and totalSize parameters for the passed unpartitioned Table by querying the warehouse if the passed Table does not already have values for these parameters.

Parameters:

tbl -

fileStatus -

newDir - if true, the directory was just created and can be assumed to be empty

forceRecompute - Recompute stats even if the passed Table already has these parameters set

Returns:

true if the stats were updated, false otherwise

Throws:

MetaException

populateQuickStats

public static void populateQuickStats(org.apache.hadoop.fs.FileStatus[] fileStatus,
                      Map<String,String> params)

requireCalStats

public static boolean requireCalStats(org.apache.hadoop.conf.Configuration hiveConf,
                      Partition oldPart,
                      Partition newPart,
                      Table tbl)

updatePartitionStatsFast

public static boolean updatePartitionStatsFast(Partition part,
                               Warehouse wh)
                                        throws MetaException

Throws:

MetaException

updatePartitionStatsFast

public static boolean updatePartitionStatsFast(Partition part,
                               Warehouse wh,
                               boolean madeDir)
                                        throws MetaException

Throws:

MetaException

updatePartitionStatsFast

public static boolean updatePartitionStatsFast(Partition part,
                               Warehouse wh,
                               boolean madeDir,
                               boolean forceRecompute)
                                        throws MetaException

Updates the numFiles and totalSize parameters for the passed Partition by querying the warehouse if the passed Partition does not already have values for these parameters.

Parameters:

part -

wh -

madeDir - if true, the directory was just created and can be assumed to be empty

forceRecompute - Recompute stats even if the passed Partition already has these parameters set

Returns:

true if the stats were updated, false otherwise

Throws:

MetaException

updatePartitionStatsFast

public static boolean updatePartitionStatsFast(PartitionSpecProxy.PartitionIterator part,
                               Warehouse wh,
                               boolean madeDir,
                               boolean forceRecompute)
                                        throws MetaException

Updates the numFiles and totalSize parameters for the passed Partition by querying the warehouse if the passed Partition does not already have values for these parameters.

Parameters:

part -

wh -

madeDir - if true, the directory was just created and can be assumed to be empty

forceRecompute - Recompute stats even if the passed Partition already has these parameters set

Returns:

true if the stats were updated, false otherwise

Throws:

MetaException

getDeserializer

public static Deserializer getDeserializer(org.apache.hadoop.conf.Configuration conf,
                           Table table,
                           boolean skipConfError)
                                    throws MetaException

getDeserializer Get the Deserializer for a table.

Parameters:

conf - - hadoop config

table - the table

Returns:

Returns instantiated deserializer by looking up class name of deserializer stored in storage descriptor of passed in table. Also, initializes the deserializer with schema of table.

Throws:

MetaException - if any problems instantiating the Deserializer todo - this should move somewhere into serde.jar

getDeserializerClass

public static Class<? extends Deserializer> getDeserializerClass(org.apache.hadoop.conf.Configuration conf,
                                                 Table table)
                                                          throws Exception

Throws:

Exception

getDeserializer

public static Deserializer getDeserializer(org.apache.hadoop.conf.Configuration conf,
                           Partition part,
                           Table table)
                                    throws MetaException

getDeserializer Get the Deserializer for a partition.

Parameters:

conf - - hadoop config

part - the partition

table - the table

Returns:

Returns instantiated deserializer by looking up class name of deserializer stored in storage descriptor of passed in partition. Also, initializes the deserializer with schema of partition.

Throws:

MetaException - if any problems instantiating the Deserializer

deleteWHDirectory

public static void deleteWHDirectory(org.apache.hadoop.fs.Path path,
                     org.apache.hadoop.conf.Configuration conf,
                     boolean use_trash)
                              throws MetaException

Throws:

MetaException

getPvals

public static List<String> getPvals(List<FieldSchema> partCols,
                    Map<String,String> partSpec)

Given a list of partition columns and a partial mapping from some partition columns to values the function returns the values for the column.

Parameters:

partCols - the list of table partition columns

partSpec - the partial mapping from partition column to values

Returns:

list of values of for given partition columns, any missing values in partSpec is replaced by an empty string

validateName

public static boolean validateName(String name)

validateName Checks the name conforms to our standars which are: "[a-zA-z_0-9]+". checks this is just characters and numbers and _

Parameters:

name - the name to validate

Returns:

true or false depending on conformance

Throws:

MetaException - if it doesn't match the pattern.

validateColumnName

public static final boolean validateColumnName(String name)

validateTblColumns

public static String validateTblColumns(List<FieldSchema> cols)

validateColumnType

public static boolean validateColumnType(String type)

validate column type if it is predefined, yes. otherwise no

Parameters:

name -

Returns:

validateSkewedColNames

public static String validateSkewedColNames(List<String> cols)

validateSkewedColNamesSubsetCol

public static String validateSkewedColNamesSubsetCol(List<String> skewedColNames,
                                     List<FieldSchema> cols)

getListType

public static String getListType(String t)

getMapType

public static String getMapType(String k,
                String v)

setSerdeParam

public static void setSerdeParam(SerDeInfo sdi,
                 Properties schema,
                 String param)

typeToThriftType

public static String typeToThriftType(String type)

Convert type to ThriftType. We do that by tokenizing the type and convert each token.

getFullDDLFromFieldSchema

public static String getFullDDLFromFieldSchema(String structName,
                               List<FieldSchema> fieldSchemas)

Convert FieldSchemas to Thrift DDL + column names and column types

Parameters:

structName - The name of the table

fieldSchemas - List of fields along with their schemas

Returns:

String containing "Thrift DDL#comma-separated-column-names#colon-separated-columntypes Example: "struct result { a string, map b}#a,b#string:map"

getDDLFromFieldSchema

public static String getDDLFromFieldSchema(String structName,
                           List<FieldSchema> fieldSchemas)

Convert FieldSchemas to Thrift DDL.

getTableMetadata

public static Properties getTableMetadata(Table table)

getPartitionMetadata

public static Properties getPartitionMetadata(Partition partition,
                              Table table)

getSchema

public static Properties getSchema(Partition part,
                   Table table)

getPartSchemaFromTableSchema

public static Properties getPartSchemaFromTableSchema(StorageDescriptor sd,
                                      StorageDescriptor tblsd,
                                      Map<String,String> parameters,
                                      String databaseName,
                                      String tableName,
                                      List<FieldSchema> partitionKeys,
                                      Properties tblSchema)

Get partition level schema from table level schema. This function will use the same column names, column types and partition keys for each partition Properties. Their values are copied from the table Properties. This is mainly to save CPU and memory. CPU is saved because the first time the StorageDescriptor column names are accessed, JDO needs to execute a SQL query to retrieve the data. If we know the data will be the same as the table level schema and they are immutable, we should just reuse the table level schema objects.

Parameters:

sd - The Partition level Storage Descriptor.

tblsd - The Table level Storage Descriptor.

parameters - partition level parameters

databaseName - DB name

tableName - table name

partitionKeys - partition columns

tblSchema - The table level schema from which this partition should be copied.

Returns:

the properties

getSchema

public static Properties getSchema(StorageDescriptor sd,
                   StorageDescriptor tblsd,
                   Map<String,String> parameters,
                   String databaseName,
                   String tableName,
                   List<FieldSchema> partitionKeys)

getColumnNamesFromFieldSchema

public static String getColumnNamesFromFieldSchema(List<FieldSchema> fieldSchemas)

Convert FieldSchemas to columnNames.

getColumnTypesFromFieldSchema

public static String getColumnTypesFromFieldSchema(List<FieldSchema> fieldSchemas)

Convert FieldSchemas to columnTypes.

getColumnCommentsFromFieldSchema

public static String getColumnCommentsFromFieldSchema(List<FieldSchema> fieldSchemas)

makeDir

public static void makeDir(org.apache.hadoop.fs.Path path,
           HiveConf hiveConf)
                    throws MetaException

Throws:

MetaException

startMetaStore

public static void startMetaStore(int port,
                  HadoopThriftAuthBridge bridge)
                           throws Exception

Throws:

Exception

startMetaStore

public static void startMetaStore(int port,
                  HadoopThriftAuthBridge bridge,
                  HiveConf hiveConf)
                           throws Exception

Throws:

Exception

findFreePort

public static int findFreePort()
                        throws IOException

Finds a free port on the machine.

Returns:

Throws:

IOException

getFieldsFromDeserializer

public static List<FieldSchema> getFieldsFromDeserializer(String tableName,
                                          Deserializer deserializer)
                                                   throws SerDeException,
                                                          MetaException

Parameters:

tableName -

deserializer -

Returns:

the list of fields

Throws:

SerDeException

MetaException

getFieldSchemaFromTypeInfo

public static FieldSchema getFieldSchemaFromTypeInfo(String fieldName,
                                     TypeInfo typeInfo)

Convert TypeInfo to FieldSchema.

isExternalTable

public static boolean isExternalTable(Table table)

Determines whether a table is an external table.

Parameters:

table - table of interest

Returns:

true if external

isImmutableTable

public static boolean isImmutableTable(Table table)

Determines whether a table is an immutable table. Immutable tables are write-once/replace, and do not support append. Partitioned immutable tables do support additions by way of creation of new partitions, but do not allow the partitions themselves to be appended to. "INSERT INTO" will not work for Immutable tables.

Parameters:

table - table of interest

Returns:

true if immutable

isArchived

public static boolean isArchived(Partition part)

getOriginalLocation

public static org.apache.hadoop.fs.Path getOriginalLocation(Partition part)

isNonNativeTable

public static boolean isNonNativeTable(Table table)

isDirEmpty

public static boolean isDirEmpty(org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.fs.Path path)
                          throws IOException

Utility method that determines if a specified directory already has contents (non-hidden files) or not - useful to determine if an immutable table already has contents, for example.

Parameters:

path -

Throws:

IOException

pvalMatches

public static boolean pvalMatches(List<String> partial,
                  List<String> full)

Returns true if partial has the same values as full for all values that aren't empty in partial.

getIndexTableName

public static String getIndexTableName(String dbName,
                       String baseTblName,
                       String indexName)

isIndexTable

public static boolean isIndexTable(Table table)

makeFilterStringFromMap

public static String makeFilterStringFromMap(Map<String,String> m)

Given a map of partition column names to values, this creates a filter string that can be used to call the *byFilter methods

Parameters:

m -

Returns:

the filter string

isView

public static boolean isView(Table table)

getClass

public static Class<?> getClass(String rawStoreClassName)
                         throws MetaException

Throws:

MetaException

newInstance

public static <T> T newInstance(Class<T> theClass,
                Class<?>[] parameterTypes,
                Object[] initargs)

Create an object of the given class.

Parameters:

theClass -

parameterTypes - an array of parameterTypes for the constructor

initargs - the list of arguments for the constructor

validatePartitionNameCharacters

public static void validatePartitionNameCharacters(List<String> partVals,
                                   Pattern partitionValidationPattern)
                                            throws MetaException

Throws:

MetaException

partitionNameHasValidCharacters

public static boolean partitionNameHasValidCharacters(List<String> partVals,
                                      Pattern partitionValidationPattern)

compareFieldColumns

public static boolean compareFieldColumns(List<FieldSchema> schema1,
                          List<FieldSchema> schema2)

Parameters:

schema1: - The first schema to be compared

schema2: - The second schema to be compared

Returns:

true if the two schemas are the same else false for comparing a field we ignore the comment it has

getMetaStoreSaslProperties

public static Map<String,String> getMetaStoreSaslProperties(HiveConf conf)

Read and return the meta store Sasl configuration. Currently it uses the default Hadoop SASL configuration and can be configured using "hadoop.rpc.protection" HADOOP-10211, made a backward incompatible change due to which this call doesn't work with Hadoop 2.4.0 and later.

Parameters:

conf -

Returns:

The SASL configuration

getProtectMode

public static ProtectMode getProtectMode(Partition partition)

getProtectMode

public static ProtectMode getProtectMode(Table table)

canDropPartition

public static boolean canDropPartition(Table table,
                       Partition partition)

getArchivingLevel

public static int getArchivingLevel(Partition part)
                             throws MetaException

Throws:

MetaException

getQualifiedName

public static String[] getQualifiedName(String defaultDbName,
                        String tableName)

trimMapNulls

public static Map<String,String> trimMapNulls(Map<String,String> dnMap,
                              boolean retrieveMapNullsAsEmptyStrings)

We have aneed to sanity-check the map before conversion from persisted objects to metadata thrift objects because null values in maps will cause a NPE if we send across thrift. Pruning is appropriate for most cases except for databases such as Oracle where Empty strings are stored as nulls, in which case we need to handle that. See HIVE-8485 for motivations for this.

addToClassPath

public static ClassLoader addToClassPath(ClassLoader cloader,
                         String[] newPaths)
                                  throws Exception

Add new elements to the classpath.

Parameters:

newPaths - Array of classpath elements

Throws:

Exception