org.apache.hadoop.hive.ql.io

Class AcidUtils

java.lang.Object

org.apache.hadoop.hive.ql.io.AcidUtils

public class AcidUtils
extends Object

Utilities that are shared by all of the ACID input and output formats. They are used by the compactor and cleaner and thus must be format agnostic.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	AcidUtils.Directory
static class	AcidUtils.Operation
static class	AcidUtils.ParsedDelta

Field Summary

Fields

Modifier and Type	Field and Description
static String	BASE_PREFIX
static org.apache.hadoop.fs.PathFilter	baseFileFilter
static Pattern	BUCKET_DIGIT_PATTERN
static String	BUCKET_DIGITS
static String	BUCKET_PREFIX
static org.apache.hadoop.fs.PathFilter	bucketFileFilter
static String	CONF_ACID_KEY
static String	DELTA_DIGITS
static String	DELTA_PREFIX
static String	DELTA_SIDE_FILE_SUFFIX
static org.apache.hadoop.fs.PathFilter	deltaFileFilter
static org.apache.hadoop.fs.PathFilter	hiddenFileFilter
static Pattern	LEGACY_BUCKET_DIGIT_PATTERN
static org.apache.hadoop.fs.PathFilter	originalBucketFilter

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.hadoop.fs.Path	createBucketFile(org.apache.hadoop.fs.Path subdir, int bucket) Create the bucket filename.
static org.apache.hadoop.fs.Path	createFilename(org.apache.hadoop.fs.Path directory, AcidOutputFormat.Options options) Create a filename for a bucket file.
static org.apache.hadoop.fs.Path[]	deserializeDeltas(org.apache.hadoop.fs.Path root, List<Long> deltas) Convert the list of begin/end transaction id pairs to a list of delta directories.
static AcidUtils.Directory	getAcidState(org.apache.hadoop.fs.Path directory, org.apache.hadoop.conf.Configuration conf, ValidTxnList txnList) Get the ACID state of the given directory.
static org.apache.hadoop.fs.Path[]	getPaths(List<AcidUtils.ParsedDelta> deltas) Convert a list of deltas to a list of delta directories.
static boolean	isAcid(org.apache.hadoop.fs.Path directory, org.apache.hadoop.conf.Configuration conf) Is the given directory in ACID format?
static AcidOutputFormat.Options	parseBaseBucketFilename(org.apache.hadoop.fs.Path bucketFile, org.apache.hadoop.conf.Configuration conf) Parse a bucket filename back into the options that would have created the file.
static List<Long>	serializeDeltas(List<AcidUtils.ParsedDelta> deltas) Convert the list of deltas into an equivalent list of begin/end transaction id pairs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONF_ACID_KEY

public static final String CONF_ACID_KEY

See Also:

Constant Field Values

BASE_PREFIX

public static final String BASE_PREFIX

See Also:

Constant Field Values

baseFileFilter

public static final org.apache.hadoop.fs.PathFilter baseFileFilter

DELTA_PREFIX

public static final String DELTA_PREFIX

See Also:

Constant Field Values

DELTA_SIDE_FILE_SUFFIX

public static final String DELTA_SIDE_FILE_SUFFIX

See Also:

Constant Field Values

deltaFileFilter

public static final org.apache.hadoop.fs.PathFilter deltaFileFilter

BUCKET_PREFIX

public static final String BUCKET_PREFIX

See Also:

Constant Field Values

bucketFileFilter

public static final org.apache.hadoop.fs.PathFilter bucketFileFilter

BUCKET_DIGITS

public static final String BUCKET_DIGITS

See Also:

Constant Field Values

DELTA_DIGITS

public static final String DELTA_DIGITS

See Also:

Constant Field Values

BUCKET_DIGIT_PATTERN

public static final Pattern BUCKET_DIGIT_PATTERN

LEGACY_BUCKET_DIGIT_PATTERN

public static final Pattern LEGACY_BUCKET_DIGIT_PATTERN

originalBucketFilter

public static final org.apache.hadoop.fs.PathFilter originalBucketFilter

hiddenFileFilter

public static final org.apache.hadoop.fs.PathFilter hiddenFileFilter

Method Detail

createBucketFile

public static org.apache.hadoop.fs.Path createBucketFile(org.apache.hadoop.fs.Path subdir,
                                         int bucket)

Create the bucket filename.

Parameters:

subdir - the subdirectory for the bucket.

bucket - the bucket number

Returns:

the filename

createFilename

public static org.apache.hadoop.fs.Path createFilename(org.apache.hadoop.fs.Path directory,
                                       AcidOutputFormat.Options options)

Create a filename for a bucket file.

Parameters:

directory - the partition directory

options - the options for writing the bucket

Returns:

the filename that should store the bucket

parseBaseBucketFilename

public static AcidOutputFormat.Options parseBaseBucketFilename(org.apache.hadoop.fs.Path bucketFile,
                                               org.apache.hadoop.conf.Configuration conf)

Parse a bucket filename back into the options that would have created the file.

Parameters:

bucketFile - the path to a bucket file

conf - the configuration

Returns:

the options used to create that filename

getPaths

public static org.apache.hadoop.fs.Path[] getPaths(List<AcidUtils.ParsedDelta> deltas)

Convert a list of deltas to a list of delta directories.

Parameters:

deltas - the list of deltas out of a Directory object.

Returns:

a list of delta directory paths that need to be read

serializeDeltas

public static List<Long> serializeDeltas(List<AcidUtils.ParsedDelta> deltas)

Convert the list of deltas into an equivalent list of begin/end transaction id pairs.

Parameters:

deltas -

Returns:

the list of transaction ids to serialize

deserializeDeltas

public static org.apache.hadoop.fs.Path[] deserializeDeltas(org.apache.hadoop.fs.Path root,
                                            List<Long> deltas)

Convert the list of begin/end transaction id pairs to a list of delta directories.

Parameters:

root - the root directory

deltas - list of begin/end transaction id pairs

Returns:

the list of delta paths

isAcid

public static boolean isAcid(org.apache.hadoop.fs.Path directory,
             org.apache.hadoop.conf.Configuration conf)
                      throws IOException

Is the given directory in ACID format?

Parameters:

directory - the partition directory to check

conf - the query configuration

Returns:

true, if it is an ACID directory

Throws:

IOException

getAcidState

public static AcidUtils.Directory getAcidState(org.apache.hadoop.fs.Path directory,
                               org.apache.hadoop.conf.Configuration conf,
                               ValidTxnList txnList)
                                        throws IOException

Get the ACID state of the given directory. It finds the minimal set of base and diff directories. Note that because major compactions don't preserve the history, we can't use a base directory that includes a transaction id that we must exclude.

Parameters:

directory - the partition directory to analyze

conf - the configuration

txnList - the list of transactions that we are reading

Returns:

the state of the directory

Throws:

IOException