org.apache.hadoop.hive.metastore.txn

Interface TxnStore

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public interface TxnStore

A handler to answer transaction related calls that come into the metastore server. Note on log messages: Please include txnid:X and lockid info using JavaUtils.txnIdToString(long) and JavaUtils.lockIdToString(long) in all messages. The txnid:X and lockid:Y matches how Thrift object toString() methods are generated, so keeping the format consistent makes grep'ing the logs much easier. Note on HIVE_LOCKS.hl_last_heartbeat. For locks that are part of transaction, we set this 0 (would rather set it to NULL but Currently the DB schema has this NOT NULL) and only update/read heartbeat from corresponding transaction in TXNS.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	TxnStore.MUTEX_KEY
static interface	TxnStore.MutexAPI This is primarily designed to provide coarse grained mutex support to operations running inside the Metastore (of which there could be several instances).

Field Summary

Fields

Modifier and Type	Field and Description
static String	ATTEMPTED_RESPONSE
static String	CLEANING_RESPONSE
static String	FAILED_RESPONSE
static String	INITIATED_RESPONSE
static String	SUCCEEDED_RESPONSE
static int	TIMED_OUT_TXN_ABORT_BATCH_SIZE
static String	WORKING_RESPONSE

Method Summary

Modifier and Type	Method and Description
void	abortTxn(AbortTxnRequest rqst) Abort (rollback) a transaction.
void	abortTxns(AbortTxnsRequest rqst) Abort (rollback) a list of transactions in one request.
void	addDynamicPartitions(AddDynamicPartitions rqst) Add information on a set of dynamic partitions that participated in a transaction.
boolean	checkFailedCompactions(CompactionInfo ci) Determine if there are enough consecutive failures compacting a table or partition that no new automatic compactions should be scheduled.
LockResponse	checkLock(CheckLockRequest rqst) Check whether a lock has been obtained.
void	cleanEmptyAbortedTxns() Clean up aborted transactions from txns that have no components in txn_components.
void	cleanupRecords(HiveObjectType type, Database db, Table table, Iterator<Partition> partitionIterator) Clean up corresponding records in metastore tables
void	commitTxn(CommitTxnRequest rqst) Commit a transaction
CompactionResponse	compact(CompactionRequest rqst) Submit a compaction request into the queue.
void	countOpenTxns() Get the count for open transactions.
List<String>	findColumnsWithStats(CompactionInfo ci) Queries metastore DB directly to find columns in the table which have statistics information.
CompactionInfo	findNextToCompact(String workerId) This will grab the next compaction request off of the queue, and assign it to the worker.
Set<CompactionInfo>	findPotentialCompactions(int maxAborted) This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction.
List<CompactionInfo>	findReadyToClean() Find entries in the queue that are ready to be cleaned.
TxnStore.MutexAPI	getMutexAPI()
GetOpenTxnsResponse	getOpenTxns() Get list of valid transactions.
GetOpenTxnsInfoResponse	getOpenTxnsInfo() Get information about open transactions.
void	heartbeat(HeartbeatRequest ids) Send a heartbeat for a lock or a transaction
HeartbeatTxnRangeResponse	heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) Heartbeat a group of transactions together
LockResponse	lock(LockRequest rqst) Obtain a lock.
void	markCleaned(CompactionInfo info) This will remove an entry from the queue after it has been compacted.
void	markCompacted(CompactionInfo info) This will mark an entry in the queue as compacted and put it in the ready to clean state.
void	markFailed(CompactionInfo info) Mark a compaction entry as failed.
int	numLocksInLockTable()
OpenTxnsResponse	openTxns(OpenTxnRequest rqst) Open a set of transactions
void	performTimeOuts() Timeout transactions and/or locks.
void	performWriteSetGC() WriteSet tracking is used to ensure proper transaction isolation.
void	purgeCompactionHistory() For any given compactable entity (partition, table if not partitioned) the history of compactions may look like "sssfffaaasffss", for example.
void	revokeFromLocalWorkers(String hostname) This will take all entries assigned to workers on a host return them to INITIATED state.
void	revokeTimedoutWorkers(long timeout) This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state.
void	setCompactionHighestTxnId(CompactionInfo ci, long highestTxnId) Record the highest txn id that the ci compaction job will pay attention to.
void	setConf(HiveConf conf)
void	setHadoopJobId(String hadoopJobId, long id) Once a ThreadPoolExecutor.Worker submits a job to the cluster, it calls this to update the metadata.
void	setRunAs(long cq_id, String user) Sets the user to run as.
long	setTimeout(long milliseconds)
ShowCompactResponse	showCompact(ShowCompactRequest rqst) Show list of current compactions
ShowLocksResponse	showLocks(ShowLocksRequest rqst) Get information on current locks.
void	unlock(UnlockRequest rqst) Unlock a lock.

Field Detail

INITIATED_RESPONSE

static final String INITIATED_RESPONSE

See Also:

Constant Field Values

WORKING_RESPONSE

static final String WORKING_RESPONSE

See Also:

Constant Field Values

CLEANING_RESPONSE

static final String CLEANING_RESPONSE

See Also:

Constant Field Values

FAILED_RESPONSE

static final String FAILED_RESPONSE

See Also:

Constant Field Values

SUCCEEDED_RESPONSE

static final String SUCCEEDED_RESPONSE

See Also:

Constant Field Values

ATTEMPTED_RESPONSE

static final String ATTEMPTED_RESPONSE

See Also:

Constant Field Values

TIMED_OUT_TXN_ABORT_BATCH_SIZE

static final int TIMED_OUT_TXN_ABORT_BATCH_SIZE

See Also:

Constant Field Values

Method Detail

setConf

void setConf(HiveConf conf)

getOpenTxnsInfo

GetOpenTxnsInfoResponse getOpenTxnsInfo()
                                 throws MetaException

Get information about open transactions. This gives extensive information about the transactions rather than just the list of transactions. This should be used when the need is to see information about the transactions (e.g. show transactions).

Returns:

information about open transactions

Throws:

MetaException

getOpenTxns

GetOpenTxnsResponse getOpenTxns()
                         throws MetaException

Get list of valid transactions. This gives just the list of transactions that are open.

Returns:

list of open transactions, as well as a high water mark.

Throws:

MetaException

countOpenTxns

void countOpenTxns()
            throws MetaException

Get the count for open transactions.

Throws:

MetaException

openTxns

OpenTxnsResponse openTxns(OpenTxnRequest rqst)
                   throws MetaException

Open a set of transactions

Parameters:

rqst - request to open transactions

Returns:

information on opened transactions

Throws:

MetaException

abortTxn

void abortTxn(AbortTxnRequest rqst)
       throws NoSuchTxnException,
              MetaException,
              TxnAbortedException

Abort (rollback) a transaction.

Parameters:

rqst - info on transaction to abort

Throws:

NoSuchTxnException

MetaException

TxnAbortedException

abortTxns

void abortTxns(AbortTxnsRequest rqst)
        throws NoSuchTxnException,
               MetaException

Abort (rollback) a list of transactions in one request.

Parameters:

rqst - info on transactions to abort

Throws:

NoSuchTxnException

MetaException

commitTxn

void commitTxn(CommitTxnRequest rqst)
        throws NoSuchTxnException,
               TxnAbortedException,
               MetaException

Commit a transaction

Parameters:

rqst - info on transaction to commit

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

lock

LockResponse lock(LockRequest rqst)
           throws NoSuchTxnException,
                  TxnAbortedException,
                  MetaException

Obtain a lock.

Parameters:

rqst - information on the lock to obtain. If the requester is part of a transaction the txn information must be included in the lock request.

Returns:

info on the lock, including whether it was obtained.

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

checkLock

LockResponse checkLock(CheckLockRequest rqst)
                throws NoSuchTxnException,
                       NoSuchLockException,
                       TxnAbortedException,
                       MetaException

Check whether a lock has been obtained. This is used after lock(org.apache.hadoop.hive.metastore.api.LockRequest) returned a wait state.

Parameters:

rqst - info on the lock to check

Returns:

info on the state of the lock

Throws:

NoSuchTxnException

NoSuchLockException

TxnAbortedException

MetaException

unlock

void unlock(UnlockRequest rqst)
     throws NoSuchLockException,
            TxnOpenException,
            MetaException

Unlock a lock. It is not legal to call this if the caller is part of a txn. In that case the txn should be committed or aborted instead. (Note someday this will change since multi-statement transactions will allow unlocking in the transaction.)

Parameters:

rqst - lock to unlock

Throws:

NoSuchLockException

TxnOpenException

MetaException

showLocks

ShowLocksResponse showLocks(ShowLocksRequest rqst)
                     throws MetaException

Get information on current locks.

Parameters:

rqst - lock information to retrieve

Returns:

lock information.

Throws:

MetaException

heartbeat

void heartbeat(HeartbeatRequest ids)
        throws NoSuchTxnException,
               NoSuchLockException,
               TxnAbortedException,
               MetaException

Send a heartbeat for a lock or a transaction

Parameters:

ids - lock and/or txn id to heartbeat

Throws:

NoSuchTxnException

NoSuchLockException

TxnAbortedException

MetaException

heartbeatTxnRange

HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
                                     throws MetaException

Heartbeat a group of transactions together

Parameters:

rqst - set of transactions to heartbat

Returns:

info on txns that were heartbeated

Throws:

MetaException

compact

CompactionResponse compact(CompactionRequest rqst)
                    throws MetaException

Submit a compaction request into the queue. This is called when a user manually requests a compaction.

Parameters:

rqst - information on what to compact

Returns:

id of the compaction that has been started or existing id if this resource is already scheduled

Throws:

MetaException

showCompact

ShowCompactResponse showCompact(ShowCompactRequest rqst)
                         throws MetaException

Show list of current compactions

Parameters:

rqst - info on which compactions to show

Returns:

compaction information

Throws:

MetaException

addDynamicPartitions

void addDynamicPartitions(AddDynamicPartitions rqst)
                   throws NoSuchTxnException,
                          TxnAbortedException,
                          MetaException

Add information on a set of dynamic partitions that participated in a transaction.

Parameters:

rqst - dynamic partition info.

Throws:

NoSuchTxnException

TxnAbortedException

MetaException

cleanupRecords

void cleanupRecords(HiveObjectType type,
                    Database db,
                    Table table,
                    Iterator<Partition> partitionIterator)
             throws MetaException

Clean up corresponding records in metastore tables

Parameters:

type - Hive object type

db - database object

table - table object

partitionIterator - partition iterator

Throws:

MetaException

performTimeOuts

void performTimeOuts()

Timeout transactions and/or locks. This should only be called by the compactor.

findPotentialCompactions

Set<CompactionInfo> findPotentialCompactions(int maxAborted)
                                      throws MetaException

This will look through the completed_txn_components table and look for partitions or tables that may be ready for compaction. Also, look through txns and txn_components tables for aborted transactions that we should add to the list.

Parameters:

maxAborted - Maximum number of aborted queries to allow before marking this as a potential compaction.

Returns:

list of CompactionInfo structs. These will not have id, type, or runAs set since these are only potential compactions not actual ones.

Throws:

MetaException

setRunAs

void setRunAs(long cq_id,
              String user)
       throws MetaException

Sets the user to run as. This is for the case where the request was generated by the user and so the worker must set this value later.

Parameters:

cq_id - id of this entry in the queue

user - user to run the jobs as

Throws:

MetaException

findNextToCompact

CompactionInfo findNextToCompact(String workerId)
                          throws MetaException

This will grab the next compaction request off of the queue, and assign it to the worker.

Parameters:

workerId - id of the worker calling this, will be recorded in the db

Returns:

an info element for this compaction request, or null if there is no work to do now.

Throws:

MetaException

markCompacted

void markCompacted(CompactionInfo info)
            throws MetaException

This will mark an entry in the queue as compacted and put it in the ready to clean state.

Parameters:

info - info on the compaction entry to mark as compacted.

Throws:

MetaException

findReadyToClean

List<CompactionInfo> findReadyToClean()
                               throws MetaException

Find entries in the queue that are ready to be cleaned.

Returns:

information on the entry in the queue.

Throws:

MetaException

markCleaned

void markCleaned(CompactionInfo info)
          throws MetaException

This will remove an entry from the queue after it has been compacted.

Parameters:

info - info on the compaction entry to remove

Throws:

MetaException

markFailed

void markFailed(CompactionInfo info)
         throws MetaException

Mark a compaction entry as failed. This will move it to the compaction history queue with a failed status. It will NOT clean up aborted transactions in the table/partition associated with this compaction.

Parameters:

info - information on the compaction that failed.

Throws:

MetaException

cleanEmptyAbortedTxns

void cleanEmptyAbortedTxns()
                    throws MetaException

Clean up aborted transactions from txns that have no components in txn_components. The reson such txns exist can be that now work was done in this txn (e.g. Streaming opened TransactionBatch and abandoned it w/o doing any work) or due to markCleaned(CompactionInfo) being called.

Throws:

MetaException

revokeFromLocalWorkers

void revokeFromLocalWorkers(String hostname)
                     throws MetaException

This will take all entries assigned to workers on a host return them to INITIATED state. The initiator should use this at start up to clean entries from any workers that were in the middle of compacting when the metastore shutdown. It does not reset entries from worker threads on other hosts as those may still be working.

Parameters:

hostname - Name of this host. It is assumed this prefixes the thread's worker id, so that like hostname% will match the worker id.

Throws:

MetaException

revokeTimedoutWorkers

void revokeTimedoutWorkers(long timeout)
                    throws MetaException

This call will return all compaction queue entries assigned to a worker but over the timeout back to the initiated state. This should be called by the initiator on start up and occasionally when running to clean up after dead threads. At start up revokeFromLocalWorkers(String) should be called first.

Parameters:

timeout - number of milliseconds since start time that should elapse before a worker is declared dead.

Throws:

MetaException

findColumnsWithStats

List<String> findColumnsWithStats(CompactionInfo ci)
                           throws MetaException

Queries metastore DB directly to find columns in the table which have statistics information. If ci includes partition info then per partition stats info is examined, otherwise table level stats are examined.

Throws:

MetaException

setCompactionHighestTxnId

void setCompactionHighestTxnId(CompactionInfo ci,
                               long highestTxnId)
                        throws MetaException

Record the highest txn id that the ci compaction job will pay attention to.

Throws:

MetaException

purgeCompactionHistory

void purgeCompactionHistory()
                     throws MetaException

For any given compactable entity (partition, table if not partitioned) the history of compactions may look like "sssfffaaasffss", for example. The idea is to retain the tail (most recent) of the history such that a configurable number of each type of state is present. Any other entries can be purged. This scheme has advantage of always retaining the last failure/success even if it's not recent.

Throws:

MetaException

performWriteSetGC

void performWriteSetGC()

WriteSet tracking is used to ensure proper transaction isolation. This method deletes the transaction metadata once it becomes unnecessary.

checkFailedCompactions

boolean checkFailedCompactions(CompactionInfo ci)
                        throws MetaException

Determine if there are enough consecutive failures compacting a table or partition that no new automatic compactions should be scheduled. User initiated compactions do not do this check.

Parameters:

ci - Table or partition to check.

Returns:

true if it is ok to compact, false if there have been too many failures.

Throws:

MetaException

numLocksInLockTable

int numLocksInLockTable()
                 throws SQLException,
                        MetaException

Throws:

SQLException

MetaException

setTimeout

long setTimeout(long milliseconds)

getMutexAPI

TxnStore.MutexAPI getMutexAPI()

setHadoopJobId

void setHadoopJobId(String hadoopJobId,
                    long id)

Once a ThreadPoolExecutor.Worker submits a job to the cluster, it calls this to update the metadata.

Parameters:

id - CompactionInfo.id