org.apache.hadoop.hive.thrift

Class TokenStoreDelegationTokenSecretManager

java.lang.Object

org.apache.hadoop.security.token.SecretManager<TokenIdent>

org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

org.apache.hadoop.hive.thrift.DelegationTokenSecretManager

org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager

public class TokenStoreDelegationTokenSecretManager
extends DelegationTokenSecretManager

Extension of DelegationTokenSecretManager to support alternative to default in-memory token management for fail-over and clustering through plug-able token store (ZooKeeper etc.). Delegation tokens will be retrieved from the store on-demand and (unlike base class behavior) not cached in memory. This avoids complexities related to token expiration. The security token is needed only at the time the transport is opened (as opposed to per interface operation). The assumption therefore is low cost of interprocess token retrieval (for random read efficient store such as ZooKeeper) compared to overhead of synchronizing per-process in-memory token caches. The wrapper incorporates the token store abstraction within the limitations of current Hive/Hadoop dependency (.20S) with minimum code duplication. Eventually this should be supported by Hadoop security directly.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	TokenStoreDelegationTokenSecretManager.ExpiredTokenRemover Cloned from AbstractDelegationTokenSecretManager to deal with private access restriction (there would not be an need to clone the remove thread if the remove logic was protected/extensible).

Nested classes/interfaces inherited from class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager

org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.DelegationTokenInformation

Nested classes/interfaces inherited from class org.apache.hadoop.security.token.SecretManager

org.apache.hadoop.security.token.SecretManager.InvalidToken

Field Summary

Fields inherited from class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager

allKeys, currentId, currentTokens, delegationTokenSequenceNumber, noInterruptsLock, running, storeTokenTrackingId

Constructor Summary

Constructors

Constructor and Description
TokenStoreDelegationTokenSecretManager(long delegationKeyUpdateInterval, long delegationTokenMaxLifetime, long delegationTokenRenewInterval, long delegationTokenRemoverScanInterval, DelegationTokenStore sharedStore)

Method Summary

Methods

Modifier and Type	Method and Description
DelegationTokenIdentifier	cancelToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token, String canceller)
protected byte[]	createPassword(DelegationTokenIdentifier id) Create the password and add it to shared store.
static void	decodeWritable(org.apache.hadoop.io.Writable w, String idStr)
static String	encodeWritable(org.apache.hadoop.io.Writable key)
protected DelegationTokenIdentifier	getTokenIdentifier(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
protected void	logUpdateMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey key) Synchronize master key updates / sequence generation for multiple nodes.
protected Map<Integer,org.apache.hadoop.security.token.delegation.DelegationKey>	reloadKeys()
protected void	removeExpiredTokens() Remove expired tokens.
long	renewToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token, String renewer)
byte[]	retrievePassword(DelegationTokenIdentifier identifier)
protected void	rollMasterKeyExt() Extension of rollMasterKey to remove expired keys from store.
void	startThreads()
void	stopThreads()

Methods inherited from class org.apache.hadoop.hive.thrift.DelegationTokenSecretManager

cancelDelegationToken, createIdentifier, getDelegationToken, getUserFromToken, renewDelegationToken

Methods inherited from class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager

addKey, addPersistedDelegationToken, checkToken, createSecretKey, decodeTokenIdentifier, getAllKeys, getCurrentKeyId, getDelegationKey, getDelegationTokenSeqNum, getTokenInfo, getTokenTrackingId, getTrackingIdIfEnabled, incrementCurrentKeyId, incrementDelegationTokenSeqNum, isRunning, logExpireToken, removeStoredMasterKey, removeStoredToken, reset, setCurrentKeyId, setDelegationTokenSeqNum, storeDelegationKey, storeNewMasterKey, storeNewToken, storeToken, updateDelegationKey, updateStoredToken, updateToken, verifyToken

Methods inherited from class org.apache.hadoop.security.token.SecretManager

checkAvailableForRead, createPassword, generateSecret, retriableRetrievePassword

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenStoreDelegationTokenSecretManager

public TokenStoreDelegationTokenSecretManager(long delegationKeyUpdateInterval,
                                      long delegationTokenMaxLifetime,
                                      long delegationTokenRenewInterval,
                                      long delegationTokenRemoverScanInterval,
                                      DelegationTokenStore sharedStore)

Method Detail

getTokenIdentifier

protected DelegationTokenIdentifier getTokenIdentifier(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                                                throws IOException

Throws:

IOException

reloadKeys

protected Map<Integer,org.apache.hadoop.security.token.delegation.DelegationKey> reloadKeys()

retrievePassword

public byte[] retrievePassword(DelegationTokenIdentifier identifier)
                        throws org.apache.hadoop.security.token.SecretManager.InvalidToken

Overrides:

retrievePassword in class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

Throws:

org.apache.hadoop.security.token.SecretManager.InvalidToken

cancelToken

public DelegationTokenIdentifier cancelToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token,
                                    String canceller)
                                      throws IOException

Overrides:

cancelToken in class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

Throws:

IOException

createPassword

protected byte[] createPassword(DelegationTokenIdentifier id)

Create the password and add it to shared store.

Overrides:

createPassword in class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

renewToken

public long renewToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token,
              String renewer)
                throws org.apache.hadoop.security.token.SecretManager.InvalidToken,
                       IOException

Overrides:

renewToken in class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

Throws:

org.apache.hadoop.security.token.SecretManager.InvalidToken

IOException

encodeWritable

public static String encodeWritable(org.apache.hadoop.io.Writable key)
                             throws IOException

Throws:

IOException

decodeWritable

public static void decodeWritable(org.apache.hadoop.io.Writable w,
                  String idStr)
                           throws IOException

Throws:

IOException

logUpdateMasterKey

protected void logUpdateMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey key)
                           throws IOException

Synchronize master key updates / sequence generation for multiple nodes. NOTE: keeps currentKey private, so we need to utilize this "hook" to manipulate the key through the object reference. This .20S workaround should cease to exist when Hadoop supports token store.

Overrides:

logUpdateMasterKey in class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

Throws:

IOException

startThreads

public void startThreads()
                  throws IOException

Overrides:

startThreads in class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

Throws:

IOException

stopThreads

public void stopThreads()

Overrides:

stopThreads in class org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager<DelegationTokenIdentifier>

removeExpiredTokens

protected void removeExpiredTokens()

Remove expired tokens. Replaces logic in AbstractDelegationTokenSecretManager that cannot be reused due to private method access. Logic here can more efficiently deal with external token store by only loading into memory the minimum data needed.

rollMasterKeyExt

protected void rollMasterKeyExt()
                         throws IOException

Extension of rollMasterKey to remove expired keys from store.

Throws:

IOException