org.apache.hadoop.hive.shims

Class HadoopShimsSecure

java.lang.Object

org.apache.hadoop.hive.shims.HadoopShimsSecure

All Implemented Interfaces:

HadoopShims

Direct Known Subclasses:

Hadoop20SShims, Hadoop23Shims

public abstract class HadoopShimsSecure
extends Object
implements HadoopShims

Base implemention for shims against secure Hadoop 0.20.3/0.23.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HadoopShimsSecure.CombineFileInputFormatShim<K,V>
static class	HadoopShimsSecure.CombineFileRecordReader<K,V>
static class	HadoopShimsSecure.InputSplitShim

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.shims.HadoopShims

HadoopShims.ByteBufferPoolShim, HadoopShims.DirectCompressionType, HadoopShims.DirectDecompressorShim, HadoopShims.HCatHadoopShims, HadoopShims.HdfsEncryptionShim, HadoopShims.HdfsFileStatus, HadoopShims.JobTrackerState, HadoopShims.KerberosNameShim, HadoopShims.MiniDFSShim, HadoopShims.MiniMrShim, HadoopShims.NoopHdfsEncryptionShim, HadoopShims.StoragePolicyShim, HadoopShims.StoragePolicyValue, HadoopShims.WebHCatJTShim, HadoopShims.ZeroCopyReaderShim

Constructor Summary

Constructors

Constructor and Description
HadoopShimsSecure()

Method Summary

Methods

Modifier and Type	Method and Description
abstract void	addDelegationTokens(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.security.Credentials cred, String uname) Get Delegation token and add it to Credential.
void	checkFileAccess(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus stat, org.apache.hadoop.fs.permission.FsAction action) Check if the configured UGI has access to the path for the given file system action.
abstract org.apache.hadoop.fs.FileSystem	createProxyFileSystem(org.apache.hadoop.fs.FileSystem fs, URI uri) Create a proxy file system that can serve a given scheme/authority using some other file system.
abstract long	getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Get the default block size for the path.
abstract short	getDefaultReplication(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Get the default replication for a path.
abstract String	getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf) All references to jobtracker/resource manager http address in the configuration should be done through this shim
abstract String	getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf) All retrieval of jobtracker/resource manager rpc address in the configuration should be done through this shim
abstract HadoopShims.JobTrackerState	getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus) Convert the ClusterStatus to its Thrift equivalent: JobTrackerState.
abstract org.apache.hadoop.fs.FileSystem	getNonCachedFileSystem(URI uri, org.apache.hadoop.conf.Configuration conf)
abstract boolean	isLocalMode(org.apache.hadoop.conf.Configuration conf) Check wether MR is configured to run in local-mode
abstract boolean	moveToAppropriateTrash(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Move the directory/file to trash.
abstract org.apache.hadoop.mapreduce.JobContext	newJobContext(org.apache.hadoop.mapreduce.Job job)
abstract org.apache.hadoop.mapreduce.TaskAttemptContext	newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.util.Progressable progressable)
protected void	run(org.apache.hadoop.fs.FsShell shell, String[] command)
abstract void	setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf, String val) All updates to jobtracker/resource manager rpc address in the configuration should be done through this shim

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.shims.HadoopShims

createHdfsEncryptionShim, getCombineFileInputFormat, getConfiguration, getCurrentTrashPath, getDirectDecompressor, getFullFileStatus, getHadoopConfNames, getHCatShim, getJobConf, getKerberosNameShim, getLocations, getLocationsWithOffset, getLongComparator, getMergedCredentials, getMiniDfs, getMiniMrCluster, getMiniSparkCluster, getMiniTezCluster, getPassword, getPathWithoutSchemeAndAuthority, getStoragePolicyShim, getTaskAttemptLogUrl, getWebHCatShim, getZeroCopyReader, hasStickyBit, hflush, isDirectory, listLocatedStatus, mergeCredentials, newTaskAttemptID, readByteBuffer, refreshDefaultQueue, runDistCp, setFullFileStatus, setTotalOrderPartitionFile, startPauseMonitor, supportStickyBit, supportTrashFeature

Constructor Detail

HadoopShimsSecure

public HadoopShimsSecure()

Method Detail

getJobTrackerState

public abstract HadoopShims.JobTrackerState getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus)
                                                        throws Exception

Description copied from interface: HadoopShims

Convert the ClusterStatus to its Thrift equivalent: JobTrackerState. See MAPREDUCE-2455 for why this is a part of the shim.

Specified by:

getJobTrackerState in interface HadoopShims

Returns:

the matching JobTrackerState

Throws:

Exception - if no equivalent JobTrackerState exists

newTaskAttemptContext

public abstract org.apache.hadoop.mapreduce.TaskAttemptContext newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf,
                                                                   org.apache.hadoop.util.Progressable progressable)

Specified by:

newTaskAttemptContext in interface HadoopShims

newJobContext

public abstract org.apache.hadoop.mapreduce.JobContext newJobContext(org.apache.hadoop.mapreduce.Job job)

Specified by:

newJobContext in interface HadoopShims

isLocalMode

public abstract boolean isLocalMode(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: HadoopShims

Check wether MR is configured to run in local-mode

Specified by:

isLocalMode in interface HadoopShims

Returns:

setJobLauncherRpcAddress

public abstract void setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf,
                            String val)

Description copied from interface: HadoopShims

All updates to jobtracker/resource manager rpc address in the configuration should be done through this shim

Specified by:

setJobLauncherRpcAddress in interface HadoopShims

getJobLauncherHttpAddress

public abstract String getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: HadoopShims

All references to jobtracker/resource manager http address in the configuration should be done through this shim

Specified by:

getJobLauncherHttpAddress in interface HadoopShims

Returns:

getJobLauncherRpcAddress

public abstract String getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: HadoopShims

All retrieval of jobtracker/resource manager rpc address in the configuration should be done through this shim

Specified by:

getJobLauncherRpcAddress in interface HadoopShims

Returns:

getDefaultReplication

public abstract short getDefaultReplication(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path path)

Description copied from interface: HadoopShims

Get the default replication for a path. In case of CSMT the given path will be used to locate the actual filesystem.

Specified by:

getDefaultReplication in interface HadoopShims

Returns:

getDefaultBlockSize

public abstract long getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.fs.Path path)

Description copied from interface: HadoopShims

Get the default block size for the path. FileSystem alone is not sufficient to determine the same, as in case of CSMT the underlying file system determines that.

Specified by:

getDefaultBlockSize in interface HadoopShims

Returns:

moveToAppropriateTrash

public abstract boolean moveToAppropriateTrash(org.apache.hadoop.fs.FileSystem fs,
                             org.apache.hadoop.fs.Path path,
                             org.apache.hadoop.conf.Configuration conf)
                                        throws IOException

Description copied from interface: HadoopShims

Move the directory/file to trash. In case of the symlinks or mount points, the file is moved to the trashbin in the actual volume of the path p being deleted

Specified by:

moveToAppropriateTrash in interface HadoopShims

Returns:

false if the item is already in the trash or trash is disabled

Throws:

IOException

createProxyFileSystem

public abstract org.apache.hadoop.fs.FileSystem createProxyFileSystem(org.apache.hadoop.fs.FileSystem fs,
                                                    URI uri)

Description copied from interface: HadoopShims

Create a proxy file system that can serve a given scheme/authority using some other file system.

Specified by:

createProxyFileSystem in interface HadoopShims

getNonCachedFileSystem

public abstract org.apache.hadoop.fs.FileSystem getNonCachedFileSystem(URI uri,
                                                     org.apache.hadoop.conf.Configuration conf)
                                                                throws IOException

Specified by:

getNonCachedFileSystem in interface HadoopShims

Throws:

IOException

run

protected void run(org.apache.hadoop.fs.FsShell shell,
       String[] command)
            throws Exception

Throws:

Exception

checkFileAccess

public void checkFileAccess(org.apache.hadoop.fs.FileSystem fs,
                   org.apache.hadoop.fs.FileStatus stat,
                   org.apache.hadoop.fs.permission.FsAction action)
                     throws IOException,
                            AccessControlException,
                            Exception

Description copied from interface: HadoopShims

Check if the configured UGI has access to the path for the given file system action. Method will return successfully if action is permitted. AccessControlExceptoin will be thrown if user does not have access to perform the action. Other exceptions may be thrown for non-access related errors.

Specified by:

checkFileAccess in interface HadoopShims

Throws:

IOException

AccessControlException

Exception

addDelegationTokens

public abstract void addDelegationTokens(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.security.Credentials cred,
                       String uname)
                                  throws IOException

Description copied from interface: HadoopShims

Get Delegation token and add it to Credential.

Specified by:

addDelegationTokens in interface HadoopShims

Parameters:

fs - FileSystem object to HDFS

cred - Credentials object to add the token to.

uname - user name.

Throws:

IOException - If an error occurred on adding the token.