org.apache.hadoop.hive.shims

Interface HadoopShims

All Known Implementing Classes:

Hadoop20SShims, Hadoop23Shims, HadoopShimsSecure

public interface HadoopShims

In order to be compatible with multiple versions of Hadoop, all parts of the Hadoop interface that are not cross-version compatible are encapsulated in an implementation of this class. Users should use the ShimLoader class as a factory to obtain an implementation of HadoopShims corresponding to the version of Hadoop currently on the classpath.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	HadoopShims.ByteBufferPoolShim a hadoop.io ByteBufferPool shim.
static interface	HadoopShims.CombineFileInputFormatShim<K,V> CombineFileInputFormatShim.
static class	HadoopShims.DirectCompressionType
static interface	HadoopShims.DirectDecompressorShim
static interface	HadoopShims.HCatHadoopShims
static interface	HadoopShims.HdfsEncryptionShim This interface encapsulates methods used to get encryption information from HDFS paths.
static interface	HadoopShims.HdfsFileStatus Includes the vanilla FileStatus, and AclStatus if it applies to this version of hadoop.
static class	HadoopShims.JobTrackerState
static interface	HadoopShims.KerberosNameShim Shim for KerberosName
static interface	HadoopShims.MiniDFSShim Shim around the functions in MiniDFSCluster that Hive uses.
static interface	HadoopShims.MiniMrShim Shim for MiniMrCluster
static class	HadoopShims.NoopHdfsEncryptionShim This is a dummy class used when the hadoop version does not support hdfs encryption.
static interface	HadoopShims.StoragePolicyShim
static class	HadoopShims.StoragePolicyValue Create a shim for DFS storage policy.
static interface	HadoopShims.WebHCatJTShim
static interface	HadoopShims.ZeroCopyReaderShim

Method Summary

Methods

Modifier and Type	Method and Description
void	addDelegationTokens(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.security.Credentials cred, String uname) Get Delegation token and add it to Credential.
void	checkFileAccess(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus status, org.apache.hadoop.fs.permission.FsAction action) Check if the configured UGI has access to the path for the given file system action.
HadoopShims.HdfsEncryptionShim	createHdfsEncryptionShim(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf) Returns a new instance of the HdfsEncryption shim.
org.apache.hadoop.fs.FileSystem	createProxyFileSystem(org.apache.hadoop.fs.FileSystem fs, URI uri) Create a proxy file system that can serve a given scheme/authority using some other file system.
HadoopShims.CombineFileInputFormatShim	getCombineFileInputFormat()
org.apache.hadoop.conf.Configuration	getConfiguration(org.apache.hadoop.mapreduce.JobContext context) Get configuration from JobContext
org.apache.hadoop.fs.Path	getCurrentTrashPath(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs)
long	getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Get the default block size for the path.
short	getDefaultReplication(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Get the default replication for a path.
HadoopShims.DirectDecompressorShim	getDirectDecompressor(HadoopShims.DirectCompressionType codec)
HadoopShims.HdfsFileStatus	getFullFileStatus(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path file) For a given file, return a file status
Map<String,String>	getHadoopConfNames()
HadoopShims.HCatHadoopShims	getHCatShim()
org.apache.hadoop.mapred.JobConf	getJobConf(org.apache.hadoop.mapred.JobContext context) Get job conf from the old style JobContext.
String	getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf) All references to jobtracker/resource manager http address in the configuration should be done through this shim
String	getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf) All retrieval of jobtracker/resource manager rpc address in the configuration should be done through this shim
HadoopShims.JobTrackerState	getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus) Convert the ClusterStatus to its Thrift equivalent: JobTrackerState.
HadoopShims.KerberosNameShim	getKerberosNameShim(String name) Returns a shim to wrap KerberosName
org.apache.hadoop.fs.BlockLocation[]	getLocations(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus status) For file status returned by listLocatedStatus, convert them into a list of block locations.
TreeMap<Long,org.apache.hadoop.fs.BlockLocation>	getLocationsWithOffset(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus status) For the block locations returned by getLocations() convert them into a Treemap by iterating over the list of blockLocation.
Comparator<org.apache.hadoop.io.LongWritable>	getLongComparator()
void	getMergedCredentials(org.apache.hadoop.mapred.JobConf jobConf)
HadoopShims.MiniDFSShim	getMiniDfs(org.apache.hadoop.conf.Configuration conf, int numDataNodes, boolean format, String[] racks) Returns a shim to wrap MiniDFSCluster.
HadoopShims.MiniMrShim	getMiniMrCluster(org.apache.hadoop.conf.Configuration conf, int numberOfTaskTrackers, String nameNode, int numDir) Returns a shim to wrap MiniMrCluster
HadoopShims.MiniMrShim	getMiniSparkCluster(org.apache.hadoop.conf.Configuration conf, int numberOfTaskTrackers, String nameNode, int numDir)
HadoopShims.MiniMrShim	getMiniTezCluster(org.apache.hadoop.conf.Configuration conf, int numberOfTaskTrackers, String nameNode, int numDir)
org.apache.hadoop.fs.FileSystem	getNonCachedFileSystem(URI uri, org.apache.hadoop.conf.Configuration conf)
String	getPassword(org.apache.hadoop.conf.Configuration conf, String name) Use password API (if available) to fetch credentials/password
org.apache.hadoop.fs.Path	getPathWithoutSchemeAndAuthority(org.apache.hadoop.fs.Path path)
HadoopShims.StoragePolicyShim	getStoragePolicyShim(org.apache.hadoop.fs.FileSystem fs) obtain a storage policy shim associated with the filesystem.
String	getTaskAttemptLogUrl(org.apache.hadoop.mapred.JobConf conf, String taskTrackerHttpAddress, String taskAttemptId) Constructs and Returns TaskAttempt Log Url or null if the TaskLogServlet is not available
HadoopShims.WebHCatJTShim	getWebHCatShim(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.security.UserGroupInformation ugi) Provides a Hadoop JobTracker shim.
HadoopShims.ZeroCopyReaderShim	getZeroCopyReader(org.apache.hadoop.fs.FSDataInputStream in, HadoopShims.ByteBufferPoolShim pool) Provides an HDFS ZeroCopyReader shim.
boolean	hasStickyBit(org.apache.hadoop.fs.permission.FsPermission permission) Check stick bit in the permission
void	hflush(org.apache.hadoop.fs.FSDataOutputStream stream) Flush and make visible to other users the changes to the given stream.
boolean	isDirectory(org.apache.hadoop.fs.FileStatus fileStatus) Check whether file is directory.
boolean	isLocalMode(org.apache.hadoop.conf.Configuration conf) Check wether MR is configured to run in local-mode
List<org.apache.hadoop.fs.FileStatus>	listLocatedStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter filter) Get the block locations for the given directory.
void	mergeCredentials(org.apache.hadoop.mapred.JobConf dest, org.apache.hadoop.mapred.JobConf src)
boolean	moveToAppropriateTrash(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Move the directory/file to trash.
org.apache.hadoop.mapreduce.JobContext	newJobContext(org.apache.hadoop.mapreduce.Job job)
org.apache.hadoop.mapreduce.TaskAttemptContext	newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.util.Progressable progressable)
org.apache.hadoop.mapreduce.TaskAttemptID	newTaskAttemptID(org.apache.hadoop.mapreduce.JobID jobId, boolean isMap, int taskId, int id)
int	readByteBuffer(org.apache.hadoop.fs.FSDataInputStream file, ByteBuffer dest) Reads data into ByteBuffer.
void	refreshDefaultQueue(org.apache.hadoop.conf.Configuration conf, String userName) Reset the default fair scheduler queue mapping to end user.
boolean	runDistCp(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, org.apache.hadoop.conf.Configuration conf) Copies a source dir/file to a destination by orchestrating the copy between hdfs nodes.
void	setFullFileStatus(org.apache.hadoop.conf.Configuration conf, HadoopShims.HdfsFileStatus sourceStatus, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path target) For a given file, set a given file status.
void	setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf, String val) All updates to jobtracker/resource manager rpc address in the configuration should be done through this shim
void	setTotalOrderPartitionFile(org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.fs.Path partition) The method sets to set the partition file has a different signature between hadoop versions.
void	startPauseMonitor(org.apache.hadoop.conf.Configuration conf)
boolean	supportStickyBit() check whether current hadoop supports sticky bit
boolean	supportTrashFeature()

Method Detail

getTaskAttemptLogUrl

String getTaskAttemptLogUrl(org.apache.hadoop.mapred.JobConf conf,
                          String taskTrackerHttpAddress,
                          String taskAttemptId)
                            throws MalformedURLException

Constructs and Returns TaskAttempt Log Url or null if the TaskLogServlet is not available

Returns:

TaskAttempt Log Url

Throws:

MalformedURLException

getMiniMrCluster

HadoopShims.MiniMrShim getMiniMrCluster(org.apache.hadoop.conf.Configuration conf,
                                      int numberOfTaskTrackers,
                                      String nameNode,
                                      int numDir)
                                        throws IOException

Returns a shim to wrap MiniMrCluster

Throws:

IOException

getMiniTezCluster

HadoopShims.MiniMrShim getMiniTezCluster(org.apache.hadoop.conf.Configuration conf,
                                       int numberOfTaskTrackers,
                                       String nameNode,
                                       int numDir)
                                         throws IOException

Throws:

IOException

getMiniSparkCluster

HadoopShims.MiniMrShim getMiniSparkCluster(org.apache.hadoop.conf.Configuration conf,
                                         int numberOfTaskTrackers,
                                         String nameNode,
                                         int numDir)
                                           throws IOException

Throws:

IOException

getMiniDfs

HadoopShims.MiniDFSShim getMiniDfs(org.apache.hadoop.conf.Configuration conf,
                                 int numDataNodes,
                                 boolean format,
                                 String[] racks)
                                   throws IOException

Returns a shim to wrap MiniDFSCluster. This is necessary since this class was moved from org.apache.hadoop.dfs to org.apache.hadoop.hdfs

Throws:

IOException

getCombineFileInputFormat

HadoopShims.CombineFileInputFormatShim getCombineFileInputFormat()

getJobTrackerState

HadoopShims.JobTrackerState getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus)
                                               throws Exception

Convert the ClusterStatus to its Thrift equivalent: JobTrackerState. See MAPREDUCE-2455 for why this is a part of the shim.

Parameters:

clusterStatus -

Returns:

the matching JobTrackerState

Throws:

Exception - if no equivalent JobTrackerState exists

newTaskAttemptContext

org.apache.hadoop.mapreduce.TaskAttemptContext newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf,
                                                                   org.apache.hadoop.util.Progressable progressable)

newTaskAttemptID

org.apache.hadoop.mapreduce.TaskAttemptID newTaskAttemptID(org.apache.hadoop.mapreduce.JobID jobId,
                                                         boolean isMap,
                                                         int taskId,
                                                         int id)

newJobContext

org.apache.hadoop.mapreduce.JobContext newJobContext(org.apache.hadoop.mapreduce.Job job)

startPauseMonitor

void startPauseMonitor(org.apache.hadoop.conf.Configuration conf)

isLocalMode

boolean isLocalMode(org.apache.hadoop.conf.Configuration conf)

Check wether MR is configured to run in local-mode

Parameters:

conf -

Returns:

getJobLauncherRpcAddress

String getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf)

All retrieval of jobtracker/resource manager rpc address in the configuration should be done through this shim

Parameters:

conf -

Returns:

setJobLauncherRpcAddress

void setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf,
                            String val)

All updates to jobtracker/resource manager rpc address in the configuration should be done through this shim

Parameters:

conf -

getJobLauncherHttpAddress

String getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf)

All references to jobtracker/resource manager http address in the configuration should be done through this shim

Parameters:

conf -

Returns:

moveToAppropriateTrash

boolean moveToAppropriateTrash(org.apache.hadoop.fs.FileSystem fs,
                             org.apache.hadoop.fs.Path path,
                             org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Move the directory/file to trash. In case of the symlinks or mount points, the file is moved to the trashbin in the actual volume of the path p being deleted

Parameters:

fs -

path -

conf -

Returns:

false if the item is already in the trash or trash is disabled

Throws:

IOException

getDefaultBlockSize

long getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.fs.Path path)

Get the default block size for the path. FileSystem alone is not sufficient to determine the same, as in case of CSMT the underlying file system determines that.

Parameters:

fs -

path -

Returns:

getDefaultReplication

short getDefaultReplication(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path path)

Get the default replication for a path. In case of CSMT the given path will be used to locate the actual filesystem.

Parameters:

fs -

path -

Returns:

refreshDefaultQueue

void refreshDefaultQueue(org.apache.hadoop.conf.Configuration conf,
                       String userName)
                         throws IOException

Reset the default fair scheduler queue mapping to end user.

Parameters:

conf -

userName - end user name

Throws:

IOException

setTotalOrderPartitionFile

void setTotalOrderPartitionFile(org.apache.hadoop.mapred.JobConf jobConf,
                              org.apache.hadoop.fs.Path partition)

The method sets to set the partition file has a different signature between hadoop versions.

Parameters:

jobConf -

partition -

getLongComparator

Comparator<org.apache.hadoop.io.LongWritable> getLongComparator()

listLocatedStatus

List<org.apache.hadoop.fs.FileStatus> listLocatedStatus(org.apache.hadoop.fs.FileSystem fs,
                                                      org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.fs.PathFilter filter)
                                                        throws IOException

Get the block locations for the given directory.

Parameters:

fs - the file system

path - the directory name to get the status and block locations

filter - a filter that needs to accept the file (or null)

Returns:

an list for the located file status objects

Throws:

IOException

getLocations

org.apache.hadoop.fs.BlockLocation[] getLocations(org.apache.hadoop.fs.FileSystem fs,
                                                org.apache.hadoop.fs.FileStatus status)
                                                  throws IOException

For file status returned by listLocatedStatus, convert them into a list of block locations.

Parameters:

fs - the file system

status - the file information

Returns:

the block locations of the file

Throws:

IOException

getLocationsWithOffset

TreeMap<Long,org.apache.hadoop.fs.BlockLocation> getLocationsWithOffset(org.apache.hadoop.fs.FileSystem fs,
                                                                      org.apache.hadoop.fs.FileStatus status)
                                                                        throws IOException

For the block locations returned by getLocations() convert them into a Treemap by iterating over the list of blockLocation. Using TreeMap from offset to blockLocation, makes it O(logn) to get a particular block based upon offset.

Parameters:

fs - the file system

status - the file information

Returns:

TreeMap

Throws:

IOException

hflush

void hflush(org.apache.hadoop.fs.FSDataOutputStream stream)
            throws IOException

Flush and make visible to other users the changes to the given stream.

Parameters:

stream - the stream to hflush.

Throws:

IOException

getFullFileStatus

HadoopShims.HdfsFileStatus getFullFileStatus(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.fs.FileSystem fs,
                                           org.apache.hadoop.fs.Path file)
                                             throws IOException

For a given file, return a file status

Parameters:

conf -

fs -

file -

Returns:

Throws:

IOException

setFullFileStatus

void setFullFileStatus(org.apache.hadoop.conf.Configuration conf,
                     HadoopShims.HdfsFileStatus sourceStatus,
                     org.apache.hadoop.fs.FileSystem fs,
                     org.apache.hadoop.fs.Path target)
                       throws IOException

For a given file, set a given file status.

Parameters:

conf -

sourceStatus -

fs -

target -

Throws:

IOException

getHCatShim

HadoopShims.HCatHadoopShims getHCatShim()

getWebHCatShim

HadoopShims.WebHCatJTShim getWebHCatShim(org.apache.hadoop.conf.Configuration conf,
                                       org.apache.hadoop.security.UserGroupInformation ugi)
                                         throws IOException

Provides a Hadoop JobTracker shim.

Parameters:

conf - not null

Throws:

IOException

createProxyFileSystem

org.apache.hadoop.fs.FileSystem createProxyFileSystem(org.apache.hadoop.fs.FileSystem fs,
                                                    URI uri)

Create a proxy file system that can serve a given scheme/authority using some other file system.

getHadoopConfNames

Map<String,String> getHadoopConfNames()

getStoragePolicyShim

HadoopShims.StoragePolicyShim getStoragePolicyShim(org.apache.hadoop.fs.FileSystem fs)

obtain a storage policy shim associated with the filesystem. Returns null when the filesystem has no storage policies.

getZeroCopyReader

HadoopShims.ZeroCopyReaderShim getZeroCopyReader(org.apache.hadoop.fs.FSDataInputStream in,
                                               HadoopShims.ByteBufferPoolShim pool)
                                                 throws IOException

Provides an HDFS ZeroCopyReader shim.

Parameters:

in - FSDataInputStream to read from (where the cached/mmap buffers are tied to)

in - ByteBufferPoolShim to allocate fallback buffers with

Returns:

returns null if not supported

Throws:

IOException

getDirectDecompressor

HadoopShims.DirectDecompressorShim getDirectDecompressor(HadoopShims.DirectCompressionType codec)

getConfiguration

org.apache.hadoop.conf.Configuration getConfiguration(org.apache.hadoop.mapreduce.JobContext context)

Get configuration from JobContext

getJobConf

org.apache.hadoop.mapred.JobConf getJobConf(org.apache.hadoop.mapred.JobContext context)

Get job conf from the old style JobContext.

Parameters:

context - job context

Returns:

job conf

getNonCachedFileSystem

org.apache.hadoop.fs.FileSystem getNonCachedFileSystem(URI uri,
                                                     org.apache.hadoop.conf.Configuration conf)
                                                       throws IOException

Throws:

IOException

getMergedCredentials

void getMergedCredentials(org.apache.hadoop.mapred.JobConf jobConf)
                          throws IOException

Throws:

IOException

mergeCredentials

void mergeCredentials(org.apache.hadoop.mapred.JobConf dest,
                    org.apache.hadoop.mapred.JobConf src)
                      throws IOException

Throws:

IOException

checkFileAccess

void checkFileAccess(org.apache.hadoop.fs.FileSystem fs,
                   org.apache.hadoop.fs.FileStatus status,
                   org.apache.hadoop.fs.permission.FsAction action)
                     throws IOException,
                            AccessControlException,
                            Exception

Check if the configured UGI has access to the path for the given file system action. Method will return successfully if action is permitted. AccessControlExceptoin will be thrown if user does not have access to perform the action. Other exceptions may be thrown for non-access related errors.

Parameters:

fs -

status -

action -

Throws:

IOException

AccessControlException

Exception

getPassword

String getPassword(org.apache.hadoop.conf.Configuration conf,
                 String name)
                   throws IOException

Use password API (if available) to fetch credentials/password

Parameters:

conf -

name -

Returns:

Throws:

IOException

supportStickyBit

boolean supportStickyBit()

check whether current hadoop supports sticky bit

Returns:

hasStickyBit

boolean hasStickyBit(org.apache.hadoop.fs.permission.FsPermission permission)

Check stick bit in the permission

Parameters:

permission -

Returns:

sticky bit

supportTrashFeature

boolean supportTrashFeature()

Returns:

True if the current hadoop supports trash feature.

getCurrentTrashPath

org.apache.hadoop.fs.Path getCurrentTrashPath(org.apache.hadoop.conf.Configuration conf,
                                            org.apache.hadoop.fs.FileSystem fs)

Returns:

Path to HDFS trash, if current hadoop supports trash feature. Null otherwise.

isDirectory

boolean isDirectory(org.apache.hadoop.fs.FileStatus fileStatus)

Check whether file is directory.

getKerberosNameShim

HadoopShims.KerberosNameShim getKerberosNameShim(String name)
                                                 throws IOException

Returns a shim to wrap KerberosName

Throws:

IOException

runDistCp

boolean runDistCp(org.apache.hadoop.fs.Path src,
                org.apache.hadoop.fs.Path dst,
                org.apache.hadoop.conf.Configuration conf)
                  throws IOException

Copies a source dir/file to a destination by orchestrating the copy between hdfs nodes. This distributed process is meant to copy huge files that could take some time if a single copy is done.

Parameters:

src - Path to the source file or directory to copy

dst - Path to the destination file or directory

conf - The hadoop configuration object

Returns:

True if it is successfull; False otherwise.

Throws:

IOException

createHdfsEncryptionShim

HadoopShims.HdfsEncryptionShim createHdfsEncryptionShim(org.apache.hadoop.fs.FileSystem fs,
                                                      org.apache.hadoop.conf.Configuration conf)
                                                        throws IOException

Returns a new instance of the HdfsEncryption shim.

Parameters:

fs - A FileSystem object to HDFS

conf - A Configuration object

Returns:

A new instance of the HdfsEncryption shim.

Throws:

IOException - If an error occurred while creating the instance.

getPathWithoutSchemeAndAuthority

org.apache.hadoop.fs.Path getPathWithoutSchemeAndAuthority(org.apache.hadoop.fs.Path path)

readByteBuffer

int readByteBuffer(org.apache.hadoop.fs.FSDataInputStream file,
                 ByteBuffer dest)
                   throws IOException

Reads data into ByteBuffer.

Parameters:

file - File.

dest - Buffer.

Returns:

Number of bytes read, just like file.read. If any bytes were read, dest position will be set to old position + number of bytes read.

Throws:

IOException

addDelegationTokens

void addDelegationTokens(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.security.Credentials cred,
                       String uname)
                         throws IOException

Get Delegation token and add it to Credential.

Parameters:

fs - FileSystem object to HDFS

cred - Credentials object to add the token to.

uname - user name.

Throws:

IOException - If an error occurred on adding the token.