org.apache.hadoop.hive.common

Class FileUtils

java.lang.Object

org.apache.hadoop.hive.common.FileUtils

public final class FileUtils
extends Object

Collection of file manipulation utilities common across Hive.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	HIDDEN_FILES_PATH_FILTER
static org.apache.hadoop.fs.PathFilter	STAGING_DIR_PATH_FILTER

Method Summary

Methods

Modifier and Type	Method and Description
static void	checkDeletePermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, String user) Checks if delete can be performed on given path by given user.
static void	checkFileAccessWithImpersonation(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus stat, org.apache.hadoop.fs.permission.FsAction action, String user) Perform a check to determine if the user is able to access the file passed in.
static boolean	copy(org.apache.hadoop.fs.FileSystem srcFS, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.FileSystem dstFS, org.apache.hadoop.fs.Path dst, boolean deleteSource, boolean overwrite, HiveConf conf) Copies files between filesystems.
static boolean	equalsFileSystem(org.apache.hadoop.fs.FileSystem fs1, org.apache.hadoop.fs.FileSystem fs2)
static String	escapePathName(String path)
static String	escapePathName(String path, String defaultPath) Escapes a path name.
static org.apache.hadoop.fs.FileStatus	getFileStatusOrNull(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Attempts to get file status.
static org.apache.hadoop.fs.FileStatus	getPathOrParentThatExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Find the parent of path that exists, if path does not exist
static boolean	isActionPermittedForFileHierarchy(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus, String userName, org.apache.hadoop.fs.permission.FsAction action) Check if user userName has permissions to perform the given FsAction action on all files under the file whose FileStatus fileStatus is provided
static boolean	isLocalFile(HiveConf conf, String fileName) A best effort attempt to determine if if the file is a local file
static boolean	isLocalFile(HiveConf conf, URI fileUri) A best effort attempt to determine if if the file is a local file
static boolean	isOwnerOfFileHierarchy(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus, String userName)
static boolean	isSubDir(org.apache.hadoop.fs.Path p1, org.apache.hadoop.fs.Path p2, org.apache.hadoop.fs.FileSystem fs) Check if first path is a subdirectory of second path.
static void	listStatusRecursively(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus fileStatus, List<org.apache.hadoop.fs.FileStatus> results) Recursively lists status for all files starting from a particular directory (or individual file as base case).
static String	makeDefaultListBucketingDirName(List<String> skewedCols, String name) default directory will have the same depth as number of skewed columns this will make future operation easy like DML merge, concatenate merge
static String	makeListBucketingDirName(List<String> lbCols, List<String> vals) Makes a valid list bucketing directory name.
static String	makePartName(List<String> partCols, List<String> vals)
static String	makePartName(List<String> partCols, List<String> vals, String defaultStr) Makes a valid partition name.
static org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI.
static boolean	mkdir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path f, boolean inheritPerms, org.apache.hadoop.conf.Configuration conf) Creates the directory and all necessary parent directories.
static boolean	moveToTrash(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path f, org.apache.hadoop.conf.Configuration conf) Move a particular file or directory to the trash.
static boolean	renameWithPerms(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath, boolean inheritPerms, org.apache.hadoop.conf.Configuration conf)
static boolean	trashFilesUnderDir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path f, org.apache.hadoop.conf.Configuration conf) Deletes all files under a directory, sending them to the trash.
static String	unescapePathName(String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HIDDEN_FILES_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter HIDDEN_FILES_PATH_FILTER

STAGING_DIR_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter STAGING_DIR_PATH_FILTER

Method Detail

makeQualified

public static org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path,
                                      org.apache.hadoop.conf.Configuration conf)
                                               throws IOException

Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI. This saves the cost of opening the Filesystem - which can involve RPCs - as well as cause errors

Parameters:

path - path to be fully qualified

conf - Configuration file

Returns:

path qualified relative to default file system

Throws:

IOException

makePartName

public static String makePartName(List<String> partCols,
                  List<String> vals)

makePartName

public static String makePartName(List<String> partCols,
                  List<String> vals,
                  String defaultStr)

Makes a valid partition name.

Parameters:

partCols - The partition keys' names

vals - The partition values

defaultStr - The default name given to a partition value if the respective value is empty or null.

Returns:

An escaped, valid partition name.

makeDefaultListBucketingDirName

public static String makeDefaultListBucketingDirName(List<String> skewedCols,
                                     String name)

default directory will have the same depth as number of skewed columns this will make future operation easy like DML merge, concatenate merge

Parameters:

skewedCols -

name -

Returns:

makeListBucketingDirName

public static String makeListBucketingDirName(List<String> lbCols,
                              List<String> vals)

Makes a valid list bucketing directory name.

Parameters:

lbCols - The skewed keys' names

vals - The skewed values

Returns:

An escaped, valid list bucketing directory name.

escapePathName

public static String escapePathName(String path)

escapePathName

public static String escapePathName(String path,
                    String defaultPath)

Escapes a path name.

Parameters:

path - The path to escape.

defaultPath - The default name for the path, if the given path is empty or null.

Returns:

An escaped path name.

unescapePathName

public static String unescapePathName(String path)

listStatusRecursively

public static void listStatusRecursively(org.apache.hadoop.fs.FileSystem fs,
                         org.apache.hadoop.fs.FileStatus fileStatus,
                         List<org.apache.hadoop.fs.FileStatus> results)
                                  throws IOException

Recursively lists status for all files starting from a particular directory (or individual file as base case).

Parameters:

fs - file system

fileStatus - starting point in file system

results - receives enumeration of all files found

Throws:

IOException

getPathOrParentThatExists

public static org.apache.hadoop.fs.FileStatus getPathOrParentThatExists(org.apache.hadoop.fs.FileSystem fs,
                                                        org.apache.hadoop.fs.Path path)
                                                                 throws IOException

Find the parent of path that exists, if path does not exist

Parameters:

fs - file system

path -

Returns:

FileStatus for argument path if it exists or the first ancestor in the path that exists

Throws:

IOException

checkFileAccessWithImpersonation

public static void checkFileAccessWithImpersonation(org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.FileStatus stat,
                                    org.apache.hadoop.fs.permission.FsAction action,
                                    String user)
                                             throws IOException,
                                                    AccessControlException,
                                                    InterruptedException,
                                                    Exception

Perform a check to determine if the user is able to access the file passed in. If the user name passed in is different from the current user, this method will attempt to do impersonate the user to do the check; the current user should be able to create proxy users in this case.

Parameters:

fs - FileSystem of the path to check

stat - FileStatus representing the file

action - FsAction that will be checked

user - User name of the user that will be checked for access. If the user name is null or the same as the current user, no user impersonation will be done and the check will be done as the current user. Otherwise the file access check will be performed within a doAs() block to use the access privileges of this user. In this case the user must be configured to impersonate other users, otherwise this check will fail with error.

Throws:

IOException

AccessControlException

InterruptedException

Exception

isActionPermittedForFileHierarchy

public static boolean isActionPermittedForFileHierarchy(org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.fs.FileStatus fileStatus,
                                        String userName,
                                        org.apache.hadoop.fs.permission.FsAction action)
                                                 throws Exception

Check if user userName has permissions to perform the given FsAction action on all files under the file whose FileStatus fileStatus is provided

Parameters:

fs -

fileStatus -

userName -

action -

Returns:

Throws:

IOException

Exception

isLocalFile

public static boolean isLocalFile(HiveConf conf,
                  String fileName)

A best effort attempt to determine if if the file is a local file

Parameters:

conf -

fileName -

Returns:

true if it was successfully able to determine that it is a local file

isLocalFile

public static boolean isLocalFile(HiveConf conf,
                  URI fileUri)

A best effort attempt to determine if if the file is a local file

Parameters:

conf -

fileUri -

Returns:

true if it was successfully able to determine that it is a local file

isOwnerOfFileHierarchy

public static boolean isOwnerOfFileHierarchy(org.apache.hadoop.fs.FileSystem fs,
                             org.apache.hadoop.fs.FileStatus fileStatus,
                             String userName)
                                      throws IOException

Throws:

IOException

mkdir

public static boolean mkdir(org.apache.hadoop.fs.FileSystem fs,
            org.apache.hadoop.fs.Path f,
            boolean inheritPerms,
            org.apache.hadoop.conf.Configuration conf)
                     throws IOException

Creates the directory and all necessary parent directories.

Parameters:

fs - FileSystem to use

f - path to create.

inheritPerms - whether directory inherits the permission of the last-existing parent path

conf - Hive configuration

Returns:

true if directory created successfully. False otherwise, including if it exists.

Throws:

IOException - exception in creating the directory

copy

public static boolean copy(org.apache.hadoop.fs.FileSystem srcFS,
           org.apache.hadoop.fs.Path src,
           org.apache.hadoop.fs.FileSystem dstFS,
           org.apache.hadoop.fs.Path dst,
           boolean deleteSource,
           boolean overwrite,
           HiveConf conf)
                    throws IOException

Copies files between filesystems.

Throws:

IOException

trashFilesUnderDir

public static boolean trashFilesUnderDir(org.apache.hadoop.fs.FileSystem fs,
                         org.apache.hadoop.fs.Path f,
                         org.apache.hadoop.conf.Configuration conf)
                                  throws FileNotFoundException,
                                         IOException

Deletes all files under a directory, sending them to the trash. Leaves the directory as is.

Parameters:

fs - FileSystem to use

f - path of directory

conf - hive configuration

Returns:

true if deletion successful

Throws:

FileNotFoundException

IOException

moveToTrash

public static boolean moveToTrash(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path f,
                  org.apache.hadoop.conf.Configuration conf)
                           throws IOException

Move a particular file or directory to the trash.

Parameters:

fs - FileSystem to use

f - path of file or directory to move to trash.

conf -

Returns:

true if move successful

Throws:

IOException

isSubDir

public static boolean isSubDir(org.apache.hadoop.fs.Path p1,
               org.apache.hadoop.fs.Path p2,
               org.apache.hadoop.fs.FileSystem fs)

Check if first path is a subdirectory of second path. Both paths must belong to the same filesystem.

Parameters:

p1 - first path

p2 - second path

fs - FileSystem, both paths must belong to the same filesystem

Returns:

renameWithPerms

public static boolean renameWithPerms(org.apache.hadoop.fs.FileSystem fs,
                      org.apache.hadoop.fs.Path sourcePath,
                      org.apache.hadoop.fs.Path destPath,
                      boolean inheritPerms,
                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Throws:

IOException

equalsFileSystem

public static boolean equalsFileSystem(org.apache.hadoop.fs.FileSystem fs1,
                       org.apache.hadoop.fs.FileSystem fs2)

Parameters:

fs1 -

fs2 -

Returns:

return true if both file system arguments point to same file system

checkDeletePermission

public static void checkDeletePermission(org.apache.hadoop.fs.Path path,
                         org.apache.hadoop.conf.Configuration conf,
                         String user)
                                  throws AccessControlException,
                                         InterruptedException,
                                         Exception

Checks if delete can be performed on given path by given user. If file does not exist it just returns without throwing an Exception

Parameters:

path -

conf -

user -

Throws:

AccessControlException

InterruptedException

Exception

getFileStatusOrNull

public static org.apache.hadoop.fs.FileStatus getFileStatusOrNull(org.apache.hadoop.fs.FileSystem fs,
                                                  org.apache.hadoop.fs.Path path)
                                                           throws IOException

Attempts to get file status. This method differs from the FileSystem API in that it returns null instead of throwing FileNotFoundException if the path does not exist.

Parameters:

fs - file system to check

path - file system path to check

Returns:

FileStatus for path or null if path does not exist

Throws:

IOException - if there is an I/O error