org.apache.hive.hcatalog.templeton.tool

Class TempletonUtils

java.lang.Object

org.apache.hive.hcatalog.templeton.tool.TempletonUtils

public class TempletonUtils
extends Object

General utility methods.

Field Summary

Fields

Modifier and Type	Field and Description
static Pattern	HIVE_COMPLETE
static Pattern[]	ID_PATTERNS
static Pattern	JAR_COMPLETE
static Pattern	JAR_ID
static Pattern	PIG_COMPLETE
static Pattern	PIG_ID
static Pattern	TEZ_COMPLETE Hive on Tez produces progress report that looks like this Map 1: -/- Reducer 2: 0/1 Map 1: -/- Reducer 2: 0(+1)/1 Map 1: -/- Reducer 2: 1/1 -/- means there are no tasks (yet) 0/1 means 1 total tasks, 0 completed 1(+2)/3 means 3 total, 1 completed and 2 running HIVE-8495, in particular https://issues.apache.org/jira/secure/attachment/12675504/Screen%20Shot%202014-10-16%20at%209.35.26%20PM.png has more examples.
static Pattern	TEZ_COUNTERS

Constructor Summary

Constructors

Constructor and Description
TempletonUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static void	addCmdForWindows(ArrayList<String> args)
static String	addUserHomeDirectoryIfApplicable(String origPathStr, String user)
static String[]	decodeArray(String s) Take an encode strings and decode it into an array of strings.
static StringBuilder	dumpPropMap(String header, Map<String,String> map)
static StringBuilder	dumpPropMap(String header, Properties props)
static String	encodeArray(List<String> list) Encode a List into a string.
static String	encodeArray(String[] plain) Take an array of strings and encode it into one string.
static String	extractChildJobId(String line) Extract the job id from jar jobs.
static String	extractPercentComplete(String line) Extract the percent complete line from Pig or Jar jobs.
static int	fetchUrl(URL url) GET the given url.
static String	findContainingJar(Class<?> clazz, String fileNamePattern) Find a jar that contains a class of the same name and which file name matches the given pattern.
static String	hadoopFsFilename(String fname, org.apache.hadoop.conf.Configuration conf, String user)
static boolean	hadoopFsIsMissing(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p)
static String[]	hadoopFsListAsArray(String files, org.apache.hadoop.conf.Configuration conf, String user)
static String	hadoopFsListAsString(String files, org.apache.hadoop.conf.Configuration conf, String user)
static List<org.apache.hadoop.fs.Path>	hadoopFsListChildren(String dirName, org.apache.hadoop.conf.Configuration conf, String user) Returns all files (non-recursive) in dirName
static org.apache.hadoop.fs.Path	hadoopFsPath(String fname, org.apache.hadoop.conf.Configuration conf, String user)
static Map<String,String>	hadoopUserEnv(String user, String overrideClasspath) Set the environment variables to specify the hadoop user.
static boolean	isset(char ch) Is the object non-empty?
static <T> boolean	isset(Collection<T> col) Is the object non-empty?
static <K,V> boolean	isset(Map<K,V> col) Is the object non-empty?
static boolean	isset(String s) Is the object non-empty?
static <T> boolean	isset(T[] a) Is the object non-empty?
static String	quoteForWindows(String param)
static String	unEscapeString(String s) replaces all occurrences of "\," with ","; returns s if no modifications needed

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

JAR_COMPLETE

public static final Pattern JAR_COMPLETE

PIG_COMPLETE

public static final Pattern PIG_COMPLETE

HIVE_COMPLETE

public static final Pattern HIVE_COMPLETE

TEZ_COMPLETE

public static final Pattern TEZ_COMPLETE

Hive on Tez produces progress report that looks like this Map 1: -/- Reducer 2: 0/1 Map 1: -/- Reducer 2: 0(+1)/1 Map 1: -/- Reducer 2: 1/1 -/- means there are no tasks (yet) 0/1 means 1 total tasks, 0 completed 1(+2)/3 means 3 total, 1 completed and 2 running HIVE-8495, in particular https://issues.apache.org/jira/secure/attachment/12675504/Screen%20Shot%202014-10-16%20at%209.35.26%20PM.png has more examples. To report progress, we'll assume all tasks are equal size and compute "completed" as percent of "total" "(Map|Reducer) (\\d+:) ((-/-)|(\\d+(\\(\\+\\d+\\))?/\\d+))" is the complete pattern but we'll drop "-/-" to exclude groups that don't add information such as "Map 1: -/-"

TEZ_COUNTERS

public static final Pattern TEZ_COUNTERS

JAR_ID

public static final Pattern JAR_ID

PIG_ID

public static final Pattern PIG_ID

ID_PATTERNS

public static final Pattern[] ID_PATTERNS

Constructor Detail

TempletonUtils

public TempletonUtils()

Method Detail

isset

public static boolean isset(String s)

Is the object non-empty?

isset

public static boolean isset(char ch)

Is the object non-empty?

isset

public static <T> boolean isset(T[] a)

Is the object non-empty?

isset

public static <T> boolean isset(Collection<T> col)

Is the object non-empty?

isset

public static <K,V> boolean isset(Map<K,V> col)

Is the object non-empty?

extractPercentComplete

public static String extractPercentComplete(String line)

Extract the percent complete line from Pig or Jar jobs.

extractChildJobId

public static String extractChildJobId(String line)

Extract the job id from jar jobs.

encodeArray

public static String encodeArray(String[] plain)

Take an array of strings and encode it into one string.

encodeArray

public static String encodeArray(List<String> list)

Encode a List into a string.

decodeArray

public static String[] decodeArray(String s)

Take an encode strings and decode it into an array of strings.

hadoopFsListAsArray

public static String[] hadoopFsListAsArray(String files,
                           org.apache.hadoop.conf.Configuration conf,
                           String user)
                                    throws URISyntaxException,
                                           FileNotFoundException,
                                           IOException,
                                           InterruptedException

Throws:

URISyntaxException

FileNotFoundException

IOException

InterruptedException

hadoopFsListAsString

public static String hadoopFsListAsString(String files,
                          org.apache.hadoop.conf.Configuration conf,
                          String user)
                                   throws URISyntaxException,
                                          FileNotFoundException,
                                          IOException,
                                          InterruptedException

Throws:

URISyntaxException

FileNotFoundException

IOException

InterruptedException

hadoopFsFilename

public static String hadoopFsFilename(String fname,
                      org.apache.hadoop.conf.Configuration conf,
                      String user)
                               throws URISyntaxException,
                                      FileNotFoundException,
                                      IOException,
                                      InterruptedException

Throws:

URISyntaxException

FileNotFoundException

IOException

InterruptedException

hadoopFsListChildren

public static List<org.apache.hadoop.fs.Path> hadoopFsListChildren(String dirName,
                                                   org.apache.hadoop.conf.Configuration conf,
                                                   String user)
                                                            throws URISyntaxException,
                                                                   IOException,
                                                                   InterruptedException

Returns all files (non-recursive) in dirName

Throws:

URISyntaxException

IOException

InterruptedException

hadoopFsIsMissing

public static boolean hadoopFsIsMissing(org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path p)

Returns:

true iff we are sure the file is not there.

addUserHomeDirectoryIfApplicable

public static String addUserHomeDirectoryIfApplicable(String origPathStr,
                                      String user)
                                               throws IOException,
                                                      URISyntaxException

Throws:

IOException

URISyntaxException

hadoopFsPath

public static org.apache.hadoop.fs.Path hadoopFsPath(String fname,
                                     org.apache.hadoop.conf.Configuration conf,
                                     String user)
                                              throws URISyntaxException,
                                                     IOException,
                                                     InterruptedException

Throws:

URISyntaxException

IOException

InterruptedException

fetchUrl

public static int fetchUrl(URL url)
                    throws IOException

GET the given url. Returns the number of bytes received.

Throws:

IOException

hadoopUserEnv

public static Map<String,String> hadoopUserEnv(String user,
                               String overrideClasspath)

Set the environment variables to specify the hadoop user.

quoteForWindows

public static String quoteForWindows(String param)
                              throws BadParam

Throws:

BadParam

addCmdForWindows

public static void addCmdForWindows(ArrayList<String> args)

unEscapeString

public static String unEscapeString(String s)

replaces all occurrences of "\," with ","; returns s if no modifications needed

findContainingJar

public static String findContainingJar(Class<?> clazz,
                       String fileNamePattern)

Find a jar that contains a class of the same name and which file name matches the given pattern.

Parameters:

clazz - the class to find.

fileNamePattern - regex pattern that must match the jar full path

Returns:

a jar file that contains the class, or null

dumpPropMap

public static StringBuilder dumpPropMap(String header,
                        Properties props)

dumpPropMap

public static StringBuilder dumpPropMap(String header,
                        Map<String,String> map)