public class SparkUtilities extends Object
Constructor and Description |
---|
SparkUtilities() |
Modifier and Type | Method and Description |
---|---|
static void |
collectOp(Collection<Operator<?>> result,
Operator<?> root,
Class<?> clazz)
Recursively find all operators under root, that are of class clazz, and
put them in result.
|
static org.apache.hadoop.io.BytesWritable |
copyBytesWritable(org.apache.hadoop.io.BytesWritable bw) |
static HiveKey |
copyHiveKey(HiveKey key) |
static SparkTask |
createSparkTask(HiveConf conf) |
static SparkTask |
createSparkTask(SparkWork work,
HiveConf conf) |
static org.apache.hadoop.fs.Path |
generateTmpPathForPartitionPruning(org.apache.hadoop.fs.Path basePath,
String id)
Generate a temporary path for dynamic partition pruning in Spark branch
TODO: no longer need this if we use accumulator!
|
static SparkSession |
getSparkSession(HiveConf conf,
SparkSessionManager sparkSessionManager) |
static URI |
getURI(String path) |
static String |
getWorkId(BaseWork work)
Return the ID for this BaseWork, in String form.
|
static boolean |
isDedicatedCluster(org.apache.hadoop.conf.Configuration conf) |
static boolean |
needUploadToHDFS(URI source,
org.apache.spark.SparkConf sparkConf) |
static String |
rddGraphToString(org.apache.spark.api.java.JavaPairRDD rdd) |
static URI |
uploadToHDFS(URI source,
HiveConf conf)
Uploads a local file to HDFS
|
public static org.apache.hadoop.io.BytesWritable copyBytesWritable(org.apache.hadoop.io.BytesWritable bw)
public static URI getURI(String path) throws URISyntaxException
URISyntaxException
public static URI uploadToHDFS(URI source, HiveConf conf) throws IOException
source
- conf
- IOException
public static boolean needUploadToHDFS(URI source, org.apache.spark.SparkConf sparkConf)
public static boolean isDedicatedCluster(org.apache.hadoop.conf.Configuration conf)
public static SparkSession getSparkSession(HiveConf conf, SparkSessionManager sparkSessionManager) throws HiveException
HiveException
public static String rddGraphToString(org.apache.spark.api.java.JavaPairRDD rdd)
public static org.apache.hadoop.fs.Path generateTmpPathForPartitionPruning(org.apache.hadoop.fs.Path basePath, String id)
basePath
- id
- public static String getWorkId(BaseWork work)
work
- the input BaseWorkpublic static void collectOp(Collection<Operator<?>> result, Operator<?> root, Class<?> clazz)
result
- all operators under root that are of class clazzroot
- the root operator under which all operators will be examinedclazz
- clas to collect. Must NOT be null.Copyright © 2016 The Apache Software Foundation. All rights reserved.