Modifier and Type | Method and Description |
---|---|
static MapWork |
Utilities.getMapWork(org.apache.hadoop.conf.Configuration conf) |
Modifier and Type | Method and Description |
---|---|
Collection<MapWork> |
Task.getMapWork() |
Modifier and Type | Method and Description |
---|---|
static void |
Utilities.cacheMapWork(org.apache.hadoop.conf.Configuration conf,
MapWork work,
org.apache.hadoop.fs.Path hiveScratchDir) |
static void |
Utilities.createTmpDirs(org.apache.hadoop.conf.Configuration conf,
MapWork mWork)
Hive uses tmp directories to capture the output of each FileSinkOperator.
|
static int |
Utilities.estimateNumberOfReducers(HiveConf conf,
org.apache.hadoop.fs.ContentSummary inputSummary,
MapWork work,
boolean finalMapRed)
Estimate the number of reducers needed for this job, based on job input,
and configuration parameters.
|
static double |
Utilities.getHighestSamplePercentage(MapWork work)
Returns the highest sample percentage of any alias in the given MapWork
|
static List<org.apache.hadoop.fs.Path> |
Utilities.getInputPaths(org.apache.hadoop.mapred.JobConf job,
MapWork work,
org.apache.hadoop.fs.Path hiveScratchDir,
Context ctx,
boolean skipDummy)
Computes a list of all input paths needed to compute the given MapWork.
|
static List<org.apache.hadoop.fs.Path> |
Utilities.getInputPathsTez(org.apache.hadoop.mapred.JobConf job,
MapWork work)
On Tez we're not creating dummy files when getting/setting input paths.
|
static org.apache.hadoop.fs.ContentSummary |
Utilities.getInputSummary(Context ctx,
MapWork work,
org.apache.hadoop.fs.PathFilter filter)
Calculate the total size of input files.
|
Operator<? extends OperatorDesc> |
Task.getReducer(MapWork work) |
static long |
Utilities.getTotalInputFileSize(org.apache.hadoop.fs.ContentSummary inputSummary,
MapWork work,
double highestSamplePercentage)
Computes the total input file size.
|
static long |
Utilities.getTotalInputNumFiles(org.apache.hadoop.fs.ContentSummary inputSummary,
MapWork work,
double highestSamplePercentage)
Computes the total number of input files.
|
static void |
Utilities.setInputAttributes(org.apache.hadoop.conf.Configuration conf,
MapWork mWork)
Set hive input format, and input format file if necessary.
|
static void |
Utilities.setMapWork(org.apache.hadoop.conf.Configuration conf,
MapWork work) |
static org.apache.hadoop.fs.Path |
Utilities.setMapWork(org.apache.hadoop.conf.Configuration conf,
MapWork w,
org.apache.hadoop.fs.Path hiveScratchDir,
boolean useCache) |
Modifier and Type | Method and Description |
---|---|
Collection<MapWork> |
ExecDriver.getMapWork() |
Modifier and Type | Method and Description |
---|---|
Operator<? extends OperatorDesc> |
MapRedTask.getReducer(MapWork mapWork) |
Modifier and Type | Method and Description |
---|---|
Collection<MapWork> |
SparkTask.getMapWork() |
Modifier and Type | Method and Description |
---|---|
Operator<? extends OperatorDesc> |
SparkTask.getReducer(MapWork mapWork) |
Modifier and Type | Method and Description |
---|---|
Collection<MapWork> |
TezTask.getMapWork() |
Modifier and Type | Method and Description |
---|---|
Operator<? extends OperatorDesc> |
TezTask.getReducer(MapWork mapWork) |
Constructor and Description |
---|
DynamicPartitionPruner(org.apache.tez.runtime.api.InputInitializerContext context,
MapWork work,
org.apache.hadoop.mapred.JobConf jobConf) |
Modifier and Type | Field and Description |
---|---|
protected MapWork |
HiveInputFormat.mrwork |
Modifier and Type | Class and Description |
---|---|
class |
MergeFileWork |
Modifier and Type | Class and Description |
---|---|
class |
PartialScanWork
Partial Scan Work.
|
Modifier and Type | Class and Description |
---|---|
class |
ColumnTruncateWork |
Modifier and Type | Method and Description |
---|---|
static MapWork |
GenMapRedUtils.createMergeTask(FileSinkDesc fsInputDesc,
org.apache.hadoop.fs.Path finalName,
boolean hasDynamicPartitions)
Create a block level merge task for RCFiles or stripe level merge task for
ORCFiles
|
Modifier and Type | Method and Description |
---|---|
static String |
GenMapRedUtils.findAlias(MapWork work,
Operator<?> operator) |
static Set<String> |
GenMapRedUtils.findAliases(MapWork work,
Operator<?> startOp) |
static void |
GenMapRedUtils.replaceMapWork(String sourceAlias,
String targetAlias,
MapWork source,
MapWork target)
Replace the Map-side operator tree associated with targetAlias in
target with the Map-side operator tree associated with sourceAlias in source.
|
static void |
GenMapRedUtils.setMapWork(MapWork plan,
ParseContext parseCtx,
Set<ReadEntity> inputs,
PrunedPartitionList partsList,
Operator<? extends OperatorDesc> topOp,
String alias_id,
HiveConf conf,
boolean local)
initialize MapWork
|
static void |
GenMapRedUtils.setTaskPlan(String path,
String alias,
Operator<? extends OperatorDesc> topOp,
MapWork plan,
boolean local,
TableDesc tt_desc)
set the current task in the mapredWork.
|
Modifier and Type | Method and Description |
---|---|
long |
AbstractJoinTaskDispatcher.getTotalKnownInputSize(Context context,
MapWork currWork,
Map<String,ArrayList<String>> pathToAliases,
HashMap<String,Long> aliasToSize) |
Modifier and Type | Method and Description |
---|---|
static void |
SparkSortMergeJoinFactory.annotateMapWork(GenSparkProcContext context,
MapWork mapWork,
SMBMapJoinOperator smbMapJoinOp,
TableScanOperator ts,
boolean local)
Annotate MapWork, input is a SMBJoinOp that is part of a MapWork, and its root TS operator.
|
Modifier and Type | Method and Description |
---|---|
MapWork |
GenTezUtils.createMapWork(GenTezProcContext context,
Operator<?> root,
TezWork tezWork,
PrunedPartitionList partitions) |
Modifier and Type | Method and Description |
---|---|
protected void |
GenTezUtils.setupMapWork(MapWork mapWork,
GenTezProcContext context,
PrunedPartitionList partitions,
Operator<? extends OperatorDesc> root,
String alias) |
Modifier and Type | Method and Description |
---|---|
MapWork |
GenSparkUtils.createMapWork(GenSparkProcContext context,
Operator<?> root,
SparkWork sparkWork,
PrunedPartitionList partitions) |
MapWork |
GenSparkUtils.createMapWork(GenSparkProcContext context,
Operator<?> root,
SparkWork sparkWork,
PrunedPartitionList partitions,
boolean deferSetup) |
Modifier and Type | Method and Description |
---|---|
protected void |
GenSparkUtils.setupMapWork(MapWork mapWork,
GenSparkProcContext context,
PrunedPartitionList partitions,
Operator<? extends OperatorDesc> root,
String alias) |
Modifier and Type | Method and Description |
---|---|
MapWork |
MapredWork.getMapWork() |
Modifier and Type | Method and Description |
---|---|
void |
MapWork.mergingInto(MapWork mapWork) |
void |
MapredWork.setMapWork(MapWork mapWork) |
Copyright © 2017 The Apache Software Foundation. All rights reserved.