public class FSStatsAggregator extends Object implements StatsAggregator
Constructor and Description |
---|
FSStatsAggregator() |
Modifier and Type | Method and Description |
---|---|
String |
aggregateStats(String partID,
String statType)
This method aggregates a given statistic from all tasks (partial stats).
|
boolean |
closeConnection(StatsCollectionContext scc)
This method closes the connection to the temporary storage.
|
boolean |
connect(StatsCollectionContext scc)
This method connects to the temporary storage.
|
public boolean connect(StatsCollectionContext scc)
StatsAggregator
connect
in interface StatsAggregator
public String aggregateStats(String partID, String statType)
StatsAggregator
aggregateStats
in interface StatsAggregator
partID
- a prefix of the keys used in StatsPublisher to publish stats.
Any rows that starts with the same prefix will be aggregated. For example, if
the StatsPublisher uses the following compound key to publish stats:
the output directory name (unique per FileSinkOperator) +
the partition specs (only for dynamic partitions) +
taskID (last component of task file)
The keyPrefix for aggregation could be first 2 components. This will aggregates stats
across all tasks for each partition.statType
- a string noting the key to be published. Ex: "numRows".public boolean closeConnection(StatsCollectionContext scc)
StatsAggregator
closeConnection
in interface StatsAggregator
Copyright © 2016 The Apache Software Foundation. All rights reserved.