public interface StatsAggregator
Modifier and Type | Method and Description |
---|---|
String |
aggregateStats(String keyPrefix,
String statType)
This method aggregates a given statistic from all tasks (partial stats).
|
boolean |
closeConnection(StatsCollectionContext scc)
This method closes the connection to the temporary storage.
|
boolean |
connect(StatsCollectionContext scc)
This method connects to the temporary storage.
|
boolean connect(StatsCollectionContext scc)
hconf
- HiveConf that contains the connection parameters.sourceTask
- String aggregateStats(String keyPrefix, String statType)
keyPrefix
- a prefix of the keys used in StatsPublisher to publish stats.
Any rows that starts with the same prefix will be aggregated. For example, if
the StatsPublisher uses the following compound key to publish stats:
the output directory name (unique per FileSinkOperator) +
the partition specs (only for dynamic partitions) +
taskID (last component of task file)
The keyPrefix for aggregation could be first 2 components. This will aggregates stats
across all tasks for each partition.statType
- a string noting the key to be published. Ex: "numRows".boolean closeConnection(StatsCollectionContext scc)
Copyright © 2016 The Apache Software Foundation. All rights reserved.