org.apache.hadoop.hive.ql.stats

Class CounterStatsAggregatorTez

java.lang.Object

org.apache.hadoop.hive.ql.stats.CounterStatsAggregatorTez

All Implemented Interfaces:

StatsAggregator, StatsCollectionTaskIndependent

public class CounterStatsAggregatorTez
extends Object
implements StatsAggregator, StatsCollectionTaskIndependent

This class aggregates stats via counters and does so for Tez Tasks. With dbclass=counters this class will compute table/partition statistics using hadoop counters. They will be published using special keys and then retrieved on the client after the insert/ctas statement ran.

Constructor Summary

Constructors

Constructor and Description
CounterStatsAggregatorTez()

Method Summary

Methods

Modifier and Type	Method and Description
String	aggregateStats(String keyPrefix, String statType) This method aggregates a given statistic from all tasks (partial stats).
boolean	cleanUp(String keyPrefix) This method is called after all statistics have been aggregated.
boolean	closeConnection() This method closes the connection to the temporary storage.
boolean	connect(org.apache.hadoop.conf.Configuration hconf, Task sourceTask) This method connects to the temporary storage.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CounterStatsAggregatorTez

public CounterStatsAggregatorTez()

Method Detail

connect

public boolean connect(org.apache.hadoop.conf.Configuration hconf,
              Task sourceTask)

Description copied from interface: StatsAggregator

This method connects to the temporary storage.

Specified by:

connect in interface StatsAggregator

Parameters:

hconf - HiveConf that contains the connection parameters.

Returns:

true if connection is successful, false otherwise.

aggregateStats

public String aggregateStats(String keyPrefix,
                    String statType)

Description copied from interface: StatsAggregator

This method aggregates a given statistic from all tasks (partial stats). After aggregation, this method also automatically removes all records that have been aggregated.

Specified by:

aggregateStats in interface StatsAggregator

Parameters:

keyPrefix - a prefix of the keys used in StatsPublisher to publish stats. Any rows that starts with the same prefix will be aggregated. For example, if the StatsPublisher uses the following compound key to publish stats: the output directory name (unique per FileSinkOperator) + the partition specs (only for dynamic partitions) + taskID (last component of task file) The keyPrefix for aggregation could be first 2 components. This will aggregates stats across all tasks for each partition.

statType - a string noting the key to be published. Ex: "numRows".

Returns:

a string representation of a long value, null if there are any error/exception.

closeConnection

public boolean closeConnection()

Description copied from interface: StatsAggregator

This method closes the connection to the temporary storage.

Specified by:

closeConnection in interface StatsAggregator

Returns:

true if close connection is successful, false otherwise.

cleanUp

public boolean cleanUp(String keyPrefix)

Description copied from interface: StatsAggregator

This method is called after all statistics have been aggregated. Since we support multiple statistics, we do not perform automatic cleanup after aggregation. After this method is called, closeConnection must be called as well. This method is also used to clear the temporary statistics that have been published without being aggregated. Typically this happens when a job fails, or is forcibly stopped after publishing some statistics.

Specified by:

cleanUp in interface StatsAggregator

Parameters:

keyPrefix - a prefix of the keys used in StatsPublisher to publish stats. It is the same as the first parameter in aggregateStats().

Returns:

true if cleanup is successful, false otherwise.