org.apache.hive.spark.counter

Class SparkCounters

java.lang.Object

org.apache.hive.spark.counter.SparkCounters

All Implemented Interfaces:

Serializable

public class SparkCounters
extends Object
implements Serializable

SparkCounters is used to collect Hive operator metric through Spark accumulator. There are few limitation of Spark accumulator, like: 1. accumulator should be created at Spark context side. 2. Spark tasks can only increment metric count. 3. User can only get accumulator value at Spark context side. These Spark Counter API is designed to fit into Hive requirement, while with several access restriction due to Spark accumulator previous mentioned: 1. Counter should be created on driver side if it would be accessed in task. 2. increment could only be invoked task side. 3. Hive could only get Counter value at driver side.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SparkCounters(org.apache.spark.api.java.JavaSparkContext javaSparkContext)

Method Summary

Methods

Modifier and Type	Method and Description
void	createCounter(Enum<?> key)
void	createCounter(String groupName, Enum<?> key)
void	createCounter(String groupName, String counterName)
void	createCounter(String groupName, String counterName, long initValue)
SparkCounter	getCounter(Enum<?> key)
SparkCounter	getCounter(String groupName, String counterName)
Map<String,SparkCounterGroup>	getSparkCounterGroups()
long	getValue(String groupName, String counterName)
void	increment(Enum<?> key, long incrValue)
void	increment(String groupName, String counterName, long value)
SparkCounters	snapshot() Create a snapshot of the current counters to send back to the client.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SparkCounters

public SparkCounters(org.apache.spark.api.java.JavaSparkContext javaSparkContext)

Method Detail

createCounter

public void createCounter(Enum<?> key)

createCounter

public void createCounter(String groupName,
                 Enum<?> key)

createCounter

public void createCounter(String groupName,
                 String counterName)

createCounter

public void createCounter(String groupName,
                 String counterName,
                 long initValue)

increment

public void increment(Enum<?> key,
             long incrValue)

increment

public void increment(String groupName,
             String counterName,
             long value)

getValue

public long getValue(String groupName,
            String counterName)

getCounter

public SparkCounter getCounter(String groupName,
                      String counterName)

getCounter

public SparkCounter getCounter(Enum<?> key)

getSparkCounterGroups

public Map<String,SparkCounterGroup> getSparkCounterGroups()

toString

public String toString()

Overrides:

toString in class Object

snapshot

public SparkCounters snapshot()

Create a snapshot of the current counters to send back to the client. This copies the values of all current counters into a new SparkCounters instance that cannot be used to update the counters, but will serialize cleanly when sent back to the RSC client.