org.apache.hadoop.hive.ql.plan

Class SparkWork

java.lang.Object

org.apache.hadoop.hive.ql.plan.AbstractOperatorDesc

org.apache.hadoop.hive.ql.plan.SparkWork

All Implemented Interfaces:

Serializable, Cloneable, OperatorDesc

public class SparkWork
extends AbstractOperatorDesc

This class encapsulates all the work objects that can be executed in a single Spark job. Currently it's basically a tree with MapWork at the roots and and ReduceWork at all other nodes.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	SparkWork.Dependency

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<org.apache.commons.lang3.tuple.Pair<BaseWork,BaseWork>,SparkEdgeProperty>	edgeProperties
protected Map<BaseWork,List<BaseWork>>	invertedWorkGraph
protected Map<BaseWork,List<BaseWork>>	workGraph

Fields inherited from class org.apache.hadoop.hive.ql.plan.AbstractOperatorDesc

opProps, opTraits, statistics, vectorMode

Constructor Summary

Constructors

Constructor and Description
SparkWork(String name)

Method Summary

Methods

Modifier and Type	Method and Description
void	add(BaseWork w) add creates a new node in the graph without any connections
void	addAll(BaseWork[] bws) Add all nodes in the collection without any connections.
void	addAll(Collection<BaseWork> c) Add all nodes in the collection without any connections.
void	connect(BaseWork a, BaseWork b, SparkEdgeProperty edgeProp) connect adds an edge between a and b.
boolean	contains(BaseWork w) Whether the specified BaseWork is a vertex in this graph
void	disconnect(BaseWork a, BaseWork b) disconnect removes an edge between a and b.
List<ReduceWork>	getAllReduceWork()
List<BaseWork>	getAllWork()
Collection<BaseWork>	getAllWorkUnsorted()
List<BaseWork>	getChildren(BaseWork work) getChildren returns all the nodes with edges leading out of work
Map<BaseWork,BaseWork>	getCloneToWork()
Map<org.apache.hadoop.hive.ql.plan.SparkWork.ComparableName,List<SparkWork.Dependency>>	getDependencyMap()
SparkEdgeProperty	getEdgeProperty(BaseWork a, BaseWork b) returns the edge type connecting work a and b
Set<BaseWork>	getLeaves() getLeaves returns all nodes that do not have a child
String	getName()
List<BaseWork>	getParents(BaseWork work) getParents returns all the nodes with edges leading into work
Map<String,List<String>>	getRequiredCounterPrefix()
Set<BaseWork>	getRoots() getRoots returns all nodes that do not have a parent.
Map<String,BaseWork>	getWorkMap()
void	remove(BaseWork work) remove removes a node from the graph and removes all edges with work as start or end point.
void	setCloneToWork(Map<BaseWork,BaseWork> cloneToWork)
void	setRequiredCounterPrefix(Map<String,List<String>> requiredCounterPrefix)

Methods inherited from class org.apache.hadoop.hive.ql.plan.AbstractOperatorDesc

clone, getOpProps, getStatistics, getTraits, getVectorMode, setOpProps, setStatistics, setTraits, setVectorMode

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

workGraph

protected final Map<BaseWork,List<BaseWork>> workGraph

invertedWorkGraph

protected final Map<BaseWork,List<BaseWork>> invertedWorkGraph

edgeProperties

protected final Map<org.apache.commons.lang3.tuple.Pair<BaseWork,BaseWork>,SparkEdgeProperty> edgeProperties

Constructor Detail

SparkWork

public SparkWork(String name)

Method Detail

getName

public String getName()

getWorkMap

public Map<String,BaseWork> getWorkMap()

Returns:

a map of "vertex name" to BaseWork

getAllWork

public List<BaseWork> getAllWork()

Returns:

a topologically sorted list of BaseWork

getAllWorkUnsorted

public Collection<BaseWork> getAllWorkUnsorted()

addAll

public void addAll(Collection<BaseWork> c)

Add all nodes in the collection without any connections.

addAll

public void addAll(BaseWork[] bws)

Add all nodes in the collection without any connections.

contains

public boolean contains(BaseWork w)

Whether the specified BaseWork is a vertex in this graph

Parameters:

w - the BaseWork to check

Returns:

whether specified BaseWork is in this graph

add

public void add(BaseWork w)

add creates a new node in the graph without any connections

disconnect

public void disconnect(BaseWork a,
              BaseWork b)

disconnect removes an edge between a and b. Both a and b have to be in the graph. If there is no matching edge no change happens.

getRoots

public Set<BaseWork> getRoots()

getRoots returns all nodes that do not have a parent.

getLeaves

public Set<BaseWork> getLeaves()

getLeaves returns all nodes that do not have a child

setRequiredCounterPrefix

public void setRequiredCounterPrefix(Map<String,List<String>> requiredCounterPrefix)

getRequiredCounterPrefix

public Map<String,List<String>> getRequiredCounterPrefix()

getParents

public List<BaseWork> getParents(BaseWork work)

getParents returns all the nodes with edges leading into work

getChildren

public List<BaseWork> getChildren(BaseWork work)

getChildren returns all the nodes with edges leading out of work

remove

public void remove(BaseWork work)

remove removes a node from the graph and removes all edges with work as start or end point. No change to the graph if the node doesn't exist.

getEdgeProperty

public SparkEdgeProperty getEdgeProperty(BaseWork a,
                                BaseWork b)

returns the edge type connecting work a and b

connect

public void connect(BaseWork a,
           BaseWork b,
           SparkEdgeProperty edgeProp)

connect adds an edge between a and b. Both nodes have to be added prior to calling connect.

Parameters:

-

getDependencyMap

public Map<org.apache.hadoop.hive.ql.plan.SparkWork.ComparableName,List<SparkWork.Dependency>> getDependencyMap()

getAllReduceWork

public List<ReduceWork> getAllReduceWork()

Returns:

all reduce works of this spark work, in sorted order.

getCloneToWork

public Map<BaseWork,BaseWork> getCloneToWork()

setCloneToWork

public void setCloneToWork(Map<BaseWork,BaseWork> cloneToWork)