org.apache.crunch.impl.spark
Class SparkRuntime
java.lang.Object
com.google.common.util.concurrent.AbstractFuture<PipelineResult>
org.apache.crunch.impl.spark.SparkRuntime
- All Implemented Interfaces:
- com.google.common.util.concurrent.ListenableFuture<PipelineResult>, Future<PipelineResult>, PipelineExecution
public class SparkRuntime
- extends com.google.common.util.concurrent.AbstractFuture<PipelineResult>
- implements PipelineExecution
Methods inherited from class com.google.common.util.concurrent.AbstractFuture |
addListener, cancel, isCancelled, isDone |
Methods inherited from interface com.google.common.util.concurrent.ListenableFuture |
addListener |
SparkRuntime
public SparkRuntime(SparkPipeline pipeline,
org.apache.spark.api.java.JavaSparkContext sparkContext,
org.apache.hadoop.conf.Configuration conf,
Map<PCollectionImpl<?>,Set<Target>> outputTargets,
Map<PCollectionImpl<?>,org.apache.crunch.materialize.MaterializableIterable> toMaterialize,
Map<PCollection<?>,org.apache.spark.storage.StorageLevel> toCache)
setCombineFn
public void setCombineFn(CombineFn combineFn)
getCombineFn
public CombineFn getCombineFn()
execute
public SparkRuntime execute()
getSparkContext
public org.apache.spark.api.java.JavaSparkContext getSparkContext()
getRuntimeContext
public SparkRuntimeContext getRuntimeContext()
getConfiguration
public org.apache.hadoop.conf.Configuration getConfiguration()
isValid
public boolean isValid(org.apache.spark.api.java.JavaRDDLike<?,?> rdd)
getStorageLevel
public org.apache.spark.storage.StorageLevel getStorageLevel(PCollection<?> pcollection)
getPlanDotFile
public String getPlanDotFile()
- Description copied from interface:
PipelineExecution
- Returns the .dot file that allows a client to graph the Crunch execution plan for this
pipeline.
- Specified by:
getPlanDotFile
in interface PipelineExecution
waitFor
public void waitFor(long timeout,
TimeUnit timeUnit)
throws InterruptedException
- Description copied from interface:
PipelineExecution
- Blocks until pipeline completes or the specified waiting time elapsed.
- Specified by:
waitFor
in interface PipelineExecution
- Throws:
InterruptedException
waitUntilDone
public void waitUntilDone()
throws InterruptedException
- Description copied from interface:
PipelineExecution
- Blocks until pipeline completes, i.e.
SUCCEEDED
, FAILED
or KILLED
.
- Specified by:
waitUntilDone
in interface PipelineExecution
- Throws:
InterruptedException
get
public PipelineResult get()
throws InterruptedException,
ExecutionException
- Specified by:
get
in interface Future<PipelineResult>
- Overrides:
get
in class com.google.common.util.concurrent.AbstractFuture<PipelineResult>
- Throws:
InterruptedException
ExecutionException
get
public PipelineResult get(long timeout,
TimeUnit unit)
throws InterruptedException,
TimeoutException,
ExecutionException
- Specified by:
get
in interface Future<PipelineResult>
- Overrides:
get
in class com.google.common.util.concurrent.AbstractFuture<PipelineResult>
- Throws:
InterruptedException
TimeoutException
ExecutionException
getStatus
public PipelineExecution.Status getStatus()
- Specified by:
getStatus
in interface PipelineExecution
getResult
public PipelineResult getResult()
- Description copied from interface:
PipelineExecution
- Retrieve the result of a pipeline if it has been completed, otherwise
null
.
- Specified by:
getResult
in interface PipelineExecution
kill
public void kill()
throws InterruptedException
- Description copied from interface:
PipelineExecution
- Kills the pipeline if it is running, no-op otherwise.
This method only delivers a kill signal to the pipeline, and does not guarantee the pipeline exits on return.
To wait for completely exits, use
PipelineExecution.waitUntilDone()
after this call.
- Specified by:
kill
in interface PipelineExecution
- Throws:
InterruptedException
Copyright © 2014 The Apache Software Foundation. All Rights Reserved.