org.apache.hive.spark.client

Interface SparkClient

All Superinterfaces:

Serializable

@InterfaceAudience.Private
public interface SparkClient
extends Serializable

Defines the API for the Spark remote client.

Method Summary

Methods

Modifier and Type	Method and Description
Future<?>	addFile(URI uri) Adds a file to the running remote context.
Future<?>	addJar(URI uri) Adds a jar file to the running remote context.
Future<Integer>	getDefaultParallelism() Get default parallelism.
Future<Integer>	getExecutorCount() Get the count of executors.
<T extends Serializable> Future<T>	run(Job<T> job) Asks the remote context to run a job immediately.
void	stop() Stops the remote context.
<T extends Serializable> JobHandle<T>	submit(Job<T> job) Submits a job for asynchronous execution.

Method Detail

submit

<T extends Serializable> JobHandle<T> submit(Job<T> job)

Submits a job for asynchronous execution.

Parameters:

job - The job to execute.

Returns:

A handle that be used to monitor the job.

run

<T extends Serializable> Future<T> run(Job<T> job)

Asks the remote context to run a job immediately.

Normally, the remote context will queue jobs and execute them based on how many worker threads have been configured. This method will run the submitted job in the same thread processing the RPC message, so that queueing does not apply.

It's recommended that this method only be used to run code that finishes quickly. This avoids interfering with the normal operation of the context.

Note: the JobContext#monitor() functionality is not available when using this method.

Parameters:

job - The job to execute.

Returns:

A future to monitor the result of the job.

stop

void stop()

Stops the remote context. Any pending jobs will be cancelled, and the remote context will be torn down.

addJar

Future<?> addJar(URI uri)

Adds a jar file to the running remote context. Note that the URL should be reachable by the Spark driver process. If running the driver in cluster mode, it may reside on a different host, meaning "file:" URLs have to exist on that node (and not on the client machine).

Parameters:

uri - The location of the jar file.

Returns:

A future that can be used to monitor the operation.

addFile

Future<?> addFile(URI uri)

Adds a file to the running remote context. Note that the URL should be reachable by the Spark driver process. If running the driver in cluster mode, it may reside on a different host, meaning "file:" URLs have to exist on that node (and not on the client machine).

Parameters:

uri - The location of the file.

Returns:

A future that can be used to monitor the operation.

getExecutorCount

Future<Integer> getExecutorCount()

Get the count of executors.

getDefaultParallelism

Future<Integer> getDefaultParallelism()

Get default parallelism. For standalone mode, this can be used to get total number of cores.