org.apache.reef.io.data.loading.impl

Class InputFormatLoadingService<K,V>

java.lang.Object

org.apache.reef.io.data.loading.impl.InputFormatLoadingService<K,V>

All Implemented Interfaces:

DataLoadingService

public class InputFormatLoadingService<K,V>
extends Object
implements DataLoadingService

An implementation of DataLoadingService that uses the Hadoop InputFormat to find partitions of data & request resources.

The InputFormat is injected using a Tang external constructor

It also tries to obtain data locality in a greedy fashion using EvaluatorToPartitionMapper

Constructor Summary

Constructors

Constructor and Description
InputFormatLoadingService(org.apache.hadoop.mapred.InputFormat<K,V> inputFormat, org.apache.hadoop.mapred.JobConf jobConf, int numberOfDesiredSplits, boolean inMemory, String inputFormatClass, String inputPath)

Method Summary

Methods

Modifier and Type	Method and Description
String	getComputeContextIdPrefix()
Configuration	getContextConfiguration(AllocatedEvaluator allocatedEvaluator)
int	getNumberOfPartitions() Access to the number of partitions suggested by this DataSource.
Configuration	getServiceConfiguration(AllocatedEvaluator allocatedEvaluator)
boolean	isComputeContext(ActiveContext context)
boolean	isDataLoadedContext(ActiveContext context) Distinguishes data loaded contexts from compute contexts.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InputFormatLoadingService

@Inject
public InputFormatLoadingService(org.apache.hadoop.mapred.InputFormat<K,V> inputFormat,
                                org.apache.hadoop.mapred.JobConf jobConf,
                                int numberOfDesiredSplits,
                                boolean inMemory,
                                String inputFormatClass,
                                String inputPath)

Method Detail

getNumberOfPartitions

public int getNumberOfPartitions()

Description copied from interface: DataLoadingService

Access to the number of partitions suggested by this DataSource.

Specified by:

getNumberOfPartitions in interface DataLoadingService

Returns:

the number of partitions suggested by this DataSource.

getContextConfiguration

public Configuration getContextConfiguration(AllocatedEvaluator allocatedEvaluator)

Specified by:

getContextConfiguration in interface DataLoadingService

Returns:

the context configuration for the given Evaluator.

getServiceConfiguration

public Configuration getServiceConfiguration(AllocatedEvaluator allocatedEvaluator)

Specified by:

getServiceConfiguration in interface DataLoadingService

Returns:

the service configuration for the given Evaluator.

getComputeContextIdPrefix

public String getComputeContextIdPrefix()

Specified by:

getComputeContextIdPrefix in interface DataLoadingService

Returns:

Return the prefix to be used to enumerate context ids for compute requests fired other than the data load contexts.

isComputeContext

public boolean isComputeContext(ActiveContext context)

Specified by:

isComputeContext in interface DataLoadingService

Returns:

true if this is a computation context, false otherwise. (e.g. this is a data loading context).

isDataLoadedContext

public boolean isDataLoadedContext(ActiveContext context)

Description copied from interface: DataLoadingService

Distinguishes data loaded contexts from compute contexts.

Specified by:

isDataLoadedContext in interface DataLoadingService

Returns:

true if this context has been loaded with data.