org.apache.tez.runtime.library.conf

Interface OrderedGroupedKVInputConfig.SpecificConfigBuilder<T>

All Known Implementing Classes:

OrderedGroupedKVInputConfig.Builder, OrderedGroupedKVInputConfig.SpecificBuilder

Enclosing class:

OrderedGroupedKVInputConfig

@InterfaceAudience.Private
public static interface OrderedGroupedKVInputConfig.SpecificConfigBuilder<T>

Configure parameters which are specific to the Input.

Method Summary

Methods

Modifier and Type	Method and Description
T	setAdditionalConfiguration(Map<String,String> confMap) Used to set additional configuration parameters which are not set via API methods.
T	setAdditionalConfiguration(String key, String value) Used to set additional configuration parameters which are not set via API methods.
T	setCombiner(String combinerClassName) Configure the combiner class
T	setCombiner(String combinerClassName, Map<String,String> combinerConf) Configure the combiner class and it's associated configuration (specified as key-value pairs).
T	setFromConfiguration(org.apache.hadoop.conf.Configuration conf) Used to build out a configuration from an existing Hadoop Configuration.
T	setMaxSingleMemorySegmentFraction(float maxSingleSegmentFraction) Sets a size limit on the maximum segment size to be shuffled to disk.
T	setMemToMemMerger(boolean enable) Enable the memory to memory merger
T	setMergeFraction(float mergeFraction) Configure the point at which in memory segments will be merged.
T	setPostMergeBufferFraction(float postMergeBufferFraction) Sets the buffer fraction, as a fraction of container size, to be used after the fetch and merge are complete.
T	setShuffleBufferFraction(float shuffleBufferFraction) Sets the buffer fraction, as a fraction of container size, to be used while fetching remote data.
T	useLegacyInput() Specifies whether the legacy version of this input should be used.

Method Detail

useLegacyInput

T useLegacyInput()

Specifies whether the legacy version of this input should be used.

Returns:

instance of the current builder

setShuffleBufferFraction

T setShuffleBufferFraction(float shuffleBufferFraction)

Sets the buffer fraction, as a fraction of container size, to be used while fetching remote data.

Parameters:

shuffleBufferFraction - fraction of container size

Returns:

instance of the current builder

setPostMergeBufferFraction

T setPostMergeBufferFraction(float postMergeBufferFraction)

Sets the buffer fraction, as a fraction of container size, to be used after the fetch and merge are complete. This buffer is used to cache merged data and avoids writing it out to disk.

Parameters:

postMergeBufferFraction - fraction of container size

Returns:

instance of the current builder

setMaxSingleMemorySegmentFraction

T setMaxSingleMemorySegmentFraction(float maxSingleSegmentFraction)

Sets a size limit on the maximum segment size to be shuffled to disk. This is a fraction of the shuffle buffer.

Parameters:

maxSingleSegmentFraction - fraction of memory determined by ShuffleBufferFraction

Returns:

instance of the current builder

setMemToMemMerger

T setMemToMemMerger(boolean enable)

Enable the memory to memory merger

Parameters:

enable - whether to enable the memory to memory merger

Returns:

instance of the current builder

setMergeFraction

T setMergeFraction(float mergeFraction)

Configure the point at which in memory segments will be merged. This is specified as a fraction of the shuffle buffer.

Parameters:

mergeFraction - fraction of memory determined by ShuffleBufferFraction, which when filled, will trigger a merge

Returns:

instance of the current builder

setCombiner

T setCombiner(String combinerClassName)

Configure the combiner class

Parameters:

combinerClassName - the combiner class name

Returns:

instance of the current builder

setCombiner

T setCombiner(String combinerClassName,
            @Nullable
            Map<String,String> combinerConf)

Configure the combiner class and it's associated configuration (specified as key-value pairs). This method should only be used if the combiner requires some specific configuration. setCombiner(String) is the preferred method for setting a combiner.

Parameters:

combinerClassName - the combiner class name

combinerConf - the combiner configuration. This can be null, and otherwise is a Map of key-value pairs. The keys should be limited to the ones required by the combiner.

Returns:

instance of the current builder

setAdditionalConfiguration

T setAdditionalConfiguration(String key,
                           String value)

Used to set additional configuration parameters which are not set via API methods. This is primarily meant for rarely used configuration options such as IFile read-ahead, configuring the number of parallel files to merge etc.

Parameters:

key - the key to set

value - the corresponding value

Returns:

this object for further chained method calls

setAdditionalConfiguration

T setAdditionalConfiguration(Map<String,String> confMap)

Used to set additional configuration parameters which are not set via API methods. This is primarily meant for rarely used configuration options such as IFile read-ahead, configuring the number of parallel files to merge etc.

Additionally keys set via this method are made available to the combiner.

Parameters:

confMap - map of configuration key-value pairs

Returns:

this object for further chained method calls

setFromConfiguration

@InterfaceAudience.LimitedPrivate(value="hive, pig")
T setFromConfiguration(org.apache.hadoop.conf.Configuration conf)

Used to build out a configuration from an existing Hadoop Configuration. This is a private API is present only for compatibility and ease of use for existing systems which rely heavily on Configuration.

Parameters:

conf -

Returns:

this object for further chained method calls