org.apache.tez.runtime.library.common.task.local.output

Class TezTaskOutput

java.lang.Object

org.apache.tez.runtime.library.common.task.local.output.TezTaskOutput

Direct Known Subclasses:

TezLocalTaskOutputFiles, TezTaskOutputFiles

@InterfaceAudience.Private
@InterfaceStability.Unstable
public abstract class TezTaskOutput
extends Object

Manipulate the working area for the transient store for components in tez-runtime-library This class is used by Inputs and Outputs in tez-runtime-library to identify the directories that they need to write to / read from for intermediate files.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected String	uniqueId

Constructor Summary

Constructors

Constructor and Description
TezTaskOutput(org.apache.hadoop.conf.Configuration conf, String uniqueId)

Method Summary

Methods

Modifier and Type	Method and Description
abstract org.apache.hadoop.fs.Path	getInputFile(InputAttemptIdentifier attemptIdentifier) Return a local input file created earlier TODO: TEZ-1855.
abstract org.apache.hadoop.fs.Path	getInputFileForWrite(int srcIdentifier, long size) Create a local input file name.
abstract org.apache.hadoop.fs.Path	getOutputFile() Return the path to local output file created earlier.
abstract org.apache.hadoop.fs.Path	getOutputFileForWrite() Create a local output file name.
abstract org.apache.hadoop.fs.Path	getOutputFileForWrite(long size) Create a local output file name.
abstract org.apache.hadoop.fs.Path	getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local output file name on the same volume.
abstract org.apache.hadoop.fs.Path	getOutputIndexFile() Return the path to a local output index file created earlier TODO TEZ-1855: Remove this.
abstract org.apache.hadoop.fs.Path	getOutputIndexFileForWrite(long size) Create a local output index file name.
abstract org.apache.hadoop.fs.Path	getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local output index file name on the same volume.
abstract org.apache.hadoop.fs.Path	getSpillFile(int spillNumber) Return a local output spill file created earlier.
abstract org.apache.hadoop.fs.Path	getSpillFileForWrite(int spillNumber, long size) Create a local output spill file name.
abstract String	getSpillFileName(int spillNum) Construct a spill file name, given a spill number
abstract org.apache.hadoop.fs.Path	getSpillIndexFile(int spillNumber) Return a local output spill index file created earlier TODO TEZ-1855: Remove this.
abstract org.apache.hadoop.fs.Path	getSpillIndexFileForWrite(int spillNumber, long size) Create a local output spill index file name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

conf

protected final org.apache.hadoop.conf.Configuration conf

uniqueId

protected final String uniqueId

Constructor Detail

TezTaskOutput

public TezTaskOutput(org.apache.hadoop.conf.Configuration conf,
             String uniqueId)

Parameters:

conf - the configuration from which local-dirs will be picked up

uniqueId - a unique identifier for the specific input / output. This is expected to be unique for all the Inputs / Outputs within a container - i.e. even if the container is used for multiple tasks, this id should be unique for inputs / outputs spanning across tasks. This is also expected to be unique across all tasks for a vertex.

Method Detail

getOutputFile

public abstract org.apache.hadoop.fs.Path getOutputFile()
                                                 throws IOException

Return the path to local output file created earlier. TODO TEZ-1855: Remove this. Leads to an extra localdir scan just to update counters.

Returns:

path the path of the local output file

Throws:

IOException

getOutputFileForWrite

public abstract org.apache.hadoop.fs.Path getOutputFileForWrite(long size)
                                                         throws IOException

Create a local output file name.

Parameters:

size - the size of the file

Returns:

path the path to write to

Throws:

IOException

getOutputFileForWrite

public abstract org.apache.hadoop.fs.Path getOutputFileForWrite()
                                                         throws IOException

Create a local output file name. This method is meant to be used *only* if the size of the file is not know up front.

Returns:

path the path to write to

Throws:

IOException

getOutputFileForWriteInVolume

public abstract org.apache.hadoop.fs.Path getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local output file name on the same volume. This is only meant to be used to rename temporary files to their final destination within the same volume.

Returns:

path the path of the output file within the same volume

getOutputIndexFile

public abstract org.apache.hadoop.fs.Path getOutputIndexFile()
                                                      throws IOException

Return the path to a local output index file created earlier TODO TEZ-1855: Remove this. Leads to an additional scan to find empty partitions.

Returns:

path the path of the index file

Throws:

IOException

getOutputIndexFileForWrite

public abstract org.apache.hadoop.fs.Path getOutputIndexFileForWrite(long size)
                                                              throws IOException

Create a local output index file name.

Parameters:

size - the size of the file

Returns:

path the path to write the index file to

Throws:

IOException

getOutputIndexFileForWriteInVolume

public abstract org.apache.hadoop.fs.Path getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local output index file name on the same volume. The intended usage of this method is to write the index file on the same volume as the associated data file.

Returns:

path the path of the index file within the same volume

getSpillFile

public abstract org.apache.hadoop.fs.Path getSpillFile(int spillNumber)
                                                throws IOException

Return a local output spill file created earlier.

Parameters:

spillNumber - the spill number

Returns:

path the path of the previously written spill file corresponding to the spillNumber

Throws:

IOException - // KKK Try removing this. Unnecessary file scans - can be stored in memory instead.

getSpillFileForWrite

public abstract org.apache.hadoop.fs.Path getSpillFileForWrite(int spillNumber,
                                             long size)
                                                        throws IOException

Create a local output spill file name.

Parameters:

spillNumber - the spill number

size - the size of the file

Returns:

path the path to write the spill file for the specific spillNumber

Throws:

IOException

getSpillIndexFile

public abstract org.apache.hadoop.fs.Path getSpillIndexFile(int spillNumber)
                                                     throws IOException

Return a local output spill index file created earlier TODO TEZ-1855: Remove this. Should be possible to cache this instead of requiring a directory scan.

Parameters:

spillNumber - the spill number

Returns:

path the path of the previously written spill index file corresponding to the spillNumber

Throws:

IOException

getSpillIndexFileForWrite

public abstract org.apache.hadoop.fs.Path getSpillIndexFileForWrite(int spillNumber,
                                                  long size)
                                                             throws IOException

Create a local output spill index file name.

Parameters:

spillNumber - the spill number

size - the size of the spill file

Returns:

path the path to write the spill index file for the specific spillNumber

Throws:

IOException

getInputFile

public abstract org.apache.hadoop.fs.Path getInputFile(InputAttemptIdentifier attemptIdentifier)
                                                throws IOException

Return a local input file created earlier TODO: TEZ-1855. Remove this.

Parameters:

attemptIdentifier - The identifier for the source

Returns:

path the path to the input file

Throws:

IOException

getInputFileForWrite

public abstract org.apache.hadoop.fs.Path getInputFileForWrite(int srcIdentifier,
                                             long size)
                                                        throws IOException

Create a local input file name.

Parameters:

srcIdentifier - The identifier for the source

size - the size of the file

Returns:

path the path to the input file.

Throws:

IOException

getSpillFileName

public abstract String getSpillFileName(int spillNum)

Construct a spill file name, given a spill number

Parameters:

spillNum -

Returns:

a spill file name independent of the unique identifier and local directories