org.apache.tez.runtime.library.common.task.local.output

Class TezTaskOutputFiles

java.lang.Object

org.apache.tez.runtime.library.common.task.local.output.TezTaskOutput

org.apache.tez.runtime.library.common.task.local.output.TezTaskOutputFiles

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class TezTaskOutputFiles
extends TezTaskOutput

Manipulate the working area for the transient store for components in tez-runtime-library This class is used by Inputs and Outputs in tez-runtime-library to identify the directories that they need to write to / read from for intermediate files.

Field Summary

Fields inherited from class org.apache.tez.runtime.library.common.task.local.output.TezTaskOutput

conf, uniqueId

Constructor Summary

Constructors

Constructor and Description
TezTaskOutputFiles(org.apache.hadoop.conf.Configuration conf, String uniqueId)

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path	getInputFile(InputAttemptIdentifier attemptIdentifier) Return a local input file created earlier ${appDir}/${uniqueId}_spill_${spillNumber}.out e.g.
org.apache.hadoop.fs.Path	getInputFileForWrite(int srcIdentifier, long size) Create a local input file name.
org.apache.hadoop.fs.Path	getOutputFile() Return the path to local output file created earlier.
org.apache.hadoop.fs.Path	getOutputFileForWrite() Create a local output file name.
org.apache.hadoop.fs.Path	getOutputFileForWrite(long size) Create a local output file name.
org.apache.hadoop.fs.Path	getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local output file name on the same volume.
org.apache.hadoop.fs.Path	getOutputIndexFile() Return the path to a local output index file created earlier ${appDir}/output/${uniqueId}/file.out.index e.g.
org.apache.hadoop.fs.Path	getOutputIndexFileForWrite(long size) Create a local output index file name.
org.apache.hadoop.fs.Path	getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local output index file name on the same volume.
org.apache.hadoop.fs.Path	getSpillFile(int spillNumber) Return a local output spill file created earlier.
org.apache.hadoop.fs.Path	getSpillFileForWrite(int spillNumber, long size) Create a local spill file name.
String	getSpillFileName(int spillNum) Construct a spill file name, given a spill number ${uniqueId}_spill_${spillNumber}.out e.g.
org.apache.hadoop.fs.Path	getSpillIndexFile(int spillNumber) Return a local map spill index file created earlier
org.apache.hadoop.fs.Path	getSpillIndexFileForWrite(int spillNumber, long size) Create a local output spill index file name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TezTaskOutputFiles

public TezTaskOutputFiles(org.apache.hadoop.conf.Configuration conf,
                  String uniqueId)

Method Detail

getOutputFile

public org.apache.hadoop.fs.Path getOutputFile()
                                        throws IOException

Return the path to local output file created earlier. ${appDir}/output/${uniqueId}/file.out e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputFile in class TezTaskOutput

Returns:

path the path of the local output file

Throws:

IOException

getOutputFileForWrite

public org.apache.hadoop.fs.Path getOutputFileForWrite(long size)
                                                throws IOException

Create a local output file name. ${appDir}/output/${uniqueId}/file.out e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputFileForWrite in class TezTaskOutput

Parameters:

size - the size of the file

Returns:

path the path to write to

Throws:

IOException

getOutputFileForWrite

public org.apache.hadoop.fs.Path getOutputFileForWrite()
                                                throws IOException

Create a local output file name. This should *only* be used if the size of the file is not known. Otherwise use the equivalent which accepts a size parameter. ${appDir}/output/${uniqueId}/file.out e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputFileForWrite in class TezTaskOutput

Returns:

path the path to write to

Throws:

IOException

getOutputFileForWriteInVolume

public org.apache.hadoop.fs.Path getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local output file name on the same volume. This is only meant to be used to rename temporary files to their final destination within the same volume. ${appDir}/output/${uniqueId}/file.out e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputFileForWriteInVolume in class TezTaskOutput

Returns:

path the path of the output file within the same volume

getOutputIndexFile

public org.apache.hadoop.fs.Path getOutputIndexFile()
                                             throws IOException

Return the path to a local output index file created earlier ${appDir}/output/${uniqueId}/file.out.index e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out.index The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputIndexFile in class TezTaskOutput

Returns:

path the path of the index file

Throws:

IOException

getOutputIndexFileForWrite

public org.apache.hadoop.fs.Path getOutputIndexFileForWrite(long size)
                                                     throws IOException

Create a local output index file name. ${appDir}/output/${uniqueId}/file.out.index e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out.index The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputIndexFileForWrite in class TezTaskOutput

Parameters:

size - the size of the file

Returns:

path the path to write the index file to

Throws:

IOException

getOutputIndexFileForWriteInVolume

public org.apache.hadoop.fs.Path getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local output index file name on the same volume. The intended usage of this method is to write the index file on the same volume as the associated data file. ${appDir}/output/${uniqueId}/file.out.index e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out.index The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputIndexFileForWriteInVolume in class TezTaskOutput

Returns:

path the path of the index file within the same volume

getSpillFile

public org.apache.hadoop.fs.Path getSpillFile(int spillNumber)
                                       throws IOException

Return a local output spill file created earlier. ${appDir}/${uniqueId}_spill_${spillNumber}.out e.g. application_1418684642047_0006/attempt_1418684642047_0006_1_00_000000_0_10003_spill_0.out

Specified by:

getSpillFile in class TezTaskOutput

Parameters:

spillNumber - the spill number

Returns:

path the path of the previously written spill file corresponding to the spillNumber

Throws:

IOException

getSpillFileForWrite

public org.apache.hadoop.fs.Path getSpillFileForWrite(int spillNumber,
                                             long size)
                                               throws IOException

Create a local spill file name. ${appDir}/${uniqueId}_spill_${spillNumber}.out e.g. application_1418684642047_0006/attempt_1418684642047_0006_1_00_000000_0_10003_spill_0.out

Specified by:

getSpillFileForWrite in class TezTaskOutput

Parameters:

spillNumber - the spill number

size - the size of the spill file

Returns:

path the path to write the spill file for the specific spillNumber

Throws:

IOException

getSpillIndexFile

public org.apache.hadoop.fs.Path getSpillIndexFile(int spillNumber)
                                            throws IOException

Return a local map spill index file created earlier

Specified by:

getSpillIndexFile in class TezTaskOutput

Parameters:

spillNumber - the number

Returns:

path the path of the previously written spill index file corresponding to the spillNumber

Throws:

IOException

getSpillIndexFileForWrite

public org.apache.hadoop.fs.Path getSpillIndexFileForWrite(int spillNumber,
                                                  long size)
                                                    throws IOException

Create a local output spill index file name. ${appDir}/${uniqueId}_spill_${spillNumber}.out.index e.g. application_1418684642047_0006/attempt_1418684642047_0006_1_00_000000_0_10003_spill_0.out.index

Specified by:

getSpillIndexFileForWrite in class TezTaskOutput

Parameters:

spillNumber - the spill number

size - the size of the file

Returns:

path the path to write the spill index file for the specific spillNumber

Throws:

IOException

getInputFile

public org.apache.hadoop.fs.Path getInputFile(InputAttemptIdentifier attemptIdentifier)
                                       throws IOException

Return a local input file created earlier ${appDir}/${uniqueId}_spill_${spillNumber}.out e.g. application_1418684642047_0006/attempt_1418684642047_0006_1_00_000000_0_10004_spill_0.out

Specified by:

getInputFile in class TezTaskOutput

Parameters:

attemptIdentifier - an identifier for a task. The attempt information is ignored.

Returns:

path the path to the input file

Throws:

IOException

getInputFileForWrite

public org.apache.hadoop.fs.Path getInputFileForWrite(int srcIdentifier,
                                             long size)
                                               throws IOException

Create a local input file name. ${appDir}/${uniqueId}_spill_${spillNumber}.out e.g. application_1418684642047_0006/attempt_1418684642047_0006_1_00_000000_0_10004_spill_0.out This is currently equivalent to getSpillFileForWrite. Files are not clobbered due to the uniqueId being different for Outputs / Inputs within the same task (and across tasks)

Specified by:

getInputFileForWrite in class TezTaskOutput

Parameters:

srcIdentifier - The identifier for the source

size - the size of the file

Returns:

path the path to the input file.

Throws:

IOException

getSpillFileName

public String getSpillFileName(int spillNum)

Construct a spill file name, given a spill number ${uniqueId}_spill_${spillNumber}.out e.g. attempt_1418684642047_0006_1_00_000000_0_10004_spill_0.out

Specified by:

getSpillFileName in class TezTaskOutput

Parameters:

spillNum -

Returns:

a spill file name independent of the unique identifier and local directories