org.apache.hadoop.hive.ql.exec

Class FileSinkOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.TerminalOperator<FileSinkDesc>

org.apache.hadoop.hive.ql.exec.FileSinkOperator

All Implemented Interfaces:

Serializable, Cloneable, Node

Direct Known Subclasses:

VectorFileSinkOperator

public class FileSinkOperator
extends TerminalOperator<FileSinkDesc>
implements Serializable

File Sink operator implementation.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileSinkOperator.Counter Counters.
class	FileSinkOperator.FSPaths
static interface	FileSinkOperator.RecordWriter RecordWriter.

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	autoDelete
protected boolean	bDynParts
protected Map<Integer,Integer>	bucketMap
protected String	childSpecPathDynLinkedPartitions
protected long	cntr
protected List<String>	dpColNames
protected DynamicPartitionCtx	dpCtx
protected int	dpStartCol
protected List<String>	dpVals
protected List<Object>	dpWritables
protected boolean	filesCreated
protected org.apache.hadoop.fs.FileSystem	fs
protected FileSinkOperator.FSPaths	fsp
protected HiveOutputFormat<?,?>	hiveOutputFormat
protected boolean	isCollectRWStats
protected boolean	isCompressed
protected boolean	isSkewedStoredAsSubDirectories
protected boolean	isTemporary
protected org.apache.hadoop.mapred.JobConf	jc
protected HiveKey	key
protected ListBucketingCtx	lbCtx
static org.apache.commons.logging.Log	LOG
protected long	logEveryNRows
protected int	maxPartitions
protected boolean	multiFileSpray
protected int	numDynParts
protected long	numRows
protected org.apache.hadoop.fs.Path	parent
protected HivePartitioner<HiveKey,Object>	prtner
protected org.apache.hadoop.io.Writable	recordValue
protected org.apache.hadoop.io.LongWritable	row_count
protected FileSinkOperator.RecordWriter[]	rowOutWriters
protected Serializer	serializer
protected org.apache.hadoop.fs.Path	specPath
protected boolean	statsCollectRawDataSize
protected boolean[]	statsFromRecordWriter
protected int	totalFiles
protected HashMap<String,FileSinkOperator.FSPaths>	valToPaths

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

alias, childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVECOUNTERCREATEDFILES, HIVECOUNTERFATAL, id, inputObjInspectors, isLogDebugEnabled, isLogInfoEnabled, isLogTraceEnabled, operatorId, out, outputObjInspector, parentOperators, PLOG, reporter, state, statsMap

Constructor Summary

Constructors

Constructor and Description
FileSinkOperator()

Method Summary

Methods

Modifier and Type	Method and Description
protected boolean	areAllTrue(boolean[] statsFromRW)
void	augmentPlan() Called during semantic analysis as operators are being added in order to give them a chance to compute any additional plan information needed.
void	checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored, org.apache.hadoop.mapred.JobConf job)
void	closeOp(boolean abort) Operator specific close routine.
protected void	createBucketFiles(FileSinkOperator.FSPaths fsp)
protected void	createBucketForFileIdx(FileSinkOperator.FSPaths fsp, int filesIdx)
protected String	generateListBucketingDirName(Object row) Generate list bucketing directory name from a row.
protected FileSinkOperator.FSPaths	getDynOutPaths(List<String> row, String lbDirName)
String	getName() Implements the getName function for the Node Interface.
static String	getOperatorName()
OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
protected Collection<Future<?>>	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
void	jobCloseOp(org.apache.hadoop.conf.Configuration hconf, boolean success)
protected FileSinkOperator.FSPaths	lookupListBucketingPaths(String lbDirName) Lookup list bucketing path.
void	process(Object row, int tag) Process the row.
protected boolean	updateProgress() Report status to JT so that JT won't kill this task if closing takes too long due to too many files to close and the NN is overloaded.

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, completeInitializationOp, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, endGroup, flush, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, logStats, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setDone, setExecContext, setGroupKeyObject, setId, setInputContext, setInputObjInspectors, setOperatorId, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, startGroup, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

valToPaths

protected transient HashMap<String,FileSinkOperator.FSPaths> valToPaths

numDynParts

protected transient int numDynParts

dpColNames

protected transient List<String> dpColNames

dpCtx

protected transient DynamicPartitionCtx dpCtx

isCompressed

protected transient boolean isCompressed

isTemporary

protected transient boolean isTemporary

parent

protected transient org.apache.hadoop.fs.Path parent

hiveOutputFormat

protected transient HiveOutputFormat<?,?> hiveOutputFormat

specPath

protected transient org.apache.hadoop.fs.Path specPath

childSpecPathDynLinkedPartitions

protected transient String childSpecPathDynLinkedPartitions

dpStartCol

protected transient int dpStartCol

dpVals

protected transient List<String> dpVals

dpWritables

protected transient List<Object> dpWritables

rowOutWriters

protected transient FileSinkOperator.RecordWriter[] rowOutWriters

maxPartitions

protected transient int maxPartitions

lbCtx

protected transient ListBucketingCtx lbCtx

isSkewedStoredAsSubDirectories

protected transient boolean isSkewedStoredAsSubDirectories

statsCollectRawDataSize

protected transient boolean statsCollectRawDataSize

statsFromRecordWriter

protected transient boolean[] statsFromRecordWriter

isCollectRWStats

protected transient boolean isCollectRWStats

numRows

protected transient long numRows

cntr

protected transient long cntr

logEveryNRows

protected transient long logEveryNRows

fs

protected transient org.apache.hadoop.fs.FileSystem fs

serializer

protected transient Serializer serializer

row_count

protected final transient org.apache.hadoop.io.LongWritable row_count

totalFiles

protected transient int totalFiles

multiFileSpray

protected transient boolean multiFileSpray

bucketMap

protected final transient Map<Integer,Integer> bucketMap

prtner

protected transient HivePartitioner<HiveKey,Object> prtner

key

protected final transient HiveKey key

fsp

protected transient FileSinkOperator.FSPaths fsp

bDynParts

protected transient boolean bDynParts

autoDelete

protected transient boolean autoDelete

jc

protected transient org.apache.hadoop.mapred.JobConf jc

filesCreated

protected boolean filesCreated

recordValue

protected org.apache.hadoop.io.Writable recordValue

Constructor Detail

FileSinkOperator

public FileSinkOperator()

Method Detail

initializeOp

protected Collection<Future<?>> initializeOp(org.apache.hadoop.conf.Configuration hconf)
                                      throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<FileSinkDesc>

Throws:

HiveException

createBucketFiles

protected void createBucketFiles(FileSinkOperator.FSPaths fsp)
                          throws HiveException

Throws:

HiveException

createBucketForFileIdx

protected void createBucketForFileIdx(FileSinkOperator.FSPaths fsp,
                          int filesIdx)
                               throws HiveException

Throws:

HiveException

updateProgress

protected boolean updateProgress()

Report status to JT so that JT won't kill this task if closing takes too long due to too many files to close and the NN is overloaded.

Returns:

true if a new progress update is reported, false otherwise.

process

public void process(Object row,
           int tag)
             throws HiveException

Description copied from class: Operator

Process the row.

Specified by:

process in class Operator<FileSinkDesc>

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

areAllTrue

protected boolean areAllTrue(boolean[] statsFromRW)

lookupListBucketingPaths

protected FileSinkOperator.FSPaths lookupListBucketingPaths(String lbDirName)
                                                     throws HiveException

Lookup list bucketing path.

Parameters:

lbDirName -

Returns:

Throws:

HiveException

generateListBucketingDirName

protected String generateListBucketingDirName(Object row)

Generate list bucketing directory name from a row.

Parameters:

row - row to process.

Returns:

directory name.

getDynOutPaths

protected FileSinkOperator.FSPaths getDynOutPaths(List<String> row,
                                      String lbDirName)
                                           throws HiveException

Throws:

HiveException

closeOp

public void closeOp(boolean abort)
             throws HiveException

Description copied from class: Operator

Operator specific close routine. Operators which inherents this class should overwrite this funtion for their specific cleanup routine.

Overrides:

closeOp in class Operator<FileSinkDesc>

Throws:

HiveException

getName

public String getName()

Description copied from class: Operator

Implements the getName function for the Node Interface.

Specified by:

getName in interface Node

Overrides:

getName in class Operator<FileSinkDesc>

Returns:

the name of the operator

getOperatorName

public static String getOperatorName()

jobCloseOp

public void jobCloseOp(org.apache.hadoop.conf.Configuration hconf,
              boolean success)
                throws HiveException

Overrides:

jobCloseOp in class Operator<FileSinkDesc>

Throws:

HiveException

getType

public OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Specified by:

getType in class Operator<FileSinkDesc>

Returns:

OperatorType.*

augmentPlan

public void augmentPlan()

Description copied from class: Operator

Called during semantic analysis as operators are being added in order to give them a chance to compute any additional plan information needed. Does nothing by default.

Overrides:

augmentPlan in class Operator<FileSinkDesc>

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
                    org.apache.hadoop.mapred.JobConf job)
                      throws IOException

Throws:

IOException