org.apache.hadoop.hive.ql.io.orc

Class WriterImpl

java.lang.Object

org.apache.hadoop.hive.ql.io.orc.WriterImpl

All Implemented Interfaces:

MemoryManager.Callback, Writer

public class WriterImpl
extends Object
implements Writer, MemoryManager.Callback

An ORC file writer. The file is divided into stripes, which is the natural unit of work when reading. Each stripe is buffered in memory until the memory reaches the stripe size and then it is written out broken down by columns. Each column is written by a TreeWriter that is specific to that type of column. TreeWriters may have children TreeWriters that handle the sub-types. Each of the TreeWriters writes the column's data as a set of streams. This class is synchronized so that multi-threaded access is ok. In particular, because the MemoryManager is shared between writers, this class assumes that checkMemory may be called from a separate thread.

Method Summary

Methods

Modifier and Type	Method and Description
void	addRow(Object row) Add a row to the ORC file.
void	addUserMetadata(String name, ByteBuffer value) Add arbitrary meta-data to the ORC file.
void	appendStripe(byte[] stripe, int offset, int length, StripeInformation stripeInfo, OrcProto.StripeStatistics stripeStatistics) Fast stripe append to ORC file.
void	appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata) When fast stripe append is used for merging ORC stripes, after appending the last stripe from a file, this interface must be used to merge any user metadata.
boolean	checkMemory(double newScale) The writer needs to check its memory usage
void	close() Flush all of the buffers and close the file.
static CompressionCodec	createCodec(CompressionKind kind)
long	getNumberOfRows() Row count gets updated when flushing the stripes.
long	getRawDataSize() Raw data size will be compute when writing the file footer.
long	writeIntermediateFooter() Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createCodec

public static CompressionCodec createCodec(CompressionKind kind)

checkMemory

public boolean checkMemory(double newScale)
                    throws IOException

Description copied from interface: MemoryManager.Callback

The writer needs to check its memory usage

Specified by:

checkMemory in interface MemoryManager.Callback

Parameters:

newScale - the current scale factor for memory allocations

Returns:

true if the writer was over the limit

Throws:

IOException

addUserMetadata

public void addUserMetadata(String name,
                   ByteBuffer value)

Description copied from interface: Writer

Add arbitrary meta-data to the ORC file. This may be called at any point until the Writer is closed. If the same key is passed a second time, the second value will replace the first.

Specified by:

addUserMetadata in interface Writer

Parameters:

name - a key to label the data with.

value - the contents of the metadata.

addRow

public void addRow(Object row)
            throws IOException

Description copied from interface: Writer

Add a row to the ORC file.

Specified by:

addRow in interface Writer

Parameters:

row - the row to add

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Writer

Flush all of the buffers and close the file. No methods on this writer should be called afterwards.

Specified by:

close in interface Writer

Throws:

IOException

getRawDataSize

public long getRawDataSize()

Raw data size will be compute when writing the file footer. Hence raw data size value will be available only after closing the writer.

Specified by:

getRawDataSize in interface Writer

Returns:

raw data size

getNumberOfRows

public long getNumberOfRows()

Row count gets updated when flushing the stripes. To get accurate row count call this method after writer is closed.

Specified by:

getNumberOfRows in interface Writer

Returns:

row count

writeIntermediateFooter

public long writeIntermediateFooter()
                             throws IOException

Description copied from interface: Writer

Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.

Specified by:

writeIntermediateFooter in interface Writer

Returns:

the offset that would be a valid end location for an ORC file

Throws:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                int offset,
                int length,
                StripeInformation stripeInfo,
                OrcProto.StripeStatistics stripeStatistics)
                  throws IOException

Description copied from interface: Writer

Fast stripe append to ORC file. This interface is used for fast ORC file merge with other ORC files. When merging, the file to be merged should pass stripe in binary form along with stripe information and stripe statistics. After appending last stripe of a file, use appendUserMetadata() to append any user metadata.

Specified by:

appendStripe in interface Writer

Parameters:

stripe - - stripe as byte array

offset - - offset within byte array

length - - length of stripe within byte array

stripeInfo - - stripe information

stripeStatistics - - stripe statistics (Protobuf objects can be merged directly)

Throws:

IOException

appendUserMetadata

public void appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata)

Description copied from interface: Writer

When fast stripe append is used for merging ORC stripes, after appending the last stripe from a file, this interface must be used to merge any user metadata.

Specified by:

appendUserMetadata in interface Writer

Parameters:

userMetadata - - user metadata