org.apache.orc.impl

Class WriterImpl

java.lang.Object

org.apache.orc.impl.WriterImpl

All Implemented Interfaces:

MemoryManager.Callback, Writer

Direct Known Subclasses:

WriterImpl

public class WriterImpl
extends Object
implements Writer, MemoryManager.Callback

An ORC file writer. The file is divided into stripes, which is the natural unit of work when reading. Each stripe is buffered in memory until the memory reaches the stripe size and then it is written out broken down by columns. Each column is written by a TreeWriter that is specific to that type of column. TreeWriters may have children TreeWriters that handle the sub-types. Each of the TreeWriters writes the column's data as a set of streams. This class is unsynchronized like most Stream objects, so from the creation of an OrcFile and all access to a single instance has to be from a single thread. There are no known cases where these happen between different threads today. Caveat: the MemoryManager is created during WriterOptions create, that has to be confined to a single thread as well.

Field Summary

Fields

Modifier and Type	Field and Description
static String	BASE_TIMESTAMP_STRING
static long	MILLIS_PER_DAY
static int	MILLIS_PER_SECOND
static long	NANOS_PER_MILLI

Constructor Summary

Constructors

Constructor and Description
WriterImpl(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, OrcFile.WriterOptions opts)

Method Summary

Modifier and Type	Method and Description
void	addRowBatch(VectorizedRowBatch batch) Add a row batch to the ORC file.
void	addUserMetadata(String name, ByteBuffer value) Add arbitrary meta-data to the ORC file.
void	appendStripe(byte[] stripe, int offset, int length, StripeInformation stripeInfo, OrcProto.StripeStatistics stripeStatistics) Fast stripe append to ORC file.
void	appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata) When fast stripe append is used for merging ORC stripes, after appending the last stripe from a file, this interface must be used to merge any user metadata.
boolean	checkMemory(double newScale) The writer needs to check its memory usage
void	close() Flush all of the buffers and close the file.
static CompressionCodec	createCodec(CompressionKind kind)
static int	getEstimatedBufferSize(long stripeSize, int numColumns, int bs)
long	getNumberOfRows() Row count gets updated when flushing the stripes.
long	getRawDataSize() Raw data size will be compute when writing the file footer.
TypeDescription	getSchema() Get the schema for this writer
org.apache.hadoop.fs.FSDataOutputStream	getStream()
long	writeIntermediateFooter() Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MILLIS_PER_DAY

public static long MILLIS_PER_DAY

NANOS_PER_MILLI

public static long NANOS_PER_MILLI

MILLIS_PER_SECOND

public static final int MILLIS_PER_SECOND

See Also:

Constant Field Values

BASE_TIMESTAMP_STRING

public static final String BASE_TIMESTAMP_STRING

See Also:

Constant Field Values

Constructor Detail

WriterImpl

public WriterImpl(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path path,
                  OrcFile.WriterOptions opts)
           throws IOException

Throws:

IOException

Method Detail

getEstimatedBufferSize

public static int getEstimatedBufferSize(long stripeSize,
                                         int numColumns,
                                         int bs)

createCodec

public static CompressionCodec createCodec(CompressionKind kind)

checkMemory

public boolean checkMemory(double newScale)
                    throws IOException

Description copied from interface: MemoryManager.Callback

The writer needs to check its memory usage

Specified by:

checkMemory in interface MemoryManager.Callback

Parameters:

newScale - the current scale factor for memory allocations

Returns:

true if the writer was over the limit

Throws:

IOException

getStream

public org.apache.hadoop.fs.FSDataOutputStream getStream()
                                                  throws IOException

Throws:

IOException

getSchema

public TypeDescription getSchema()

Description copied from interface: Writer

Get the schema for this writer

Specified by:

getSchema in interface Writer

Returns:

the file schema

addUserMetadata

public void addUserMetadata(String name,
                            ByteBuffer value)

Description copied from interface: Writer

Add arbitrary meta-data to the ORC file. This may be called at any point until the Writer is closed. If the same key is passed a second time, the second value will replace the first.

Specified by:

addUserMetadata in interface Writer

Parameters:

name - a key to label the data with.

value - the contents of the metadata.

addRowBatch

public void addRowBatch(VectorizedRowBatch batch)
                 throws IOException

Description copied from interface: Writer

Add a row batch to the ORC file.

Specified by:

addRowBatch in interface Writer

Parameters:

batch - the rows to add

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Writer

Flush all of the buffers and close the file. No methods on this writer should be called afterwards.

Specified by:

close in interface Writer

Throws:

IOException

getRawDataSize

public long getRawDataSize()

Raw data size will be compute when writing the file footer. Hence raw data size value will be available only after closing the writer.

Specified by:

getRawDataSize in interface Writer

Returns:

raw data size

getNumberOfRows

public long getNumberOfRows()

Row count gets updated when flushing the stripes. To get accurate row count call this method after writer is closed.

Specified by:

getNumberOfRows in interface Writer

Returns:

row count

writeIntermediateFooter

public long writeIntermediateFooter()
                             throws IOException

Description copied from interface: Writer

Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.

Specified by:

writeIntermediateFooter in interface Writer

Returns:

the offset that would be a valid end location for an ORC file

Throws:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                         int offset,
                         int length,
                         StripeInformation stripeInfo,
                         OrcProto.StripeStatistics stripeStatistics)
                  throws IOException

Description copied from interface: Writer

Fast stripe append to ORC file. This interface is used for fast ORC file merge with other ORC files. When merging, the file to be merged should pass stripe in binary form along with stripe information and stripe statistics. After appending last stripe of a file, use appendUserMetadata() to append any user metadata.

Specified by:

appendStripe in interface Writer

Parameters:

stripe - - stripe as byte array

offset - - offset within byte array

length - - length of stripe within byte array

stripeInfo - - stripe information

stripeStatistics - - stripe statistics (Protobuf objects can be merged directly)

Throws:

IOException

appendUserMetadata

public void appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata)

Description copied from interface: Writer

When fast stripe append is used for merging ORC stripes, after appending the last stripe from a file, this interface must be used to merge any user metadata.

Specified by:

appendUserMetadata in interface Writer

Parameters:

userMetadata - - user metadata