org.apache.hadoop.hive.ql.io

Class RCFile.Writer

java.lang.Object

org.apache.hadoop.hive.ql.io.RCFile.Writer

Enclosing class:

RCFile

public static class RCFile.Writer
extends Object

Write KeyBuffer/ValueBuffer pairs to a RCFile. RCFile's format is compatible with SequenceFile's.

Field Summary

Fields

Modifier and Type	Field and Description
static String	COLUMNS_BUFFER_SIZE_CONF_STR

Constructor Summary

Constructors

Constructor and Description
RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name) Constructs a RCFile Writer.
RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.io.SequenceFile.Metadata metadata, org.apache.hadoop.io.compress.CompressionCodec codec) Constructs a RCFile Writer.
RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.io.compress.CompressionCodec codec) Constructs a RCFile Writer.
RCFile.Writer(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path name, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.io.SequenceFile.Metadata metadata, org.apache.hadoop.io.compress.CompressionCodec codec) Constructs a RCFile Writer.

Method Summary

Methods

Modifier and Type	Method and Description
void	append(org.apache.hadoop.io.Writable val) Append a row of values.
void	close()
void	flushBlock(RCFile.KeyBuffer keyBuffer, RCFile.ValueBuffer valueBuffer, int recordLen, int keyLength, int compressedKeyLen) flush a block out without doing anything except compressing the key part.
org.apache.hadoop.io.compress.CompressionCodec	getCompressionCodec() Deprecated.
long	getLength()
void	sync() create a sync point.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COLUMNS_BUFFER_SIZE_CONF_STR

public static String COLUMNS_BUFFER_SIZE_CONF_STR

Constructor Detail

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
             org.apache.hadoop.conf.Configuration conf,
             org.apache.hadoop.fs.Path name)
              throws IOException

Constructs a RCFile Writer.

Throws:

IOException

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
             org.apache.hadoop.conf.Configuration conf,
             org.apache.hadoop.fs.Path name,
             org.apache.hadoop.util.Progressable progress,
             org.apache.hadoop.io.compress.CompressionCodec codec)
              throws IOException

Constructs a RCFile Writer.

Parameters:

fs - the file system used

conf - the configuration file

name - the file name

Throws:

IOException

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
             org.apache.hadoop.conf.Configuration conf,
             org.apache.hadoop.fs.Path name,
             org.apache.hadoop.util.Progressable progress,
             org.apache.hadoop.io.SequenceFile.Metadata metadata,
             org.apache.hadoop.io.compress.CompressionCodec codec)
              throws IOException

Constructs a RCFile Writer.

Parameters:

fs - the file system used

conf - the configuration file

name - the file name

progress - a progress meter to update as the file is written

metadata - a string to string map in the file header

Throws:

IOException

RCFile.Writer

public RCFile.Writer(org.apache.hadoop.fs.FileSystem fs,
             org.apache.hadoop.conf.Configuration conf,
             org.apache.hadoop.fs.Path name,
             int bufferSize,
             short replication,
             long blockSize,
             org.apache.hadoop.util.Progressable progress,
             org.apache.hadoop.io.SequenceFile.Metadata metadata,
             org.apache.hadoop.io.compress.CompressionCodec codec)
              throws IOException

Constructs a RCFile Writer.

Parameters:

fs - the file system used

conf - the configuration file

name - the file name

bufferSize - the size of the file buffer

replication - the number of replicas for the file

blockSize - the block size of the file

progress - the progress meter for writing the file

metadata - a string to string map in the file header

Throws:

IOException

Method Detail

getLength

public long getLength()
               throws IOException

Throws:

IOException

getCompressionCodec

@Deprecated
public org.apache.hadoop.io.compress.CompressionCodec getCompressionCodec()

Deprecated.

Returns the compression codec of data in this file.

sync

public void sync()
          throws IOException

create a sync point.

Throws:

IOException

append

public void append(org.apache.hadoop.io.Writable val)
            throws IOException

Append a row of values. Currently it only can accept < BytesRefArrayWritable. If its size() is less than the column number in the file, zero bytes are appended for the empty columns. If its size() is greater then the column number in the file, the exceeded columns' bytes are ignored.

Parameters:

val - a BytesRefArrayWritable with the list of serialized columns

Throws:

IOException

flushBlock

public void flushBlock(RCFile.KeyBuffer keyBuffer,
              RCFile.ValueBuffer valueBuffer,
              int recordLen,
              int keyLength,
              int compressedKeyLen)
                throws IOException

flush a block out without doing anything except compressing the key part.

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException