org.apache.hadoop.hive.ql.io.orc

Class ReaderImpl

java.lang.Object

org.apache.hadoop.hive.ql.io.orc.ReaderImpl

All Implemented Interfaces:

Reader

public class ReaderImpl
extends Object
implements Reader

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.ql.io.orc.Reader

Reader.Options

Field Summary

Fields

Modifier and Type	Field and Description
protected int	bufferSize
protected CompressionCodec	codec
protected CompressionKind	compressionKind
protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.hadoop.fs.FileSystem	fileSystem
protected OrcProto.Footer	footer
protected org.apache.hadoop.fs.Path	path

Constructor Summary

Constructors

Constructor and Description
ReaderImpl(org.apache.hadoop.fs.Path path, OrcFile.ReaderOptions options) Constructor that let's the user specify additional options.

Method Summary

Methods

Modifier and Type	Method and Description
CompressionKind	getCompression() Get the compression kind.
int	getCompressionSize() Get the buffer size for the compression.
long	getContentLength() Get the length of the file.
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.FileMetaInfo	getFileMetaInfo()
OrcFile.Version	getFileVersion() Get the file format version.
Metadata	getMetadata() Get the metadata information like stripe level column statistics etc.
List<String>	getMetadataKeys() Get the user metadata keys.
ByteBuffer	getMetadataValue(String key) Get a user metadata value.
long	getNumberOfRows() Get the number of rows in the file.
ObjectInspector	getObjectInspector() Get the object inspector for looking at the objects.
List<OrcProto.UserMetadataItem>	getOrcProtoUserMetadata()
long	getRawDataSize() Get the deserialized data size of the file
long	getRawDataSizeOfColumns(List<String> colNames) Get the deserialized data size of the specified columns
int	getRowIndexStride() Get the number of rows per a entry in the row index.
ColumnStatistics[]	getStatistics() Get the statistics about the columns in the file.
List<StripeInformation>	getStripes() Get the list of stripes.
List<OrcProto.Type>	getTypes() Get the list of types contained in the file.
OrcFile.WriterVersion	getWriterVersion() Get the version of the writer of this file.
boolean	hasMetadataValue(String key) Did the user set the given metadata value.
MetadataReader	metadata()
RecordReader	rows() Create a RecordReader that reads everything with the default options.
RecordReader	rows(boolean[] include) Create a RecordReader that will scan the entire file.
RecordReader	rows(long offset, long length, boolean[] include) Create a RecordReader that will start reading at the first stripe after offset up to the stripe that starts at offset + length.
RecordReader	rows(long offset, long length, boolean[] include, SearchArgument sarg, String[] columnNames) Create a RecordReader that will read a section of a file.
RecordReader	rowsOptions(Reader.Options options) Create a RecordReader that uses the options given.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fileSystem

protected final org.apache.hadoop.fs.FileSystem fileSystem

path

protected final org.apache.hadoop.fs.Path path

compressionKind

protected final CompressionKind compressionKind

codec

protected final CompressionCodec codec

bufferSize

protected final int bufferSize

footer

protected final OrcProto.Footer footer

conf

protected final org.apache.hadoop.conf.Configuration conf

Constructor Detail

ReaderImpl

public ReaderImpl(org.apache.hadoop.fs.Path path,
          OrcFile.ReaderOptions options)
           throws IOException

Constructor that let's the user specify additional options.

Parameters:

path - pathname for file

options - options for reading

Throws:

IOException

Method Detail

getNumberOfRows

public long getNumberOfRows()

Description copied from interface: Reader

Get the number of rows in the file.

Specified by:

getNumberOfRows in interface Reader

Returns:

the number of rows

getMetadataKeys

public List<String> getMetadataKeys()

Description copied from interface: Reader

Get the user metadata keys.

Specified by:

getMetadataKeys in interface Reader

Returns:

the set of metadata keys

getMetadataValue

public ByteBuffer getMetadataValue(String key)

Description copied from interface: Reader

Get a user metadata value.

Specified by:

getMetadataValue in interface Reader

Parameters:

key - a key given by the user

Returns:

the bytes associated with the given key

hasMetadataValue

public boolean hasMetadataValue(String key)

Description copied from interface: Reader

Did the user set the given metadata value.

Specified by:

hasMetadataValue in interface Reader

Parameters:

key - the key to check

Returns:

true if the metadata value was set

getCompression

public CompressionKind getCompression()

Description copied from interface: Reader

Get the compression kind.

Specified by:

getCompression in interface Reader

Returns:

the kind of compression in the file

getCompressionSize

public int getCompressionSize()

Description copied from interface: Reader

Get the buffer size for the compression.

Specified by:

getCompressionSize in interface Reader

Returns:

number of bytes to buffer for the compression codec.

getStripes

public List<StripeInformation> getStripes()

Description copied from interface: Reader

Get the list of stripes.

Specified by:

getStripes in interface Reader

Returns:

the information about the stripes in order

getObjectInspector

public ObjectInspector getObjectInspector()

Description copied from interface: Reader

Get the object inspector for looking at the objects.

Specified by:

getObjectInspector in interface Reader

Returns:

an object inspector for each row returned

getContentLength

public long getContentLength()

Description copied from interface: Reader

Get the length of the file.

Specified by:

getContentLength in interface Reader

Returns:

the number of bytes in the file

getTypes

public List<OrcProto.Type> getTypes()

Description copied from interface: Reader

Get the list of types contained in the file. The root type is the first type in the list.

Specified by:

getTypes in interface Reader

Returns:

the list of flattened types

getFileVersion

public OrcFile.Version getFileVersion()

Description copied from interface: Reader

Get the file format version.

Specified by:

getFileVersion in interface Reader

getWriterVersion

public OrcFile.WriterVersion getWriterVersion()

Description copied from interface: Reader

Get the version of the writer of this file.

Specified by:

getWriterVersion in interface Reader

getRowIndexStride

public int getRowIndexStride()

Description copied from interface: Reader

Get the number of rows per a entry in the row index.

Specified by:

getRowIndexStride in interface Reader

Returns:

the number of rows per an entry in the row index or 0 if there is no row index.

getStatistics

public ColumnStatistics[] getStatistics()

Description copied from interface: Reader

Get the statistics about the columns in the file.

Specified by:

getStatistics in interface Reader

Returns:

the information about the column

getFileMetaInfo

public org.apache.hadoop.hive.ql.io.orc.ReaderImpl.FileMetaInfo getFileMetaInfo()

rows

public RecordReader rows()
                  throws IOException

Description copied from interface: Reader

Create a RecordReader that reads everything with the default options.

Specified by:

rows in interface Reader

Returns:

a new RecordReader

Throws:

IOException

rowsOptions

public RecordReader rowsOptions(Reader.Options options)
                         throws IOException

Description copied from interface: Reader

Create a RecordReader that uses the options given. This method can't be named rows, because many callers used rows(null) before the rows() method was introduced.

Specified by:

rowsOptions in interface Reader

Parameters:

options - the options to read with

Returns:

a new RecordReader

Throws:

IOException

rows

public RecordReader rows(boolean[] include)
                  throws IOException

Description copied from interface: Reader

Create a RecordReader that will scan the entire file. This is a legacy method and rowsOptions is preferred.

Specified by:

rows in interface Reader

Parameters:

include - true for each column that should be included

Returns:

A new RecordReader

Throws:

IOException

rows

public RecordReader rows(long offset,
                long length,
                boolean[] include)
                  throws IOException

Description copied from interface: Reader

Create a RecordReader that will start reading at the first stripe after offset up to the stripe that starts at offset + length. This is intended to work with MapReduce's FileInputFormat where divisions are picked blindly, but they must cover all of the rows. This is a legacy method and rowsOptions is preferred.

Specified by:

rows in interface Reader

Parameters:

offset - a byte offset in the file

length - a number of bytes in the file

include - true for each column that should be included

Returns:

a new RecordReader that will read the specified rows.

Throws:

IOException

rows

public RecordReader rows(long offset,
                long length,
                boolean[] include,
                SearchArgument sarg,
                String[] columnNames)
                  throws IOException

Description copied from interface: Reader

Create a RecordReader that will read a section of a file. It starts reading at the first stripe after the offset and continues to the stripe that starts at offset + length. It also accepts a list of columns to read and a search argument. This is a legacy method and rowsOptions is preferred.

Specified by:

rows in interface Reader

Parameters:

offset - the minimum offset of the first stripe to read

length - the distance from offset of the first address to stop reading at

include - true for each column that should be included

sarg - a search argument that limits the rows that should be read.

columnNames - the names of the included columns

Returns:

the record reader for the rows

Throws:

IOException

getRawDataSize

public long getRawDataSize()

Description copied from interface: Reader

Get the deserialized data size of the file

Specified by:

getRawDataSize in interface Reader

Returns:

raw data size

getRawDataSizeOfColumns

public long getRawDataSizeOfColumns(List<String> colNames)

Description copied from interface: Reader

Get the deserialized data size of the specified columns

Specified by:

getRawDataSizeOfColumns in interface Reader

Returns:

raw data size of columns

getMetadata

public Metadata getMetadata()
                     throws IOException

Description copied from interface: Reader

Get the metadata information like stripe level column statistics etc.

Specified by:

getMetadata in interface Reader

Returns:

the information about the column

Throws:

IOException

getOrcProtoUserMetadata

public List<OrcProto.UserMetadataItem> getOrcProtoUserMetadata()

metadata

public MetadataReader metadata()
                        throws IOException

Specified by:

metadata in interface Reader

Throws:

IOException