org.apache.hadoop.hive.serde2.binarysortable

Class BinarySortableSerDe

java.lang.Object

org.apache.hadoop.hive.serde2.AbstractSerDe

org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe

All Implemented Interfaces:

Deserializer, SerDe, Serializer

public class BinarySortableSerDe
extends AbstractSerDe

BinarySortableSerDe can be used to write data in a way that the data can be compared byte-by-byte with the same order. The data format: NULL: a single byte \0 NON-NULL Primitives: ALWAYS prepend a single byte \1, and then: Boolean: FALSE = \1, TRUE = \2 Byte: flip the sign-bit to make sure negative comes before positive Short: flip the sign-bit to make sure negative comes before positive Int: flip the sign-bit to make sure negative comes before positive Long: flip the sign-bit to make sure negative comes before positive Double: flip the sign-bit for positive double, and all bits for negative double values String: NULL-terminated UTF-8 string, with NULL escaped to \1 \1, and \1 escaped to \1 \2 NON-NULL Complex Types: ALWAYS prepend a single byte \1, and then: Struct: one field by one field. List: \1 followed by each element, and \0 to terminate Map: \1 followed by each key and then each value, and \0 to terminate This SerDe takes an additional parameter SERIALIZATION_SORT_ORDER which is a string containing only "+" and "-". The length of the string should equal to the number of fields in the top-level struct for serialization. "+" means the field should be sorted ascendingly, and "-" means descendingly. The sub fields in the same top-level field will have the same sort order.

Field Summary

Fields

Modifier and Type	Field and Description
static Charset	decimalCharSet
static org.apache.commons.logging.Log	LOG

Fields inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

configErrors

Constructor Summary

Constructors

Constructor and Description
BinarySortableSerDe()

Method Summary

Methods

Modifier and Type	Method and Description
Object	deserialize(org.apache.hadoop.io.Writable blob) Deserialize an object out of a Writable blob.
static org.apache.hadoop.io.Text	deserializeText(InputByteBuffer buffer, boolean invert, org.apache.hadoop.io.Text r)
ObjectInspector	getObjectInspector() Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).
SerDeStats	getSerDeStats() Returns statistics collected when serializing
Class<? extends org.apache.hadoop.io.Writable>	getSerializedClass() Returns the Writable class that would be returned by the serialize method.
boolean[]	getSortOrders()
void	initialize(org.apache.hadoop.conf.Configuration conf, Properties tbl) Initialize the HiveSerializer.
org.apache.hadoop.io.Writable	serialize(Object obj, ObjectInspector objInspector) Serialize an object by navigating inside the Object with the ObjectInspector.
static void	serializeBytes(ByteStream.Output buffer, byte[] data, int length, boolean invert)
static void	serializeBytes(ByteStream.Output buffer, byte[] data, int offset, int length, boolean invert)
static void	serializeDouble(ByteStream.Output buffer, double vd, boolean invert)
static void	serializeFloat(ByteStream.Output buffer, float vf, boolean invert)
static void	serializeHiveDecimal(ByteStream.Output buffer, HiveDecimal dec, boolean invert)
static void	serializeHiveIntervalDayTime(ByteStream.Output buffer, HiveIntervalDayTime intervalDayTime, boolean invert)
static void	serializeHiveIntervalYearMonth(ByteStream.Output buffer, HiveIntervalYearMonth intervalYearMonth, boolean invert)
static void	serializeInt(ByteStream.Output buffer, int v, boolean invert)
static void	serializeLong(ByteStream.Output buffer, long v, boolean invert)
static void	serializeShort(ByteStream.Output buffer, short v, boolean invert)
static void	serializeStruct(ByteStream.Output byteStream, Object[] fieldData, List<ObjectInspector> fieldOis, boolean[] sortableSortOrders)
static void	serializeTimestampWritable(ByteStream.Output buffer, TimestampWritable t, boolean invert)
static void	writeByte(ByteStream.RandomAccessOutput buffer, byte b, boolean invert)

Methods inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

getConfigurationErrors, initialize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

decimalCharSet

public static Charset decimalCharSet

Constructor Detail

BinarySortableSerDe

public BinarySortableSerDe()

Method Detail

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf,
              Properties tbl)
                throws SerDeException

Description copied from class: AbstractSerDe

Initialize the HiveSerializer.

Specified by:

initialize in interface Deserializer

Specified by:

initialize in interface Serializer

Specified by:

initialize in class AbstractSerDe

Parameters:

conf - System properties. Can be null in compile time

tbl - table properties

Throws:

SerDeException

getSerializedClass

public Class<? extends org.apache.hadoop.io.Writable> getSerializedClass()

Description copied from class: AbstractSerDe

Returns the Writable class that would be returned by the serialize method. This is used to initialize SequenceFile header.

Specified by:

getSerializedClass in interface Serializer

Specified by:

getSerializedClass in class AbstractSerDe

getObjectInspector

public ObjectInspector getObjectInspector()
                                   throws SerDeException

Description copied from class: AbstractSerDe

Get the object inspector that can be used to navigate through the internal structure of the Object returned from deserialize(...).

Specified by:

getObjectInspector in interface Deserializer

Specified by:

getObjectInspector in class AbstractSerDe

Throws:

SerDeException

deserialize

public Object deserialize(org.apache.hadoop.io.Writable blob)
                   throws SerDeException

Description copied from class: AbstractSerDe

Deserialize an object out of a Writable blob. In most cases, the return value of this function will be constant since the function will reuse the returned object. If the client wants to keep a copy of the object, the client needs to clone the returned value by calling ObjectInspectorUtils.getStandardObject().

Specified by:

deserialize in interface Deserializer

Specified by:

deserialize in class AbstractSerDe

Parameters:

blob - The Writable object containing a serialized object

Returns:

A Java object representing the contents in the blob.

Throws:

SerDeException

deserializeText

public static org.apache.hadoop.io.Text deserializeText(InputByteBuffer buffer,
                                        boolean invert,
                                        org.apache.hadoop.io.Text r)
                                                 throws IOException

Throws:

IOException

serialize

public org.apache.hadoop.io.Writable serialize(Object obj,
                                      ObjectInspector objInspector)
                                        throws SerDeException

Description copied from class: AbstractSerDe

Serialize an object by navigating inside the Object with the ObjectInspector. In most cases, the return value of this function will be constant since the function will reuse the Writable object. If the client wants to keep a copy of the Writable, the client needs to clone the returned value.

Specified by:

serialize in interface Serializer

Specified by:

serialize in class AbstractSerDe

Throws:

SerDeException

writeByte

public static void writeByte(ByteStream.RandomAccessOutput buffer,
             byte b,
             boolean invert)

serializeBytes

public static void serializeBytes(ByteStream.Output buffer,
                  byte[] data,
                  int length,
                  boolean invert)

serializeBytes

public static void serializeBytes(ByteStream.Output buffer,
                  byte[] data,
                  int offset,
                  int length,
                  boolean invert)

serializeShort

public static void serializeShort(ByteStream.Output buffer,
                  short v,
                  boolean invert)

serializeInt

public static void serializeInt(ByteStream.Output buffer,
                int v,
                boolean invert)

serializeLong

public static void serializeLong(ByteStream.Output buffer,
                 long v,
                 boolean invert)

serializeFloat

public static void serializeFloat(ByteStream.Output buffer,
                  float vf,
                  boolean invert)

serializeDouble

public static void serializeDouble(ByteStream.Output buffer,
                   double vd,
                   boolean invert)

serializeTimestampWritable

public static void serializeTimestampWritable(ByteStream.Output buffer,
                              TimestampWritable t,
                              boolean invert)

serializeHiveIntervalYearMonth

public static void serializeHiveIntervalYearMonth(ByteStream.Output buffer,
                                  HiveIntervalYearMonth intervalYearMonth,
                                  boolean invert)

serializeHiveIntervalDayTime

public static void serializeHiveIntervalDayTime(ByteStream.Output buffer,
                                HiveIntervalDayTime intervalDayTime,
                                boolean invert)

serializeHiveDecimal

public static void serializeHiveDecimal(ByteStream.Output buffer,
                        HiveDecimal dec,
                        boolean invert)

getSerDeStats

public SerDeStats getSerDeStats()

Description copied from class: AbstractSerDe

Returns statistics collected when serializing

Specified by:

getSerDeStats in interface Deserializer

Specified by:

getSerDeStats in interface Serializer

Specified by:

getSerDeStats in class AbstractSerDe

serializeStruct

public static void serializeStruct(ByteStream.Output byteStream,
                   Object[] fieldData,
                   List<ObjectInspector> fieldOis,
                   boolean[] sortableSortOrders)
                            throws SerDeException

Throws:

SerDeException

getSortOrders

public boolean[] getSortOrders()