org.apache.hadoop.hive.serde2.avro

Class AvroSerdeUtils

java.lang.Object

org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils

public class AvroSerdeUtils
extends Object

Utilities useful only to the AvroSerde itself. Not mean to be used by end-users but public for interop to the ql package.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroSerdeUtils.AvroTableProperties Enum container for all avro table properties.

Field Summary

Fields

Modifier and Type	Field and Description
static String	AVRO_SERDE_SCHEMA Deprecated.
static String	EXCEPTION_MESSAGE
static String	SCHEMA_DOC Deprecated.
static String	SCHEMA_LITERAL Deprecated.
static String	SCHEMA_NAME Deprecated.
static String	SCHEMA_NAMESPACE Deprecated.
static String	SCHEMA_NONE
static String	SCHEMA_RETRIEVER Deprecated.
static String	SCHEMA_URL Deprecated.

Constructor Summary

Constructors

Constructor and Description
AvroSerdeUtils()

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.avro.Schema	determineSchemaOrThrowException(org.apache.hadoop.conf.Configuration conf, Properties properties) Determine the schema to that's been provided for Avro serde work.
static Buffer	getBufferFromBytes(byte[] input)
static Buffer	getBufferFromDecimal(HiveDecimal dec, int scale)
static byte[]	getBytesFromByteBuffer(ByteBuffer byteBuffer)
static HiveDecimal	getHiveDecimalFromByteBuffer(ByteBuffer byteBuffer, int scale)
static org.apache.avro.Schema	getOtherTypeFromNullableType(org.apache.avro.Schema schema) In a nullable type, get the schema for the non-nullable type.
static org.apache.avro.Schema	getSchemaFor(File file)
static org.apache.avro.Schema	getSchemaFor(InputStream stream)
static org.apache.avro.Schema	getSchemaFor(String str)
protected static org.apache.avro.Schema	getSchemaFromFS(String schemaFSUrl, org.apache.hadoop.conf.Configuration conf)
static boolean	insideMRJob(org.apache.hadoop.mapred.JobConf job) Determine if we're being executed from within an MR job or as part of a select * statement.
static boolean	isNullableType(org.apache.avro.Schema schema) Determine if an Avro schema is of type Union[T, NULL].

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCHEMA_LITERAL

@Deprecated
public static final String SCHEMA_LITERAL

Deprecated.

See Also:

Constant Field Values

SCHEMA_URL

@Deprecated
public static final String SCHEMA_URL

Deprecated.

See Also:

Constant Field Values

SCHEMA_NAMESPACE

@Deprecated
public static final String SCHEMA_NAMESPACE

Deprecated.

See Also:

Constant Field Values

SCHEMA_NAME

@Deprecated
public static final String SCHEMA_NAME

Deprecated.

See Also:

Constant Field Values

SCHEMA_DOC

@Deprecated
public static final String SCHEMA_DOC

Deprecated.

See Also:

Constant Field Values

AVRO_SERDE_SCHEMA

@Deprecated
public static final String AVRO_SERDE_SCHEMA

Deprecated.

SCHEMA_RETRIEVER

@Deprecated
public static final String SCHEMA_RETRIEVER

Deprecated.

SCHEMA_NONE

public static final String SCHEMA_NONE

See Also:

Constant Field Values

EXCEPTION_MESSAGE

public static final String EXCEPTION_MESSAGE

Constructor Detail

AvroSerdeUtils

public AvroSerdeUtils()

Method Detail

determineSchemaOrThrowException

public static org.apache.avro.Schema determineSchemaOrThrowException(org.apache.hadoop.conf.Configuration conf,
                                                     Properties properties)
                                                              throws IOException,
                                                                     AvroSerdeException

Determine the schema to that's been provided for Avro serde work.

Parameters:

properties - containing a key pointing to the schema, one way or another

Returns:

schema to use while serdeing the avro file

Throws:

IOException - if error while trying to read the schema from another location

AvroSerdeException - if unable to find a schema or pointer to it in the properties

getSchemaFromFS

protected static org.apache.avro.Schema getSchemaFromFS(String schemaFSUrl,
                                     org.apache.hadoop.conf.Configuration conf)
                                                 throws IOException,
                                                        URISyntaxException

Throws:

IOException

URISyntaxException

isNullableType

public static boolean isNullableType(org.apache.avro.Schema schema)

Determine if an Avro schema is of type Union[T, NULL]. Avro supports nullable types via a union of type T and null. This is a very common use case. As such, we want to silently convert it to just T and allow the value to be null.

Returns:

true if type represents Union[T, Null], false otherwise

getOtherTypeFromNullableType

public static org.apache.avro.Schema getOtherTypeFromNullableType(org.apache.avro.Schema schema)

In a nullable type, get the schema for the non-nullable type. This method does no checking that the provides Schema is nullable.

insideMRJob

public static boolean insideMRJob(org.apache.hadoop.mapred.JobConf job)

Determine if we're being executed from within an MR job or as part of a select * statement. The signals for this varies between Hive versions.

Parameters:

job - that contains things that are or are not set in a job

Returns:

Are we in a job or not?

getBufferFromBytes

public static Buffer getBufferFromBytes(byte[] input)

getBufferFromDecimal

public static Buffer getBufferFromDecimal(HiveDecimal dec,
                          int scale)

getBytesFromByteBuffer

public static byte[] getBytesFromByteBuffer(ByteBuffer byteBuffer)

getHiveDecimalFromByteBuffer

public static HiveDecimal getHiveDecimalFromByteBuffer(ByteBuffer byteBuffer,
                                       int scale)

getSchemaFor

public static org.apache.avro.Schema getSchemaFor(String str)

getSchemaFor

public static org.apache.avro.Schema getSchemaFor(File file)

getSchemaFor

public static org.apache.avro.Schema getSchemaFor(InputStream stream)