org.apache.crunch.types.avro

Class AvroType<T>

java.lang.Object

org.apache.crunch.types.avro.AvroType<T>

All Implemented Interfaces:

Serializable, PType<T>

public class AvroType<T>
extends Object
implements PType<T>

The implementation of the PType interface for Avro-based serialization.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroType.AvroRecordType

Constructor Summary

Constructors

Constructor and Description
AvroType(Class<T> typeClass, org.apache.avro.Schema schema, DeepCopier<T> deepCopier, PType... ptypes)
AvroType(Class<T> typeClass, org.apache.avro.Schema schema, MapFn inputMapFn, MapFn outputMapFn, DeepCopier<T> deepCopier, AvroType.AvroRecordType recordType, PType... ptypes)

Method Summary

Methods

Modifier and Type	Method and Description
ReadableSource<T>	createSourceTarget(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path, Iterable<T> contents, int parallelism) Returns a ReadableSource that contains the data in the given Iterable.
boolean	equals(Object other)
Converter	getConverter()
ReadableSourceTarget<T>	getDefaultFileSource(org.apache.hadoop.fs.Path path) Returns a SourceTarget that is able to read/write data using the serialization format specified by this PType.
T	getDetachedValue(T value) Returns a copy of a value (or the value itself) that can safely be retained.
PTypeFamily	getFamily() Returns the PTypeFamily that this PType belongs to.
MapFn<Object,T>	getInputMapFn()
MapFn<T,Object>	getOutputMapFn()
AvroType.AvroRecordType	getRecordType()
org.apache.avro.Schema	getSchema()
List<PType>	getSubTypes() Returns the sub-types that make up this PType if it is a composite instance, such as a tuple.
Class<T>	getTypeClass() Returns the Java type represented by this PType.
int	hashCode()
boolean	hasReflect() Determine if the wrapped type is a reflection-based avro type or wraps one.
boolean	hasSpecific() Determine if the wrapped type is a specific data avro type or wraps one.
void	initialize(org.apache.hadoop.conf.Configuration conf) Initialize this PType for use within a DoFn.
boolean	isGeneric() Determine if the wrapped type is a generic data avro type.

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroType

public AvroType(Class<T> typeClass,
        org.apache.avro.Schema schema,
        DeepCopier<T> deepCopier,
        PType... ptypes)

AvroType

public AvroType(Class<T> typeClass,
        org.apache.avro.Schema schema,
        MapFn inputMapFn,
        MapFn outputMapFn,
        DeepCopier<T> deepCopier,
        AvroType.AvroRecordType recordType,
        PType... ptypes)

Method Detail

getRecordType

public AvroType.AvroRecordType getRecordType()

getTypeClass

public Class<T> getTypeClass()

Description copied from interface: PType

Returns the Java type represented by this PType.

Specified by:

getTypeClass in interface PType<T>

getFamily

public PTypeFamily getFamily()

Description copied from interface: PType

Returns the PTypeFamily that this PType belongs to.

Specified by:

getFamily in interface PType<T>

getSubTypes

public List<PType> getSubTypes()

Description copied from interface: PType

Returns the sub-types that make up this PType if it is a composite instance, such as a tuple.

Specified by:

getSubTypes in interface PType<T>

getSchema

public org.apache.avro.Schema getSchema()

hasSpecific

public boolean hasSpecific()

Determine if the wrapped type is a specific data avro type or wraps one.

Returns:

true if the wrapped type is a specific data type or wraps one

isGeneric

public boolean isGeneric()

Determine if the wrapped type is a generic data avro type.

Returns:

true if the wrapped type is a generic type

hasReflect

public boolean hasReflect()

Determine if the wrapped type is a reflection-based avro type or wraps one.

Returns:

true if the wrapped type is a reflection-based type or wraps one.

getInputMapFn

public MapFn<Object,T> getInputMapFn()

Specified by:

getInputMapFn in interface PType<T>

getOutputMapFn

public MapFn<T,Object> getOutputMapFn()

Specified by:

getOutputMapFn in interface PType<T>

getConverter

public Converter getConverter()

Specified by:

getConverter in interface PType<T>

getDefaultFileSource

public ReadableSourceTarget<T> getDefaultFileSource(org.apache.hadoop.fs.Path path)

Description copied from interface: PType

Returns a SourceTarget that is able to read/write data using the serialization format specified by this PType.

Specified by:

getDefaultFileSource in interface PType<T>

createSourceTarget

public ReadableSource<T> createSourceTarget(org.apache.hadoop.conf.Configuration conf,
                                   org.apache.hadoop.fs.Path path,
                                   Iterable<T> contents,
                                   int parallelism)
                                     throws IOException

Description copied from interface: PType

Returns a ReadableSource that contains the data in the given Iterable.

Specified by:

createSourceTarget in interface PType<T>

Parameters:

conf - The Configuration to use

path - The path to write the data to

contents - The contents to write

parallelism - The desired parallelism

Returns:

A new instance of ReadableSource

Throws:

IOException

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf)

Description copied from interface: PType

Initialize this PType for use within a DoFn. This generally only needs to be called when using a PType for PType.getDetachedValue(Object).

Specified by:

initialize in interface PType<T>

Parameters:

conf - Configuration object

See Also:

PType.getDetachedValue(Object)

getDetachedValue

public T getDetachedValue(T value)

Description copied from interface: PType

Returns a copy of a value (or the value itself) that can safely be retained.

This is useful when iterable values being processed in a DoFn (via a reducer) need to be held on to for more than the scope of a single iteration, as a reducer (and therefore also a DoFn that has an Iterable as input) re-use deserialized values. More information on object reuse is available in the DoFn class documentation.

Specified by:

getDetachedValue in interface PType<T>

Parameters:

value - The value to be deep-copied

Returns:

A deep copy of the input value

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object