avro Namespace Reference

A bunch of templates and specializations for encoding and decoding specific types. More...

Namespaces
	concepts
	The concept classes are used to simplify NodeImpl.

Classes
class	ArraySchema
class	BoolSchema
class	BytesSchema
struct	codec_traits
	Codec_traits tells avro how to encode and decode an object of given type. More...
struct	codec_traits< avro::null >
	codec_traits for Avro null. More...
struct	codec_traits< bool >
	codec_traits for Avro boolean. More...
struct	codec_traits< double >
	codec_traits for Avro double. More...
struct	codec_traits< float >
	codec_traits for Avro float. More...
struct	codec_traits< GenericDatum >
	Specialization of codec_traits for GenericDatum. More...
struct	codec_traits< int32_t >
	codec_traits for Avro int. More...
struct	codec_traits< int64_t >
	codec_traits for Avro long. More...
struct	codec_traits< std::array< uint8_t, N > >
	codec_traits for Avro fixed. More...
struct	codec_traits< std::conditional< avro::is_not_defined< bool_codec_traits >::value, std::vector< bool >::const_reference, void >::type >
struct	codec_traits< std::map< std::string, T > >
	codec_traits for Avro maps. More...
struct	codec_traits< std::pair< ValidSchema, GenericDatum > >
	Specialization of codec_traits for Generic datum along with its schema. More...
struct	codec_traits< std::string >
	codec_traits for Avro string. More...
struct	codec_traits< std::vector< T > >
	codec_traits for Avro arrays. More...
struct	codec_traits< std::vector< uint8_t > >
	codec_traits for Avro bytes. More...
class	CompoundLayout
class	DataFileReader
	Reads the contents of data file one after another. More...
class	DataFileReaderBase
	The type independent portion of reader. More...
class	DataFileWriter
	An Avro datafile that can store objects of type T. More...
class	DataFileWriterBase
	Type-independent portion of DataFileWriter. More...
class	Decoder
	Decoder is an interface implemented by every decoder capable of decoding Avro data. More...
class	DoubleSchema
class	Encoder
	The abstract base class for all Avro encoders. More...
class	EnumSchema
class	Exception
	Wrapper for std::runtime_error that provides convenience constructor for boost::format objects. More...
class	FixedSchema
class	FloatSchema
class	GenericArray
	The generic container for Avro arrays. More...
class	GenericContainer
	The base class for all generic type for containers. More...
class	GenericDatum
	Generic datum which can hold any Avro type. More...
class	GenericEnum
	Generic container for Avro enum. More...
class	GenericFixed
	Generic container for Avro fixed. More...
class	GenericMap
	The generic container for Avro maps. More...
class	GenericReader
	A utility class to read generic datum from decoders. More...
class	GenericRecord
	The generic container for Avro records. More...
class	GenericUnion
	Generic container for unions. More...
class	GenericWriter
	A utility class to write generic datum to encoders. More...
class	InputStream
	A no-copy input stream. More...
class	IntSchema
struct	is_defined
	Check if a T is a complete type i.e. More...
struct	is_not_defined
	Similar to is_defined, but used to check if T is not defined. More...
struct	is_promotable
struct	is_serializable
	Define an is_serializable trait for types we can serialize natively. More...
class	Layout
class	LogicalType
class	LongSchema
class	MapSchema
class	Name
class	Node
	Node is the building block for parse trees. More...
class	NodeArray
class	NodeEnum
class	NodeFixed
class	NodeImpl
	Implementation details for Node. More...
class	NodeMap
class	NodePrimitive
class	NodeRecord
class	NodeSymbolic
class	NodeUnion
struct	Null
	define a type to identify Null in template functions More...
class	NullSchema
class	NullValidator
class	OutputStream
	A no-copy output stream. More...
class	Parser
	Class that wraps a reader or ValidatingReade with an interface that uses explicit get* names instead of getValue. More...
class	PrimitiveLayout
class	ReaderImpl
	Parses from an avro encoding to the requested type. More...
class	RecordSchema
class	Resolver
class	ResolverSchema
class	ResolvingDecoder
	ResolvingDecoder is derived from Decoder, with an additional function to obtain the field ordering of fields within a record. More...
class	ResolvingReader
class	Schema
	The root Schema object is a base class. Nobody constructs this class directly. More...
class	SeekableInputStream
	An InputStream which also supports seeking to a specific offset. More...
class	Serializer
	Class that wraps a Writer or ValidatingWriter with an interface that uses explicit write* names instead of writeValue. More...
struct	StreamReader
	A convenience class for reading from an InputStream. More...
struct	StreamWriter
	A convinience class to write data into an OutputStream. More...
class	StringSchema
class	SymbolicSchema
struct	type_to_avro
class	UnionSchema
class	Validator
	This class is used by both the ValidatingSerializer and ValidationParser objects. More...
class	ValidSchema
	A ValidSchema is basically a non-mutable Schema that has passed some minimum of sanity checks. More...
class	WriterImpl
	Class for writing avro data to a stream. More...

Typedefs
typedef std::array< uint8_t, SyncSize >	DataFileSync
	The sync value.
typedef std::shared_ptr< Decoder >	DecoderPtr
	Shared pointer to Decoder.
typedef std::shared_ptr< ResolvingDecoder >	ResolvingDecoderPtr
	Shared pointer to ResolvingDecoder.
typedef std::shared_ptr< Encoder >	EncoderPtr
	Shared pointer to Encoder.
typedef std::shared_ptr< Node >	NodePtr
typedef concepts::NoAttribute< Name >	NoName
typedef concepts::SingleAttribute< Name >	HasName
typedef concepts::SingleAttribute< std::string >	HasDoc
typedef concepts::NoAttribute< NodePtr >	NoLeaves
typedef concepts::SingleAttribute< NodePtr >	SingleLeaf
typedef concepts::MultiAttribute< NodePtr >	MultiLeaves
typedef concepts::NoAttribute< std::string >	NoLeafNames
typedef concepts::MultiAttribute< std::string >	LeafNames
typedef concepts::NoAttribute< int >	NoSize
typedef concepts::SingleAttribute< int >	HasSize
typedef NodeImpl< NoName, NoLeaves, NoLeafNames, NoSize >	NodeImplPrimitive
typedef NodeImpl< HasName, NoLeaves, NoLeafNames, NoSize >	NodeImplSymbolic
typedef NodeImpl< HasName, MultiLeaves, LeafNames, NoSize >	NodeImplRecord
typedef NodeImpl< HasName, NoLeaves, LeafNames, NoSize >	NodeImplEnum
typedef NodeImpl< NoName, SingleLeaf, NoLeafNames, NoSize >	NodeImplArray
typedef NodeImpl< NoName, MultiLeaves, NoLeafNames, NoSize >	NodeImplMap
typedef NodeImpl< NoName, MultiLeaves, NoLeafNames, NoSize >	NodeImplUnion
typedef NodeImpl< HasName, NoLeaves, NoLeafNames, HasSize >	NodeImplFixed
typedef ReaderImpl< NullValidator >	Reader
typedef ReaderImpl< Validator >	ValidatingReader
typedef boost::blank	null
typedef codec_traits< std::vector< bool >::const_reference >	bool_codec_traits
typedef std::unique_ptr< InputStream >	InputStreamPtr
typedef std::unique_ptr< SeekableInputStream >	SeekableInputStreamPtr
typedef std::unique_ptr< OutputStream >	OutputStreamPtr
typedef WriterImpl< NullValidator >	Writer
typedef WriterImpl< Validator >	ValidatingWriter

Enumerations
enum	Codec { NULL_CODEC, DEFLATE_CODEC }
	Specify type of compression to use when writing data files. More...
enum	SchemaResolution {   RESOLVE_NO_MATCH, RESOLVE_MATCH, RESOLVE_PROMOTABLE_TO_LONG, RESOLVE_PROMOTABLE_TO_FLOAT,   RESOLVE_PROMOTABLE_TO_DOUBLE }
enum	Type {   AVRO_STRING, AVRO_BYTES, AVRO_INT, AVRO_LONG,   AVRO_FLOAT, AVRO_DOUBLE, AVRO_BOOL, AVRO_NULL,   AVRO_RECORD, AVRO_ENUM, AVRO_ARRAY, AVRO_MAP,   AVRO_UNION, AVRO_FIXED, AVRO_NUM_TYPES, AVRO_SYMBOLIC = AVRO_NUM_TYPES,   AVRO_UNKNOWN = -1 }
	The "type" for the schema. More...

Functions
template<typename Reader , typename T >
void	parse (Reader &p, T &val)
	The main parse entry point function. More...
template<typename T >
void	parse (ResolvingReader &p, T &val)
template<typename Reader , typename T >
void	parse (Reader &p, T &val, const std::false_type &)
	Type trait should be set to is_serializable in otherwise force the compiler to complain.
template<typename Reader , typename T >
void	translatingParse (Reader &p, T &val, const std::false_type &)
template<typename Writer , typename T >
void	serialize (Writer &s, const T &val)
	The main serializer entry point function. More...
template<typename Writer , typename T >
void	serialize (Writer &s, const T &val, const std::false_type &)
	Type trait should be set to is_serializable in otherwise force the compiler to complain.
AVRO_DECL void	compileJsonSchema (std::istream &is, ValidSchema &schema)
	Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object. More...
AVRO_DECL bool	compileJsonSchema (std::istream &is, ValidSchema &schema, std::string &error)
	Non-throwing version of compileJsonSchema. More...
AVRO_DECL ValidSchema	compileJsonSchemaFromStream (InputStream &is)
AVRO_DECL ValidSchema	compileJsonSchemaFromMemory (const uint8_t *input, size_t len)
AVRO_DECL ValidSchema	compileJsonSchemaFromString (const char *input)
AVRO_DECL ValidSchema	compileJsonSchemaFromString (const std::string &input)
AVRO_DECL ValidSchema	compileJsonSchemaFromFile (const char *filename)
AVRO_DECL DecoderPtr	binaryDecoder ()
	Returns an decoder that can decode binary Avro standard.
AVRO_DECL DecoderPtr	validatingDecoder (const ValidSchema &schema, const DecoderPtr &base)
	Returns an decoder that validates sequence of calls to an underlying Decoder against the given schema.
AVRO_DECL DecoderPtr	jsonDecoder (const ValidSchema &schema)
	Returns an decoder that can decode Avro standard for JSON.
AVRO_DECL ResolvingDecoderPtr	resolvingDecoder (const ValidSchema &writer, const ValidSchema &reader, const DecoderPtr &base)
	Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema. More...
AVRO_DECL EncoderPtr	binaryEncoder ()
	Returns an encoder that can encode binary Avro standard.
AVRO_DECL EncoderPtr	validatingEncoder (const ValidSchema &schema, const EncoderPtr &base)
	Returns an encoder that validates sequence of calls to an underlying Encoder against the given schema.
AVRO_DECL EncoderPtr	jsonEncoder (const ValidSchema &schema)
	Returns an encoder that encodes Avro standard for JSON.
AVRO_DECL EncoderPtr	jsonPrettyEncoder (const ValidSchema &schema)
	Returns an encoder that encodes Avro standard for pretty printed JSON.
std::ostream &	operator<< (std::ostream &os, const Name &n)
NodePtr	resolveSymbol (const NodePtr &node)
template<typename T >
std::string	intToHex (T i)
Type	nextType (Parser< ValidatingReader > &p)
bool	currentRecordName (Parser< ValidatingReader > &p, std::string &name)
bool	nextFieldName (Parser< ValidatingReader > &p, std::string &name)
std::unique_ptr< Resolver >	constructResolver (const ValidSchema &rwriterSchema, const ValidSchema &readerSchema, const Layout &readerLayout)
template<typename T >
void	encode (Encoder &e, const T &t)
	Generic encoder function that makes use of the codec_traits.
template<typename T >
void	decode (Decoder &d, T &t)
	Generic decoder function that makes use of the codec_traits.
AVRO_DECL OutputStreamPtr	memoryOutputStream (size_t chunkSize=4 *1024)
	Returns a new OutputStream, which grows in memory chunks of specified size.
AVRO_DECL InputStreamPtr	memoryInputStream (const uint8_t *data, size_t len)
	Returns a new InputStream, with the data from the given byte array. More...
AVRO_DECL InputStreamPtr	memoryInputStream (const OutputStream &source)
	Returns a new InputStream with the contents written into an outputstream. More...
AVRO_DECL std::shared_ptr< std::vector< uint8_t > >	snapshot (const OutputStream &source)
	Returns the contents written so far into the output stream, which should be a memory output stream. More...
AVRO_DECL OutputStreamPtr	fileOutputStream (const char *filename, size_t bufferSize=8 *1024)
	Returns a new OutputStream whose contents would be stored in a file. More...
AVRO_DECL InputStreamPtr	fileInputStream (const char *filename, size_t bufferSize=8 *1024)
	Returns a new InputStream whose contents come from the given file. More...
AVRO_DECL SeekableInputStreamPtr	fileSeekableInputStream (const char *filename, size_t bufferSize=8 *1024)
AVRO_DECL OutputStreamPtr	ostreamOutputStream (std::ostream &os, size_t bufferSize=8 *1024)
	Returns a new OutputStream whose contents will be sent to the given std::ostream. More...
AVRO_DECL InputStreamPtr	istreamInputStream (std::istream &in, size_t bufferSize=8 *1024)
	Returns a new InputStream whose contents come from the given std::istream. More...
AVRO_DECL InputStreamPtr	nonSeekableIstreamInputStream (std::istream &is, size_t bufferSize=8 *1024)
	Returns a new InputStream whose contents come from the given std::istream. More...
void	copy (InputStream &in, OutputStream &out)
	A convenience function to copy all the contents of an input stream into an output stream.
bool	isPrimitive (Type t)
	Returns true if and only if the given type is a primitive. More...
bool	isCompound (Type t)
	Returns true if and only if the given type is a non primitive valid type. More...
bool	isAvroType (Type t)
	Returns true if and only if the given type is a valid avro type.
bool	isAvroTypeOrPseudoType (Type t)
	Returns true if and only if the given type is within the valid range of enumeration.
AVRO_DECL const std::string &	toString (Type type)
	Converts the given type into a string. More...
AVRO_DECL std::ostream &	operator<< (std::ostream &os, avro::Type type)
	Writes a string form of the given type into the given ostream.
std::ostream &	operator<< (std::ostream &os, const Null &null)
	Writes schema for null null type to os. More...
AVRO_DECL uint64_t	encodeZigzag64 (int64_t input)
AVRO_DECL int64_t	decodeZigzag64 (uint64_t input)
AVRO_DECL uint32_t	encodeZigzag32 (int32_t input)
AVRO_DECL int32_t	decodeZigzag32 (uint32_t input)
AVRO_DECL size_t	encodeInt32 (int32_t input, std::array< uint8_t, 5 > &output)
AVRO_DECL size_t	encodeInt64 (int64_t input, std::array< uint8_t, 10 > &output)
template<typename Reader , typename T >
void	parse (Reader &p, T &val, const std::true_type &)
	The remainder of the file includes default implementations for serializable types.
template<typename Reader >
void	parse (Reader &p, std::vector< uint8_t > &val, const std::true_type &)
template<typename T >
void	translatingParse (ResolvingReader &p, T &val, const std::true_type &)
template<typename Writer , typename T >
void	serialize (Writer &s, T val, const std::true_type &)
	The remainder of the file includes default implementations for serializable types.
template<typename Writer >
void	serialize (Writer &s, const std::vector< uint8_t > &val, const std::true_type &)

Variables
class AVRO_DECL	InputStream
class AVRO_DECL	ValidSchema
	This class is used to implement an avro spec parser using a flex/bison compiler. More...
const int	SyncSize = 16
class AVRO_DECL	Schema

Detailed Description

A bunch of templates and specializations for encoding and decoding specific types.

Primitive AVRO types BOOLEAN, INT, LONG, FLOAT, DOUBLE, STRING and BYTES get decoded to and encoded from C++ types bool, int32_t, int64_t, float, double, std::string and std::vector<uint8_t> respectively. In addition, std::vector<T> for aribtrary type T gets encoded as an Avro array of T. Similarly, std::map<std::string, T> for arbitrary type T gets encoded as an Avro map with value type T.

Users can have their custom types encoded/decoded by specializing avro::codec_traits class for their types.

Enumeration Type Documentation

Codec

enum avro::Codec

Specify type of compression to use when writing data files.

SchemaResolution

enum avro::SchemaResolution

Enumerator
RESOLVE_NO_MATCH	The schemas definitely do not match.
RESOLVE_MATCH	The schemas match at a cursory level. For records and enums, this means the name is the same, but it does not necessarily mean that every symbol or field is an exact match.
RESOLVE_PROMOTABLE_TO_LONG	For primitives, the matching may occur if the type is promotable. This means that the writer matches reader if the writer's type is promoted the specified type.

Type

enum avro::Type

The "type" for the schema.

Enumerator
AVRO_STRING	String
AVRO_BYTES	Sequence of variable length bytes data
AVRO_INT	32-bit integer
AVRO_LONG	64-bit integer
AVRO_FLOAT	Floating point number
AVRO_DOUBLE	Double precision floating point number
AVRO_BOOL	Boolean value
AVRO_NULL	Null
AVRO_RECORD	Record, a sequence of fields
AVRO_ENUM	Enumeration
AVRO_ARRAY	Homogeneous array of some specific type
AVRO_MAP	Homogeneous map from string to some specific type
AVRO_UNION	Union of one or more types
AVRO_FIXED	Fixed number of bytes
AVRO_NUM_TYPES	Marker
AVRO_SYMBOLIC	User internally to avoid circular references.
AVRO_UNKNOWN	Used internally.

Function Documentation

compileJsonSchema() [1/2]

AVRO_DECL void avro::compileJsonSchema	(	std::istream &	is,
		ValidSchema &	schema
	)

Given a stream comtaining a JSON schema, compiles the schema to a ValidSchema object.

Throws if the schema cannot be compiled to a valid schema

compileJsonSchema() [2/2]

AVRO_DECL bool avro::compileJsonSchema	(	std::istream &	is,
		ValidSchema &	schema,
		std::string &	error
	)

Non-throwing version of compileJsonSchema.

Returns

True if no error, false if error (with the error string set)

fileInputStream()

AVRO_DECL InputStreamPtr avro::fileInputStream	(	const char *	filename,
		size_t	bufferSize = 8 *1024
	)

Returns a new InputStream whose contents come from the given file.

Data is read in chunks of given buffer size.

Referenced by avro::OutputStream::~OutputStream().

fileOutputStream()

AVRO_DECL OutputStreamPtr avro::fileOutputStream	(	const char *	filename,
		size_t	bufferSize = 8 *1024
	)

Returns a new OutputStream whose contents would be stored in a file.

Data is written in chunks of given buffer size.

If there is a file with the given name, it is truncated and overwritten. If there is no file with the given name, it is created.

Referenced by avro::OutputStream::~OutputStream().

isCompound()

bool avro::isCompound ( Type  t )

inline

Returns true if and only if the given type is a non primitive valid type.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_NUM_TYPES, and AVRO_RECORD.

isPrimitive()

bool avro::isPrimitive ( Type  t )

inline

Returns true if and only if the given type is a primitive.

Primitive types are: string, bytes, int, long, float, double, boolean and null

References AVRO_RECORD, and AVRO_STRING.

istreamInputStream()

AVRO_DECL InputStreamPtr avro::istreamInputStream	(	std::istream &	in,
		size_t	bufferSize = 8 *1024
	)

Returns a new InputStream whose contents come from the given std::istream.

The std::istream object should outlive the returned InputStream.

Referenced by avro::OutputStream::~OutputStream().

memoryInputStream() [1/2]

AVRO_DECL InputStreamPtr avro::memoryInputStream	(	const uint8_t *	data,
		size_t	len
	)

Returns a new InputStream, with the data from the given byte array.

It does not copy the data, the byte array should remain valid until the InputStream is used.

Referenced by avro::OutputStream::~OutputStream().

memoryInputStream() [2/2]

AVRO_DECL InputStreamPtr avro::memoryInputStream

(

const OutputStream &

source

)

Returns a new InputStream with the contents written into an outputstream.

The output stream must have been returned by an earlier call to memoryOutputStream(). The contents for the new input stream are the snapshot of the outputstream. One can construct any number of memory input stream from a single memory output stream.

nonSeekableIstreamInputStream()

AVRO_DECL InputStreamPtr avro::nonSeekableIstreamInputStream	(	std::istream &	is,
		size_t	bufferSize = 8 *1024
	)

Returns a new InputStream whose contents come from the given std::istream.

Use this instead of istreamInputStream if the istream does not support seekg (e.g. compressed streams). The returned InputStream would read off bytes instead of seeking. Of, course it has a performance penalty when reading instead of seeking; So, use this only when seekg does not work. The std::istream object should outlive the returned InputStream.

Referenced by avro::OutputStream::~OutputStream().

operator<<()

std::ostream& avro::operator<<	(	std::ostream &	os,
		const Null &	null
	)

Writes schema for null null type to os.

Parameters

os	The ostream to write to.
null	The value to be written.

ostreamOutputStream()

AVRO_DECL OutputStreamPtr avro::ostreamOutputStream	(	std::ostream &	os,
		size_t	bufferSize = 8 *1024
	)

Returns a new OutputStream whose contents will be sent to the given std::ostream.

The std::ostream object should outlive the returned OutputStream.

Referenced by avro::OutputStream::~OutputStream().

parse()

template<typename Reader , typename T >

void avro::parse	(	Reader &	p,
		T &	val
	)

The main parse entry point function.

Takes a parser (either validating or plain) and the object that should receive the parsed data.

Referenced by parse().

resolvingDecoder()

AVRO_DECL ResolvingDecoderPtr avro::resolvingDecoder	(	const ValidSchema &	writer,
		const ValidSchema &	reader,
		const DecoderPtr &	base
	)

Returns a decoder that decodes avro data from base written according to writerSchema and resolves against readerSchema.

The client uses the decoder as if the data were written using readerSchema. // FIXME: Handle out of order fields.

serialize()

template<typename Writer , typename T >

void avro::serialize	(	Writer &	s,
		const T &	val
	)

The main serializer entry point function.

Takes a serializer (either validating or plain) and the object that should be serialized.

Referenced by serialize().

snapshot()

AVRO_DECL std::shared_ptr<std::vector<uint8_t> > avro::snapshot

(

const OutputStream &

source

)

Returns the contents written so far into the output stream, which should be a memory output stream.

That is it must have been returned by a pervious call to memoryOutputStream().

Referenced by avro::OutputStream::~OutputStream().

toString()

AVRO_DECL const std::string& avro::toString

(

Type

type

)

Converts the given type into a string.

Useful for generating messages.

Referenced by isAvroTypeOrPseudoType().

Variable Documentation

ValidSchema

class AVRO_DECL avro::ValidSchema

This class is used to implement an avro spec parser using a flex/bison compiler.

In order for the lexer to be reentrant, this class provides a lexer object for each parse. The bison parser also uses this class to build up an avro parse tree as the avro spec is parsed.