All files in the filesystem are migrated by re-writing the block metadata - no datafiles are touched.

]]> update and remove().]]> hostname or hostname:port. If the specs string is null, defaults to localhost:defaultPort. @return a list of InetSocketAddress objects.]]> org.apache.hadoop.metrics.file and org.apache.hadoop.metrics.ganglia.

Plugging in an implementation involves writing a concrete subclass of AbstractMetricsContext. The subclass should get its configuration information using the getAttribute(attributeName) method.]]> Avro.]]> Avro.]]> = getCount(). @param newCapacity The new capacity in bytes.]]> Avro.]]> Avro.]]> Avro.]]> Index idx = startVector(...); while (!idx.done()) { .... // read element of a vector idx.incr(); } @deprecated Replaced by Avro.]]> Avro.]]> Avro.]]> Avro.]]> Avro.]]> Avro.]]> Avro.]]> Avro.]]> (DEPRECATED) Hadoop record I/O contains classes and a record description language translator for simplifying serialization and deserialization of records in a language-neutral manner.

DEPRECATED: Replaced by Avro.

Introduction

• enabling the specification of simple serializable data types (records)
• enabling the generation of code in multiple target languages for marshaling and unmarshaling such types
• providing target language specific support that will enable application programmers to incorporate generated code into their applications

Goals

• Support for commonly used primitive types. Hadoop should include as primitives commonly used builtin types from programming languages we intend to support.
• Support for common data compositions (including recursive compositions). Hadoop should support widely used composite types such as structs and vectors.
• Code generation in multiple target languages. Hadoop should be capable of generating serialization code in multiple target languages and should be easily extensible to new target languages. The initial target languages are C++ and Java.
• Support for generated target languages. Hadooop should include support in the form of headers, libraries, packages for supported target languages that enable easy inclusion and use of generated code in applications.
• Support for multiple output encodings. Candidates include packed binary, comma-separated text, XML etc.
• Support for specifying record types in a backwards/forwards compatible manner. This will probably be in the form of support for optional fields in records. This version of the document does not include a description of the planned mechanism, we intend to include it in the next iteration.

Non-Goals

• Serializing existing arbitrary C++ classes.
• Serializing complex data structures such as trees, linked lists etc.
• Built-in indexing schemes, compression, or check-sums.
• Dynamic construction of objects from an XML schema.

Data Types and Streams

Primitive Types

• byte: An 8-bit unsigned integer.
• boolean: A boolean value.
• int: A 32-bit signed integer.
• long: A 64-bit signed integer.
• float: A single precision floating point number as described by IEEE-754.
• double: A double precision floating point number as described by IEEE-754.
• ustring: A string consisting of Unicode characters.
• buffer: An arbitrary sequence of bytes.

Composite Types

• record: An aggregate type like a C-struct. This is a list of typed fields that are together considered a single unit of data. A record is serialized by sequentially serializing its constituent fields. In addition to serialization a record has comparison operations (equality and less-than) implemented for it, these are defined as memberwise comparisons.
• vector: A sequence of entries of the same data type, primitive or composite.
• map: An associative container mapping instances of a key type to instances of a value type. The key and value types may themselves be primitive or composite types.

Streams

DDL Syntax and Examples

Hadoop DDL Syntax

recfile = *include module *record
include = "include" path
path = (relative-path / absolute-path)
module = "module" module-name
module-name = name *("." name)
record := "class" name "{" 1*(field) "}"
field := type name ";"
name :=  ALPHA (ALPHA / DIGIT / "_" )*
type := (ptype / ctype)
ptype := ("byte" / "boolean" / "int" |
          "long" / "float" / "double"
          "ustring" / "buffer")
ctype := (("vector" "<" type ">") /
          ("map" "<" type "," type ">" ) ) / name)

• include: An include declaration specifies a DDL file to be referenced when generating code for types in the current DDL file. Record types in the current compilation unit may refer to types in all included files. File inclusion is recursive. An include does not trigger code generation for the referenced file.
• module: Every Hadoop DDL file must have a single module declaration that follows the list of includes and precedes all record declarations. A module declaration identifies a scope within which the names of all types in the current file are visible. Module names are mapped to C++ namespaces, Java packages etc. in generated code.
• class: Records types are specified through class declarations. A class declaration is like a Java class declaration. It specifies a named record type and a list of fields that constitute records of the type. Usage is illustrated in the following examples.

Examples

• A simple DDL file links.jr with just one record declaration.

  
  module links {
      class Link {
          ustring URL;
          boolean isRelative;
          ustring anchorText;
      };
  }
  

• A DDL file outlinks.jr which includes another

  
  include "links.jr"
  
  module outlinks {
      class OutLinks {
          ustring baseURL;
          vector outLinks;
      };
  }
  

Code Generation

$ rcc -l C++  ...

Target Language Mappings and Support

C++

namespace hadoop {

  enum RecFormat { kBinary, kXML, kCSV };

  class InStream {
  public:
    virtual ssize_t read(void *buf, size_t n) = 0;
  };

  class OutStream {
  public:
    virtual ssize_t write(const void *buf, size_t n) = 0;
  };

  class IOError : public runtime_error {
  public:
    explicit IOError(const std::string& msg);
  };

  class IArchive;
  class OArchive;

  class RecordReader {
  public:
    RecordReader(InStream& in, RecFormat fmt);
    virtual ~RecordReader(void);

    virtual void read(Record& rec);
  };

  class RecordWriter {
  public:
    RecordWriter(OutStream& out, RecFormat fmt);
    virtual ~RecordWriter(void);

    virtual void write(Record& rec);
  };


  class Record {
  public:
    virtual std::string type(void) const = 0;
    virtual std::string signature(void) const = 0;
  protected:
    virtual bool validate(void) const = 0;

    virtual void
    serialize(OArchive& oa, const std::string& tag) const = 0;

    virtual void
    deserialize(IArchive& ia, const std::string& tag) = 0;
  };
}

• RecFormat: An enumeration of the serialization encodings supported by this implementation of Hadoop.
• InStream: A simple abstraction for an input stream. This has a single public read method that reads n bytes from the stream into the buffer buf. Has the same semantics as a blocking read system call. Returns the number of bytes read or -1 if an error occurs.
• OutStream: A simple abstraction for an output stream. This has a single write method that writes n bytes to the stream from the buffer buf. Has the same semantics as a blocking write system call. Returns the number of bytes written or -1 if an error occurs.
• RecordReader: A RecordReader reads records one at a time from an underlying stream in a specified record format. The reader is instantiated with a stream and a serialization format. It has a read method that takes an instance of a record and deserializes the record from the stream.
• RecordWriter: A RecordWriter writes records one at a time to an underlying stream in a specified record format. The writer is instantiated with a stream and a serialization format. It has a write method that takes an instance of a record and serializes the record to the stream.
• Record: The base class for all generated record types. This has two public methods type and signature that return the typename and the type signature of the record.

namespace links {
  class Link : public hadoop::Record {
    // ....
  };
};

...
private:
  int32_t mMyField;
  ...
public:
  int32_t getMyField(void) const {
    return mMyField;
  };

  void setMyField(int32_t m) {
    mMyField = m;
  };
  ...

...
private:
  std::string mMyBuf;
  ...
public:

  std::string& getMyBuf() {
    return mMyBuf;
  };

  const std::string& getMyBuf() const {
    return mMyBuf;
  };
  ...

Examples

module inclrec {
    class RI {
        int      I32;
        double   D;
        ustring  S;
    };
}

include "inclrec.jr"
module testrec {
    class R {
        vector VF;
        RI            Rec;
        buffer        Buf;
    };
}

$ rcc -l c++ inclrec.jr testrec.jr

#ifndef _INCLREC_JR_HH_
#define _INCLREC_JR_HH_

#include "recordio.hh"

namespace inclrec {
  
  class RI : public hadoop::Record {

  private:

    int32_t      I32;
    double       D;
    std::string  S;

  public:

    RI(void);
    virtual ~RI(void);

    virtual bool operator==(const RI& peer) const;
    virtual bool operator<(const RI& peer) const;

    virtual int32_t getI32(void) const { return I32; }
    virtual void setI32(int32_t v) { I32 = v; }

    virtual double getD(void) const { return D; }
    virtual void setD(double v) { D = v; }

    virtual std::string& getS(void) const { return S; }
    virtual const std::string& getS(void) const { return S; }

    virtual std::string type(void) const;
    virtual std::string signature(void) const;

  protected:

    virtual void serialize(hadoop::OArchive& a) const;
    virtual void deserialize(hadoop::IArchive& a);
  };
} // end namespace inclrec

#endif /* _INCLREC_JR_HH_ */

#ifndef _TESTREC_JR_HH_
#define _TESTREC_JR_HH_

#include "inclrec.jr.hh"

namespace testrec {
  class R : public hadoop::Record {

  private:

    std::vector VF;
    inclrec::RI        Rec;
    std::string        Buf;

  public:

    R(void);
    virtual ~R(void);

    virtual bool operator==(const R& peer) const;
    virtual bool operator<(const R& peer) const;

    virtual std::vector& getVF(void) const;
    virtual const std::vector& getVF(void) const;

    virtual std::string& getBuf(void) const ;
    virtual const std::string& getBuf(void) const;

    virtual inclrec::RI& getRec(void) const;
    virtual const inclrec::RI& getRec(void) const;
    
    virtual bool serialize(hadoop::OutArchive& a) const;
    virtual bool deserialize(hadoop::InArchive& a);
    
    virtual std::string type(void) const;
    virtual std::string signature(void) const;
  };
}; // end namespace testrec
#endif /* _TESTREC_JR_HH_ */

Java

Mapping Summary

DDL Type        C++ Type            Java Type 

boolean         bool                boolean
byte            int8_t              byte
int             int32_t             int
long            int64_t             long
float           float               float
double          double              double
ustring         std::string         java.lang.String
buffer          std::string         org.apache.hadoop.record.Buffer
class type      class type          class type
vector    std::vector   java.util.ArrayList
map  std::map java.util.TreeMap

Data encodings

Binary Serialization Format

• class: Sequence of serialized members.
• vector: The number of elements serialized as an int. Followed by a sequence of serialized elements.
• map: The number of key value pairs serialized as an int. Followed by a sequence of serialized (key,value) pairs.

• byte: Represented by 1 byte, as is.
• boolean: Represented by 1-byte (0 or 1)
• int/long: Integers and longs are serialized zero compressed. Represented as 1-byte if -120 <= value < 128. Otherwise, serialized as a sequence of 2-5 bytes for ints, 2-9 bytes for longs. The first byte represents the number of trailing bytes, N, as the negative number (-120-N). For example, the number 1024 (0x400) is represented by the byte sequence 'x86 x04 x00'. This doesn't help much for 4-byte integers but does a reasonably good job with longs without bit twiddling.
• float/double: Serialized in IEEE 754 single and double precision format in network byte order. This is the format used by Java.
• ustring: Serialized as 4-byte zero compressed length followed by data encoded as UTF-8. Strings are normalized to UTF-8 regardless of native language representation.
• buffer: Serialized as a 4-byte zero compressed length followed by the raw bytes in the buffer.

CSV Serialization Format

• it makes parsing a lot easier without detracting too much from legibility
• the delimiters around composites make it obvious when one is reading a sequence of Hadoop records

• A string field begins with a single quote (').
• A buffer field begins with a sharp (#).
• A class, vector or map begins with 's{', 'v{' or 'm{' respectively and ends with '}'.

record = primitive / struct / vector / map
primitive = boolean / int / long / float / double / ustring / buffer

boolean = "T" / "F"
int = ["-"] 1*DIGIT
long = ";" ["-"] 1*DIGIT
float = ["-"] 1*DIGIT "." 1*DIGIT ["E" / "e" ["-"] 1*DIGIT]
double = ";" ["-"] 1*DIGIT "." 1*DIGIT ["E" / "e" ["-"] 1*DIGIT]

ustring = "'" *(UTF8 char except NULL, LF, % and , / "%00" / "%0a" / "%25" / "%2c" )

buffer = "#" *(BYTE except NULL, LF, % and , / "%00" / "%0a" / "%25" / "%2c" )

struct = "s{" record *("," record) "}"
vector = "v{" [record *("," record)] "}"
map = "m{" [*(record "," record)] "}"

XML Serialization Format

• byte: XML tag <ex:i1>. Values: 1-byte unsigned integers represented in US-ASCII
• boolean: XML tag <boolean>. Values: "0" or "1"
• int: XML tags <i4> or <int>. Values: 4-byte signed integers represented in US-ASCII.
• long: XML tag <ex:i8>. Values: 8-byte signed integers represented in US-ASCII.
• float: XML tag <ex:float>. Values: Single precision floating point numbers represented in US-ASCII.
• double: XML tag <double>. Values: Double precision floating point numbers represented in US-ASCII.
• ustring: XML tag <;string>. Values: String values represented as UTF-8. XML does not permit all Unicode characters in literal data. In particular, NULLs and control chars are not allowed. Additionally, XML processors are required to replace carriage returns with line feeds and to replace CRLF sequences with line feeds. Programming languages that we work with do not impose these restrictions on string types. To work around these restrictions, disallowed characters and CRs are percent escaped in strings. The '%' character is also percent escaped.
• buffer: XML tag <string&>. Values: Arbitrary binary data. Represented as hexBinary, each byte is replaced by its 2-byte hexadecimal representation.

• class: XML tag <struct>. A struct is a sequence of <member> elements. Each <member> element has a <name> element and a <value> element. The <name> is a string that must match /[a-zA-Z][a-zA-Z0-9_]*/. The value of the member is represented by a <value> element.
• vector: XML tag <array<. An <array> contains a single <data> element. The <data> element is a sequence of <value> elements each of which represents an element of the vector.
• map: XML tag <array>. Same as vector.

class {
  int           MY_INT;            // value 5
  vector MY_VEC;            // values 0.1, -0.89, 2.45e4
  buffer        MY_BUF;            // value '\00\n\tabc%'
}

<value>
  <struct>
    <member>
      <name>MY_INT</name>
      <value><i4>5</i4></value>
    </member>
    <member>
      <name>MY_VEC</name>
      <value>
        <array>
          <data>
            <value><ex:float>0.1</ex:float></value>
            <value><ex:float>-0.89</ex:float></value>
            <value><ex:float>2.45e4</ex:float></value>
          </data>
        </array>
      </value>
    </member>
    <member>
      <name>MY_BUF</name>
      <value><string>%00\n\tabc%25</string></value>
    </member>
  </struct>
</value>