Data Structure

Column {"name" => "value"}

Record {
  "recordId" => "1234",
  "family" => "family1",
  "columns" => [
    Column {"column1" => "value1"},
    Column {"column2" => "value2"},
    Column {"column2" => "value3"},
    Column {"column3" => "value4"}
  ]
}

Row {
  "id" => "r-5678",
  "records" => [
    Record {
      "recordId" => "1234",
      "family" => "family1",
      "columns" => [
        Column {"column1" => "value1"},
        Column {"column2" => "value2"},
        Column {"column2" => "value3"},
        Column {"column3" => "value4"}
      ]
    },
    Record {
      "recordId" => "9012",
      "family" => "family1",
      "columns" => [
        Column {"column1" => "value1"}
      ]
    },
    Record {
      "recordId" => "4321",
      "family" => "family2",
      "columns" => [
        Column {"column16" => "value1"}
      ]
    }
  ]
}

Querying

All queries follow the basic Lucene query syntax see (http://lucene.apache.org/core/4_4_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html) for an extensive explanation on the syntax.

All queries can have boolean logic like:

+docs.body:hadoop +docs.author:jon

Which is the same as:

docs.body:hadoop AND docs.author:jon

Row Queries

Row queries allow you to execute queries across Records within the same Row. Row queries are a similar idea to an inner join. Let's say you want to find all the Rows that contain a Record with the family "author" and has a "name" Column that has that contains a term "Jon" and another Record with the family "docs" and has a "body" Column with a term of "Hadoop".

+<author.name:Jon> +<docs.body:Hadoop>

Text

Text fields are analyzed with Lucene's standard analyzers, which mean that the string is broken down into terms and the terms capitalization is removed as well as any special punctuation. See Lucene's documentation for further explanation.

Examples:

To run a query to find all the rows that contain a column with a term of "hadoop" where the family is "docs" and the column is "body".

docs.body:hadoop

To run a query to find all the rows that contain a column with a term of "hadoop" and "awesome" where the family is "docs" and the column is "body".

docs.body:(+hadoop +awesome)

To run a query to find all the rows that contain a column with a phrase of "hadoop is awesome" where the family is "docs" and the column is "body".

docs.body:"hadoop is awesome"

To run a query to find all the rows that contain a column with a word of "hadoop" and we want to deal with misspellings (or a Fuzzy query) where the family is "docs" and the column is "body".

docs.body:hadoop~

To run a query to find all the rows that contain a column with a word that matches a wildcard pattern of "h*d?op" where the family is "docs" and the column is "body".

docs.body:h*d?op

String

String fields are indexed Columns that are not analyzed they are indexed as is. Do not use String fields for large amount of text, this will increase the size of your index and probably not give you the desired behavior. So given the string "Hadoop" and "hadoop", these will be indexed as two different term because the String field is case sensitive. Also if the string contains "The cow jumps over the moon." the single term that will be placed into the index is "The cow jumps over the moon." as a single string. This field type is normally used for id or type lookups.

Examples:

To run a query to find all the rows that contain a column with a term of "Hadoop" where the family is "docs" and the column is "type".

docs.type:Hadoop

Numeric

The numerics types are:

• int
• long
• float
• double

All numerics types can perform two types of queries:

• Exact Match
• Range

Examples:

To run a query to find all the rows that contain a column with a value of "12345" where the family is "docs" and the column is "id".

docs.id:12345

To run a query to find all the rows that contain a column with a starting value of "12345" and an ending values of "54321" where the family is "docs" and the column is "id".

docs.id:[12345 TO 54321]

To run a query to find all the rows that contain a column with a value less than "12345" where the family is "docs" and the column is "id".

docs.id:[MIN TO 12345}

To run a query to find all the rows that contain a column with a value less than or equal to "12345" where the family is "docs" and the column is "id".

docs.id:[MIN TO 12345]

To run a query to find all the rows that contain a column with a value great than to "12345" where the family is "docs" and the column is "id".

docs.id:{12345 TO MAX]

To run a query to find all the rows that contain a column with a value great than or equal to "12345" where the family is "docs" and the column is "id".

docs.id:[12345 TO MAX]

Date

Date types are basically a long field type with a date parser built-in. The date type can perform two types of queries:

• Exact Match
• Range

Examples:

To run a query to find all the rows that contain a column with a value of "2012-09-11" where the family is "docs" and the column is "published_date".

docs.published_date:2013-09-11

To run a query to find all the rows that contain a column with a starting value of "2012-09-11" and an ending values of "2013-09-11" where the family is "docs" and the column is "published_date".

docs.published_date:[2012-09-11 TO 2013-09-11]

Spatial

Spatial queries are supported through the Lucene spatial module. There are currently three different types of strategies:

• Point Vector
• Term Query Prefix Tree
• Recursive Prefix Tree

Currently all of the built in spatial types are using a GEO spatial context, meaning the assumed coordinates are on the planet Earth. Each of these strategies support different indexing features such as the shapes they can index, the shapes they can query with, and the operations they support during the query. See the type section below to get a list of supported operations and shapes per strategy.

Examples:

To run a query to find all the rows that contain a location within 10 km (0.089932 10 km in degrees) of gis coordinate "33.0, -88.0" where the family is "docs" and the column is "location".

docs.location:"Intersects(Circle(33.0, -88.0 d=0.089932))"

To run a query to find all the rows that contain a location within 10 km of gis coordinate "33.0, -88.0" where the family is "docs" and the column is "location".

docs.location:"Intersects(Circle(33.0, -88.0 d=10.0km))"

To run a query to find all the rows that contain a location within 10 miles of gis coordinate "33.0, -88.0" where the family is "docs" and the column is "location".

docs.location:"Intersects(Circle(33.0, -88.0 d=10.0m))"

Types

Text

The Text Type has the type name of:

text

Property Options:

• "stopWordPath" -Optional- default value is no stop words. This should be a HDFS path.
  This will load stop words into the StandardAnalyzer for this field, one term per line.
• "analyzerClass" -Optional- default value is a standard analyzer with no stop words.
  This could be any Analyzer class that has a default constructor or one that takes a Lucene Version enum.

String

The String Type has the type name of:

string

Property Options:

• None

Long

The Long Type has the type name of:

long

Property Options:

• "numericPrecisionStep" -Optional- default value is "4"

Integer

The Integer Type has the type name of:

int

Property Options:

• "numericPrecisionStep" -Optional- default value is "4"

Float

The Float Type has the type name of:

float

Property Options:

• "numericPrecisionStep" -Optional- default value is "4"

Double

The Double Type has the type name of:

double

Property Options:

• "numericPrecisionStep" -Optional- default value is "4"

Date

The Date Type has the type name of:

date

Property Options:

• "dateFormat" -Required- Examples: "yyyy-MM-dd", "MM/dd/yyyy" or anything that SimpleDateFormat can parse.
• "timeUnit" -Optional- Default is SECONDS. Other options (DAYS, HOURS, MINUTES, SECONDS, MILLISECONDS)
• "numericPrecisionStep" -Optional- default value is "4"

Stored

The Stored Type has the type name of:

stored

Property Options:

• None

Spatial

Point Vector

The Point Vector Spatial Type has the type name of:

geo-pointvector

Property Options:

• None

Supported Indexing Shapes:

• Point

Supported Querying Shapes:

• Circle
• Rectangle

Supported Querying Operations:

• Intersects

Term Prefix

The Term Prefix Spatial Type has the type name of:

geo-termprefix

Property Options:

• "spatialPrefixTree" can either equal to "GeohashPrefixTree" or "QuadPrefixTree"
• "maxLevels" -Optional- default value is "11"

Supported Indexing Shapes:

• Point

Supported Querying Shapes:

• Circle
• Rectangle

Supported Querying Operations:

• Intersects

Recursive Prefix

The Term Prefix Spatial Type has the type name of:

geo-recursiveprefix

Property Options:

• "spatialPrefixTree" can either equal to "GeohashPrefixTree" or "QuadPrefixTree"
• "maxLevels" -Optional- default value is "11"

Supported Indexing Shapes:

• Point
• Circle
• Rectangle

Supported Querying Shapes:

• Circle
• Rectangle

Supported Querying Operations:

• IsDisjointTo
• Intersects
• IsWithin
• Contains

Custom Types

public class ExampleType extends CustomFieldTypeDefinition {

  private String _fieldNameForThisInstance;

  /**
   * Get the name of the type.
   * 
   * @return the name.
   */
  @Override
  public String getName() {
    return "example";
  }

  /**
   * Configures this instance for the type.
   * 
   * @param fieldNameForThisInstance
   *          the field name for this instance.
   * @param properties
   *          the properties passed into this type definition from the
   *          {@link Blur.Iface#addColumnDefinition(String, ColumnDefinition)}
   *          method.
   */
  @Override
  public void configure(String fieldNameForThisInstance, Map properties, 
                        Configuration configuration) {
    _fieldNameForThisInstance = fieldNameForThisInstance;
  }

  /**
   * Create {@link Field}s for the index as well as for storing the original
   * data for retrieval.
   * 
   * @param family
   *          the family name.
   * @param column
   *          the column that holds the name and value.
   * 
   * @return the {@link Iterable} of {@link Field}s.
   */
  @Override
  public Iterable<? extends Field> getFieldsForColumn(String family, Column column) {
    String name = family + "." + column.getName();
    String value = column.getValue();
    return makeIterable(new StringField(name, value, Store.YES));
  }

  /**
   * Create {@link Field}s for the index do NOT store the data because the is a
   * sub column.
   * 
   * @param family
   *          the family name.
   * @param column
   *          the column that holds the name and value.
   * @param subName
   *          the sub column name.
   * 
   * @return the {@link Iterable} of {@link Field}s.
   */
  @Override
  public Iterable<? extends Field> getFieldsForSubColumn(String family, Column column, 
       String subName) {
    String name = family + "." + column.getName() + "." + subName;
    String value = column.getValue();
    return makeIterable(new StringField(name, value, Store.NO));
  }

  /**
   * Gets the query from the text provided by the query parser.
   * 
   * @param text
   *          the text provided by the query parser.
   * @return the {@link Query}.
   */
  @Override
  public Query getCustomQuery(String text) {
    return new TermQuery(new Term(_fieldNameForThisInstance, text));
  }

}

blur.fieldtype.customtype1=org.apache.blur.analysis.type.ExampleType1
blur.fieldtype.customtype2=org.apache.blur.analysis.type.ExampleType2
...

tableDescriptor.putToTableProperties("blur.fieldtype.customtype1", 
	"org.apache.blur.analysis.type.ExampleType1");
tableDescriptor.putToTableProperties("blur.fieldtype.customtype2", 
	"org.apache.blur.analysis.type.ExampleType2");
...