Oak Documentation

Indexing Rules

Indexing rules define which types of nodes and properties are indexed. An index configuration can define one or more indexingRules for different nodeTypes.

fulltextIndex
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + app:Page
      + properties
        - jcr:primaryType = "nt:unstructured"
        + publishedDate
          - propertyIndex = true
          - name = "jcr:content/publishedDate"
    + app:Asset
      + properties
        - jcr:primaryType = "nt:unstructured"
        + imageType
          - propertyIndex = true
          - name = "jcr:content/metadata/imageType"

Rules are defined per nodeType and each rule has one or more property definitions that determine which properties are indexed. Below is the canonical index definition structure

ruleName (nt:unstructured)
  - inherited (boolean) = true
  - indexNodeName (boolean) = false
  - includePropertyTypes (string) multiple
  + properties (nt:unstructured)

Following are the config options which can be defined at the index rule level

inherited

Optional boolean property defaults to true

Determines if the rule is applicable on exact match or can be applied if match is done on basis of nodeType inheritance

includePropertyTypes

Applicable when index is enabled for fulltext indexing

For full text index defaults to include all types

String array of property types which should be indexed. The values can be one specified in PropertyType Names

indexNodeName

@since Oak 1.0.20, 1.2.5

Default to false. If set to true then index would also be created for node name. This would enable faster evaluation of queries involving constraints on Node name. For example

• select [jcr:path] from [nt:base] where NAME() = 'kite'
• select [jcr:path] from [nt:base] where NAME() LIKE 'kite%'
• /jcr:root//kite
• /jcr:root//*[jcr:like(fn:name(), 'kite%')]
• /jcr:root//element(*, app:Asset)[fn:name() = 'kite']
• /jcr:root//element(kite, app:Asset)

- costPerExecution (Double) = 1.0
- costPerEntry (Double) = 1.0

propNode (nt:unstructured)
  - name (string)
  - boost (double) = '1.0'
  - index (boolean) = true
  - useInExcerpt (boolean) = false
  - analyzed (boolean) = false
  - nodeScopeIndex (boolean) = false
  - ordered (boolean) = false
  - isRegexp (boolean) = false
  - type (string) = 'undefined'
  - propertyIndex (boolean) = false
  - notNullCheckEnabled (boolean) = false
  - nullCheckEnabled (boolean) = false
  - excludeFromAggregation (boolean) = false
  - weight (long) = 5
  - function (string)

select * from [app:Asset] as a where isdescendantnode(a, [/content/app/old]) AND contains(*, 'white')

/oak:index/abc
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - includedPaths = ["/content", "/home"]
  - queryPaths = ["/content", "/home"]

Aggregation

Sometimes it is useful to include the contents of descendant nodes into a single node to easier search on content that is scattered across multiple nodes.

Oak allows you to define index aggregates based on relative path patterns and primary node types. Changes to aggregated items cause the main item to be reindexed, even if it was not modified.

Please note that aggregation does not support nodeType inheritance. To support aggregation on child nodeTypes, they need to be explicitly defined as a separate aggregation configuration in the index definition.

Aggregation configuration is defined under the aggregates node under index configuration. The following example creates an index aggregate on nt:file that includes the content of the jcr:content node:

fulltextIndex
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + aggregates
    + nt:file
      + include0
        - path = "jcr:content"

By default, all properties whose type matches includePropertyTypes and are part of child nodes as per the aggregation pattern are included for indexing. For excluding certain properties define a property definition with relative path and set excludeFromAggregation to true. Such properties would then be excluded from fulltext index

For a given nodeType multiple includes can be defined. Below is the aggregate definition structure for any specific include rule

aggregateNodeInclude (nt:unstructured)
  - path (string) mandatory
  - primaryType (string)
  - relativeNode (boolean) = false

Following are the details about the above mentioned config options which can be defined as part of aggregation include. (Refer to OAK-2268 for implementation details)

path

Path pattern to include. Example

• jcr:content - Name explicitly specified
• * - Any child node at depth 1
• */* - Any child node at depth 2

primaryType

Restrict the included nodes to a certain type. The restriction would be applied on the last node in given path

  + aggregates
    + nt:file
      + include0
        - path = "jcr:content"
        - primaryType = "nt:resource"

relativeNode

Boolean property indicates that query can be performed against specific node For example for following content

  + space.txt (app:Asset)
    + renditions (nt:folder)
      + original (nt:file)
        + jcr:content (nt:resource)
          - jcr:data

And a query like

  select * from [app:Asset] where contains([renditions/original/*], "pluto")

Following index configuration would be required

  fulltextIndex
    - jcr:primaryType = "oak:QueryIndexDefinition"
    - compatVersion = 2
    - type = "lucene"
    - async = "async"
    + aggregates
      + nt:file
        + include0
          - path = "jcr:content"
      + app:Asset
        + include0
          - path = "renditions/original"
          - relativeNode = true
    + indexRules
      - jcr:primaryType = "nt:unstructured"
      + app:Asset

Aggregation and Recursion

While performing aggregation the aggregation rules are again applied on node being aggregated. For example while aggregating for app:Asset above when renditions/original/* is being aggregated then aggregation rule would again be applied. In this case as renditions/original is nt:file then aggregation rule applicable for nt:file would be applied. Such a logic might result in recursion. (See JCR-2989 for details).

For such case reaggregateLimit is set on aggregate definition node and defaults to 5

  + aggregates
    + app:Asset
      - reaggregateLimit (long) = 5
      + include0
        - path = "renditions/original"
        - relativeNode = true

Analyzers

If no analyzer is specified, then OakAnalyzer is used, which uses the Apache Lucene StandardTokenizer, the LowerCaseFilter, and the WordDelimiterFilter with the following options: GENERATE_WORD_PARTS, STEM_ENGLISH_POSSESSIVE, and GENERATE_NUMBER_PARTS.

@since Oak 1.5.5, 1.4.7, 1.2.19 Unless custom analyzer is explicitly configured (as documented below), the built-in analyzer can be configured to include the original term as well (PRESERVE_ORIGINAL). This is controlled by setting boolean property indexOriginalTerm on the analyzers node:

/oak:index/assetType
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  + analyzers
    - indexOriginalTerm = true

(See OAK-4516 for details)

@since Oak 1.2.0

Analyzers can be configured as part of index definition via analyzers node. The default analyzer can be configured via analyzers/default node

    + sampleIndex
      - jcr:primaryType = "oak:QueryIndexDefinition"
      + analyzers
        + default
          ...

    + analyzers
      + default
        - class = "org.apache.lucene.analysis.standard.StandardAnalyzer"
        - luceneMatchVersion = "LUCENE_47" (optional)

    + analyzers
      + default
        - class = "org.apache.lucene.analysis.standard.StandardAnalyzer"
        - luceneMatchVersion = "LUCENE_47" (optional)
        + stopwords (nt:file)

    + analyzers
      + default
        + charFilters (nt:unstructured) // the filters needs to be ordered
          + HTMLStrip
          + Mapping
        + tokenizer
          - name = "Standard"
        + filters (nt:unstructured) // the filters needs to be ordered
          + LowerCase
          + Stop
            - words = "stop1.txt, stop2.txt"
            + stop1.txt (nt:file)
            + stop2.txt (nt:file)
          + PorterStem
          + Synonym
            - synonyms = "synonym.txt"
            + synonym.txt (nt:file)

Examples

To convert umlauts using ASCII folding, use the following. (ASCII folding converts characters to Basic Latin where possible. This includes umlauts, characters with accents, and so on.)

    + analyzers
      + default
        + tokenizer
          - name = "Standard"
        + filters (nt:unstructured) // the filters needs to be ordered
          + ASCIIFolding

For stemming support, use:

1. Use an analyzer which has stemming included by default e.g. EnglishAnalyzer which has PorterStemFilter.
    + analyzers
      + default
        - class = "org.apache.lucene.analysis.en.EnglishAnalyzer"

2. Use stemming as part of analyzer composition (using org.apache.lucene.analysis.hunspell.HunspellStemFilterFactory)
    + analyzers
      + default
        + tokenizer
          - name = "Standard"
        + filters (nt:unstructured) // the filters needs to be ordered
          + LowerCase
          + HunspellStem
            - dictionary = "en_gb.dic"
            - affix = "en_gb.aff"
            + en_gb.aff (nt:file)
            + en_gb.dic (nt:file)

Points to note

1. Name of filters, charFilters and tokenizer are formed by removing the factory suffixes. So
   • org.apache.lucene.analysis.standard.StandardTokenizerFactory -> Standard
   • org.apache.lucene.analysis.charfilter.MappingCharFilterFactory -> Mapping
   • org.apache.lucene.analysis.core.StopFilterFactory -> Stop
2. Any config parameter required for the factory is specified as property of that node
   • If the factory requires to load a file e.g. stop words from some file then file content can be provided via creating child nt:file node of the filename
   • The property value MUST be of type String. No other JCR type should be used for them like array or integer etc
3. The analyzer-chain processes text from nodes as well text passed in query. So, do take care that any mapping configuration (e.g. synonym mappings) factor in the chain of analyzers. E.g a common mistake for synonym mapping would be to have domain => Range while there's a lower case filter configured as well (see the example above). For such a setup an indexed value domain would actually get indexed as Range (mapped value doesn't have lower case filter below it) but a query for Range would actually query for range (due to lower case filter) and won't give the result (as might be expected). An easy work-around for this example could be to have lower case mappings i.e. just use domain => range.
4. Precedence: Specifying analyzer class directly has precedence over analyzer configuration by composition. If you want to configure analyzers by composition then analyzer class MUST NOT be specified. In-build analyzer has least precedence and comes into play only if no custom analyzer has been configured. Similarly, setting indexOriginalTerm on analyzers node to modify behavior of in-built analyzer also works only when no custom analyzer has been configured.
5. To determine list of supported factories have a look at Lucene javadocs for
   • TokenizerFactory
   • CharFilterFactory
   • FilterFactory
6. Oak support for composing analyzer is based on Lucene. So some helpful docs around this
   • https://cwiki.apache.org/confluence/display/solr/Understanding+Analyzers%2C+Tokenizers%2C+and+Filters
   • https://cwiki.apache.org/confluence/display/solr/CharFilterFactories
   • https://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#Specifying_an_Analyzer_in_the_schema
7. When defining synonyms:
   • in the synonym file, lines like plane, airplane, aircraft refer to tokens that are mutual synoyms whereas lines like plane => airplane refer to one way synonyms, so that plane will be expanded to airplane but not vice versa
   • continuing with the point above, since oak would use the same analyzer for indexing as well as querying, using one-way synonyms in any practical way is not supported at the moment.
   • special characters have to be escaped
   • multi word synonyms need particular attention (see https://lucidworks.com/post/solution-for-multi-term-synonyms-in-lucenesolr-using-the-auto-phrasing-tokenfilter/)

Note that currently only one analyzer can be configured per index. Its not possible to specify separate analyzer for query and index time currently.

Codec

Name of Lucene Codec to use. By default, if the index involves fulltext indexing then Oak Lucene uses OakCodec which disables compression. Due to this the index size may grow large. To enable compression you can set the codec to Lucene46

/oak:index/assetType
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - codec = "Lucene46"

Refer to OAK-2853 for details. Enabling the Lucene46 codec would lead to smaller and compact indexes.

Boost and Search Relevancy

@since Oak 1.2.5

When fulltext indexing is enabled then internally Oak would create a fulltext field which consists of text extracted from various other fields i.e. fields for which nodeScopeIndex is true. This allows search like /jcr:root/content//*[jcr:contains(., 'foo')] to perform search across any indexable field containing foo (See contains function for details)

In certain cases its desirable that those nodes where the searched term is present in a specific property are ranked higher (come earlier in search result) compared to those node where the searched term is found in some other property.

In such cases it should be possible to boost specific text contributed by individual property. Meaning that if a title field is boosted more than description, then search result would those node coming earlier where searched term is found in title field

For that to work ensure that for each such property (which need to be preferred) both nodeScopeIndex and analyzed are set to true. In addition, you can specify boost property so give higher weightage to values found in specific property

Note that even without setting explicit boost and just setting nodeScopeIndex and analyzed to true would improve the search result due to the way Lucene does scoring. Internally Oak would create separate Lucene fields for those jcr properties and would perform a search across all such fields. For more details refer to OAK-3367

  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + app:Asset
      + properties
        - jcr:primaryType = "nt:unstructured"
        + description
          - nodeScopeIndex = true
          - analyzed = true
          - name = "jcr:content/metadata/jcr:description"
        + title
          - analyzed = true
          - nodeScopeIndex = true
          - name = "jcr:content/metadata/jcr:title"
          - boost = 2.0

With above index config a search like

SELECT * FROM [app:Asset]
WHERE CONTAINS(., 'Batman')

Would have those node (of type app:Asset) come first where Batman is found in jcr:title. While those nodes where search text is found in other field like aggregated content would come later

Effective Index Definition

@since Oak 1.6

Prior to Oak 1.6 index definition as defined in content was directly used for query execution and indexing. It was possible that index definition is modified in incompatible way and that would start affecting the query execution leading to inconsistent result.

Since Oak 1.6 the index definitions are cloned upon reindexing and stored in a hidden structure. For further incremental indexing and for query plan calculation the stored index definition is used. So any changes done post reindex to index definition would not be applicable until a reindex is done.

There would be some cases where changes in index definition does not require a reindex. For e.g. if a new property is being introduced in content model and no prior content exist with such a property then it's safe to index such a property without doing a reindex. For such cases user must follow below steps

1. Make the required changes
2. Set refresh property to true in index definition node
3. Save the changes

On next async indexing cycle this flag would be picked up and stored index definition would be refreshed. Post this the flag would be automatically removed and a log message would be logged. You would also see a log message like below

LuceneIndexEditorContext - Refreshed the index definition for [/oak:index/fooLuceneIndex]

To simplify troubleshooting the stored index definition can be accessed from LuceneIndexMBean via getStoredIndexDefinition operation. It would dump the string representation of stored NodeState

This feature can be disabled by setting OSGi property disableStoredIndexDefinition for LuceneIndexProviderService to true. Once disable any change in index definition would start effecting the query plans

Refer to OAK-4400 for more details.

Generating Index Definition

To simplify generating index definition suitable for evaluating certain set of queries you can make use of http://oakutils.appspot.com/generate/index. Here you can provide a set of queries and then it would generate the suitable index definitions for those queries.

Note that you would still need to tweak the definition for aggregation, path include exclude etc as that data cannot be inferred from the query

Mime type usage

A binary is only indexed if there is an associated property jcr:mimeType defined and that is supported by Tika. By default, indexer uses TypeDetector instead of default DefaultDetector which relies on the jcr:mimeType to pick up the right parser.

Mime type mapping

@since Oak 1.7.7

In certain circumstances, it may be desired to pass a value other than the jcr:mimeType property into the Tika parser. For example, this would be necessary if a binary has an application-specific mime type, but is parsable by the standard Tika parser for some generic type. To support these cases, create a node structure under the tika/mimeTypes node following the mime type structure, e.g.

+ tika
    + mimeTypes (nt:unstructured)
      + application (nt:unstructured)
        + vnd.mycompany-document (nt:unstructured)
          - mappedType = application/pdf

When this index is indexing a binary of type application/vnd.mycompany-document it will force Tika to treat it as a binary of type application/pdf.

Suggestions

@since Oak 1.1.17, 1.0.15

In order to use Lucene index to perform search suggestions, the index definition node (the one of type oak:QueryIndexDefinition) needs to have the compatVersion set to 2, then one or more property nodes, depending on use case, need to have the property useInSuggest set to true, such setting controls from which properties terms to be used for suggestions will be taken.

Once the above configuration has been done, by default, the Lucene suggester is updated every 10 minutes but that can be changed by setting the property suggestUpdateFrequencyMinutes in suggestion node under the index definition node to a different value. Note that up till Oak 1.3.14/1.2.14, suggestUpdateFrequencyMinutes was to be setup at index definition node itself. That is is still supported for backward compatibility, but having a separate suggestion node is preferred.

Sample configuration for suggestions based on terms contained in jcr:description property.

/oak:index/lucene-suggest
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + suggestion
    - suggestUpdateFrequencyMinutes = 20
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + jcr:description
          - propertyIndex = true
          - analyzed = true
          - useInSuggest = true

@since Oak 1.3.12, 1.2.14 the index Analyzer can be used to perform a have more fine grained suggestions, e.g. single words (whereas default suggest configuration returns entire property values, see [OAK-3407]: https://issues.apache.org/jira/browse/OAK-3407). Analyzed suggestions can be enabled by setting “suggestAnalyzed” property to true, e.g.:

/oak:index/lucene-suggest
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + suggestion
    - suggestUpdateFrequencyMinutes = 20
    - suggestAnalyzed = true

Note that up till Oak 1.3.14/1.2.14, suggestAnalyzed was to be setup at index definition node itself. That is still supported for backward compatibility, but having a separate suggestion node is preferred.

Setting up useInSuggest=true for a property definition having name=:nodeName would add node names to suggestion dictionary (See property name for node name indexing)

Since, Oak 1.3.16/1.2.14, very little support exists for queries with ISDESCENDANTNODE constraint to subset suggestions on a sub-tree. It requires evaluatePathRestrictions=true on index definition. e.g.

SELECT rep:suggest() FROM [nt:base] WHERE SUGGEST('test') AND ISDESCENDANTNODE('/a/b')

or

/jcr:root/a/b//[rep:suggest('in 201')]/(rep:suggest())

Note, the subset is done by filtering top 10 suggestions. So, it's possible to get no suggestions for a subtree query, if top 10 suggestions are not part of that subtree. For details look at OAK-3994 and related issues.

Spellchecking

@since Oak 1.1.17, 1.0.13

In order to use Lucene index to perform spellchecking, the index definition node (the one of type oak:QueryIndexDefinition) needs to have the compatVersion set to 2, then one or more property nodes, depending on use case, need to have the property useInSpellcheck set to true, such setting controls from which properties terms to be used for spellcheck corrections will be taken.

Sample configuration for spellchecking based on terms contained in jcr:title property.

Since Oak 1.3.11/1.2.14, each suggestion would be returned per row.

/oak:index/lucene-spellcheck
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + jcr:title
          - propertyIndex = true
          - analyzed = true
          - useInSpellcheck = true

Since, Oak 1.3.16/1.2.14, very little support exists for queries with ISDESCENDANTNODE constraint to subset suggestions on a sub-tree. It requires evaluatePathRestrictions=true on index definition. e.g.

SELECT rep:suggest() FROM [nt:base] WHERE SUGGEST('test') AND ISDESCENDANTNODE('/a/b')

or

/jcr:root/a/b//[rep:suggest('in 201')]/(rep:suggest())

Note, the subset is done by filtering top 10 spellchecks. So, it's possible to get no results for a subtree query, if top 10 spellchecks are not part of that subtree. For details look at OAK-3994 and related issues.

Facets

@since Oak 1.3.14

Lucene property indexes can also be used for retrieving facets, in order to do so the property facets must be set to true on the property definition.

/oak:index/lucene-with-facets
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + tags
          - facets = true
          - propertyIndex = true

Specific facet related features for Lucene property index can be configured in a separate facets node below the index definition. @since Oak 1.5.15 The no. of facets to be retrieved is configurable via the topChildren property, which defaults to 10.

/oak:index/lucene-with-more-facets
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + facets
    - topChildren = 100
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + tags
          - facets = true
          - propertyIndex = true

By default, ACL checks are always performed on facets by the Lucene property index. This is secure (no information leakage is possible), but can be slow. The secure configuration property allows to configure how facet counts are performed. @since Oak 1.6.16, 1.8.10, 1.9.13 secure property is a string with allowed values of secure, statistical and insecure - secure being the default value. Before that secure was a boolean property and to maintain compatibility false maps to insecure while true (default at the time) maps to secure. The following configuration options are supported:

• secure (the default) means all results of a query are checked for access permissions. Facets and counts returned by index reflect what is accessible to the given user. The query result therefore only reflects information the user has access rights for. This can be slow, specially for large result set.

• insecure means the facet counts are reported as stored in the index, without performing access rights checks. This setting may allow users to infer the existence of content to which they do not have access. It is recommended to use this setting only if either the index is guaranteed to only contain data that is public (e.g. a public subtree of the repository), or if the information is not sensitive.

• statistical means the data is sampled randomly (default 1000 configurable via sampleSize), and ACL checks are performed on this sample. Facet counts returned are proportional to the percentage of accessible samples that were checked for ACL. Warning: this setting potentially leaks repository information the user that runs the query may not see. It must only be used if either the index is guaranteed to only contain data that is public (e.g. a public subtree of the repository), or if the leaked information is not sensitive. Do note that the beauty of sampling is that a sample size of 1000 has an error rate of 3% with 95% confidence, if ACLs are evenly distributed over the sampled data. However, often ACLs are not evenly distributed. Also, a low rate of accessible documents decreases chances to reach that average rate. To have a sense of expectation of error rate, here's how errors looked like in different scenarios of test runs with sample size of 1000 with error averaged over 1000 random runs for each scenario.

|-----------------|-----------------------|------------------------|
| Result set size | %age accessible nodes | Avg error in 1000 runs |
|-----------------|-----------------------|------------------------|
| 2000            |  5                    |  5.79                  |
| 5000            |  5                    |  9.99                  |
| 10000           |  5                    |  10.938                |
| 100000          |  5                    |  11.13                 |
|                 |                       |                        |
| 2000            | 25                    | 2.4192004              |
| 5000            | 25                    | 3.8087976              |
| 10000           | 25                    | 4.096                  |
| 100000          | 25                    | 4.3699985              |
|                 |                       |                        |
| 2000            | 50                    | 1.3990011              |
| 5000            | 50                    | 2.2695997              |
| 10000           | 50                    | 2.5303981              |
| 100000          | 50                    | 2.594599               |
|                 |                       |                        |
| 2000            | 75                    | 0.80360085             |
| 5000            | 75                    | 1.1929348              |
| 10000           | 75                    | 1.4357346              |
| 100000          | 75                    | 1.4272015              |
|                 |                       |                        |
| 2000            | 95                    | 0.30958                |
| 5000            | 95                    | 0.52715933             |
| 10000           | 95                    | 0.5109484              |
| 100000          | 95                    | 0.5481065              |
|-----------------|-----------------------|------------------------|

Notice that error rate does increase with large result set sizes but it flattens after around 10000 results. Also, note that even with 50% results being accessible, error rate averages at less that 3%.

So, in most cases, sampling size of 1000 should give fairly decent estimation of facet counts. On the off chance that the setup is such that error rates are intolerable, sample size can be configured with sampleSize property under facets configuration node. Error rates are generally inversely proportional to √sample-size. So, to reduce error rate by 1/2 sample size needs to increased 4 times.

Canonical example of statistical configuration would look like:

/oak:index/lucene-with-statistical-facets
  + facets
    - secure = "statistical"
    - sampleSize = 1500
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + tags
          - facets = true
          - propertyIndex = true

See query-engine regarding how to query to evaluate facets alongwith. Also check out some examples of queries and required index definitions for faceting in use case 5.

Score Explanation

@since Oak 1.3.12

Lucene supports explanation of scores which can be selected in a query using a virtual column oak:scoreExplanation. e.g. select [oak:scoreExplanation], * from [nt:base] where foo='bar'

Note that showing explanation score is expensive. So, this feature should be used for debug purposes only.

Custom hooks

@since Oak 1.3.14

The following features is now deprecated: In OSGi environment, implementations of IndexFieldProvider and FulltextQueryTermsProvider under org.apache.jackrabbit.oak.plugins.index.lucene.spi (see javadoc here) are called during indexing and querying as documented in javadocs.

A - Simple queries

In many cases the query is purely based on some specific property and is not restricted to any specific nodeType

SELECT
  *
FROM [nt:base] AS s
WHERE ISDESCENDANTNODE([/content/public/platform])
AND s.code = 'DRAFT'

Following index definition would allow using Lucene index for above query

/oak:index/assetType
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  - evaluatePathRestrictions = true
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + code
          - propertyIndex = true
          - name = "code"

Above definition

• Indexes code property present on any node
• Supports evaluation of path restriction i.e. ISDESCENDANTNODE([/content/public/platform]) via evaluatePathRestrictions
• Has a single indexRule for nt:base as queries do not specify any explicit nodeType restriction

Now you have another query like

SELECT
  *
FROM [nt:base] AS s
WHERE
  s.status = 'DONE'

Here we can either add another property to the above definition or create a new index definition altogether. By default, prefer to club such indexes together

/oak:index/assetType
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  - evaluatePathRestrictions = true
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + code
          - propertyIndex = true
          - name = "code"
        + status
          - propertyIndex = true
          - name = "status"

Taking another example. Lets say you perform a range query like

SELECT
  [jcr:path],
  [jcr:score],
  *
FROM [nt:base] AS a
WHERE isdescendantnode(a, '/content')
AND [offTime] > CAST('2015-04-06T02:28:33.032-05:00' AS date)

This can also be clubbed in same index definition above

/oak:index/assetType
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  - evaluatePathRestrictions = true
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + nt:base
      + properties
        - jcr:primaryType = "nt:unstructured"
        + code
          - propertyIndex = true
          - name = "code"
        + status
          - propertyIndex = true
          - name = "status"
        + offTime
          - propertyIndex = true
          - name = "offTime"

B - Queries for structured content

Queries in previous examples were based on mostly unstructured content where no nodeType restrictions were applied. However, in many cases the nodes being queried confirm to certain structure. For example, you have the following content

/content/dam/assets/december/banner.png
  - jcr:primaryType = "app:Asset"
  + jcr:content
    - jcr:primaryType = "app:AssetContent"
    + metadata
      - dc:format = "image/png"
      - status = "published"
      - jcr:lastModified = "2009-10-9T21:52:31"
      - app:tags = ["properties:orientation/landscape", "marketing:interest/product"]
      - size = 450
      - comment = "Image for december launch"
      - jcr:title = "December Banner"
      + xmpMM:History
        + 1
          - softwareAgent = "Adobe Photoshop"
          - author = "David"
    + renditions (nt:folder)
      + original (nt:file)
        + jcr:content
          - jcr:data = ...

Content like above is then queried in multiple ways. So lets take first query

UC1 - Find all assets which are having status as published

SELECT * FROM [app:Asset] AS a
WHERE a.[jcr:content/metadata/status] = 'published'

For this following index definition would have to be created

/oak:index/assetType
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + app:Asset
      + properties
        - jcr:primaryType = "nt:unstructured"
        + status
          - propertyIndex = true
          - name = "jcr:content/metadata/status"

Above index definition

• Indexes all nodes of type app:Asset only
• Indexes relative property jcr:content/metadata/status for all such nodes

UC2 - Find all assets which are having status as published sorted by last modified date

SELECT * FROM [app:Asset] AS a
WHERE a.[jcr:content/metadata/status] = 'published'
ORDER BY a.[jcr:content/metadata/jcr:lastModified] DESC

To enable above query the index definition needs to be updated to following

    + app:Asset
      + properties
        - jcr:primaryType = "nt:unstructured"
        + status
          - propertyIndex = true
          - name = "jcr:content/metadata/status"
        + lastModified
          - propertyIndex = true
          - name = "jcr:content/metadata/jcr:lastModified"
          - ordered = true
          - type = Date

Above index definition

• jcr:content/metadata/jcr:lastModified is marked as ordered enabling support order by evaluation i.e. sorting for such properties
• Property type is set to Date
• Indexes both status and jcr:lastModified

UC3 - Find all assets where comment contains december

SELECT * FROM [app:Asset]
WHERE CONTAINS([jcr:content/metadata/comment], 'december')

To enable above query the index definition needs to be updated to following

    + app:Asset
      + properties
        - jcr:primaryType = "nt:unstructured"
        + comment
          - name = "jcr:content/metadata/comment"
          - analyzed = true

Above index definition

• jcr:content/metadata/comment is marked as analyzed enabling evaluation of contains i.e. fulltext search
• propertyIndex is not enabled as this property is not going to be used to perform equality check

**UC4 - Find all assets which are created by David and refer to december **

SELECT * FROM [app:Asset]
WHERE CONTAINS(., 'december david')

Here we want to create a fulltext index for all assets. It would index all the properties in app:Asset including all relative nodes. To enable that we need to make use of aggregation

/oak:index/assetType
  - jcr:primaryType = "oak:QueryIndexDefinition"
  - compatVersion = 2
  - type = "lucene"
  - async = "async"
  + aggregates
    + app:Asset
      + include0
        - path = "jcr:content"
      + include1
        - path = "jcr:content/metadata"
      + include2
        - path = "jcr:content/metadata/*"
      + include3
        - path = "jcr:content/metadata/*/*"
      + include4
        - path = "jcr:content/renditions"
      + include5
        - path = "jcr:content/renditions/original"
    + nt:file
      + include0
        - path = "jcr:content"
  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + app:Asset
      - includePropertyTypes = ["String", "Binary"]
      + properties
        - jcr:primaryType = "nt:unstructured"
        + comment
          - propertyIndex = true
          - nodeScopeIndex = true
          - name = "jcr:content/metadata/comment"

Above index definition

• Only indexes String and Binary properties as part of fulltext index via includePropertyTypes

• Has aggregates defined for various relative paths like jcr:content, jcr:content/metadata, jcr:content/renditions/original etc.

  With these rules properties like banner.png/metadata/comment, banner.png/metadata/xmpMM:History/1/author get indexed as part for fulltext index for banner.png node.

• Inclusion of jcr:content/renditions/original would lead to aggregation of jcr:content/renditions/original/jcr:content/jcr:data property also as aggregation logic would apply rules for nt:file while aggregating the original node

• Aggregation would include by default all properties which are part of includePropertyTypes. However, if any property has an explicit property definition provided like comment then nodeScopeIndex would need to be set to true

Above definition would allow fulltext query to be performed. But we can do more. Suppose you want to give more preference to those nodes where the fulltext term is found in jcr:title compared to any other field. In such cases we can boost such fields

  + indexRules
    - jcr:primaryType = "nt:unstructured"
    + app:Asset
      + properties
        - jcr:primaryType = "nt:unstructured"
        + comment
          - propertyIndex = true
          - nodeScopeIndex = true
          - name = "jcr:content/metadata/comment"
        + title
          - propertyIndex = true
          - nodeScopeIndex = true
          - name = "jcr:content/metadata/jcr:title"
          - boost = 2.0

    SELECT [rep:facet(title)] FROM [app:Asset]
or
    /jcr:root//element(*, app:Asset)/(rep:facet(title))
or
    SELECT [rep:facet(title)], [rep:facet(tags)] FROM [app:Asset]
or
    /jcr:root//element(*, app:Asset)/(rep:facet(title) | rep:facet(tags))

   + /oak:index/index1
      - ...
      + aggregates
        + app:Asset
          + include0
            - path = "jcr:content"
      + indexRules
        + app:Asset
          + properties
            + title
              - facets = true
            + tags
              - facets = true
              - propertyIndex = true
or
   + /oak:index/index2
      - ...
      + indexRules
        + app:Asset
          + properties
            + title
              - facets = true
              - nodeScopeIndex = true
            + tags
              - facets = true
              - propertyIndex = true
or
   + /oak:index/index3
      - ...
      + indexRules
        + app:Asset
          + properties
            + nodeType
              - propertyIndex = true
              - name = jcr:primaryType
            + title
              - facets = true
            + tags
              - facets = true

SELECT rep:facet(title) FROM [app:Asset] WHERE ISDESCENDANTNODE('/some/path')

+ /oak:index/index2
  - ...
  - evaluatePathRestrictions = true
  + indexRules
    + app:Asset
      + properties
        + title
          - facets = true
          - propertyIndex = true

SELECT rep:facet(title) FROM [app:Asset] WHERE CONTAINS(., 'foo')

+ /oak:index/index2
  - ...
  + indexRules
    + app:Asset
      + properties
        + title
          - facets = true
          - propertyIndex = true
          - nodeScopeIndex = true

SELECT rep:facet(title) FROM [app:Asset] WHERE [title] IS NOT NULL

+ /oak:index/index2
  - ...
  + indexRules
    + app:Asset
      + properties
        + title
          - facets = true
          - propertyIndex = true