A - out-of-band indexing

Out of band indexing has following phases

1. Get checkpoint issued
2. Perform indexing with read only connection to NodeStore upto checkpoint state
3. Import the generated indexes
4. Complete the increment indexing from checkpoint state to current head

 java -jar oak-run*.jar index --reindex \
 --index-paths=/oak:index/indexName \
 --checkpoint=0fd2a388-de87-47d3-8f30-e86b1cf0a081 \	
 --fds-path=/path/to/datastore  /path/to/segmentstore/ 

java -jar oak-run*.jar index --index-import --read-write \
    --index-import-dir=<index dir>  \
    --fds-path=/path/to/datastore /path/to/segmentstore

B - Online indexing

Online indexing automates some of the manual steps which are required for out-of-band indexing.

This mode should only be used when repository is from Oak version 1.7+ as oak-run connects to the repository in read-write mode.

java -jar oak-run*.jar index --reindex --index-paths=/oak:index/lucene --read-write --fds-path=/path/to/datastore /path/to/segmentstore

Updating or Adding New Index Definitions

@since Oak 1.7.5

Index tooling support updating and adding new index definitions to existing setups. This can be done by passing in path of a json file which contains index definitions

java -jar oak-run*.jar index --reindex --index-paths=/oak:index/newAssetIndex \
--index-definitions-file=index-definitions.json \
--fds-path=/path/to/datastore /path/to/segmentstore  

Where index-definitions.json has following structure

{
  "/oak:index/newAssetIndex": {
    "evaluatePathRestrictions": true,
    "compatVersion": 2,
    "type": "lucene",
    "async": "async",
    "jcr:primaryType": "oak:QueryIndexDefinition",
    "indexRules": {
      "jcr:primaryType": "nt:unstructured",
      "dam:Asset": {
        "jcr:primaryType": "nt:unstructured",
        "properties": {
          "jcr:primaryType": "nt:unstructured",
          "valid": {
            "name": "valid",
            "propertyIndex": true,
            "jcr:primaryType": "nt:unstructured",
            "notNullCheckEnabled": true
          },
          "mimetype": {
            "name": "mimetype",
            "analyzed": true,
            "jcr:primaryType": "nt:unstructured"
          }
        }
      }
    }
  }
}

Some points to note about this json file

• Each key of top level object refers to the index path
• The value of each such key refers to complete index definition
• If the index path is not present in existing repository then it would result in a new index being created
• In case of new index it must be ensured that parent path structure must already exist in repository. So if a new index is being created at /content/en/oak:index/contentIndex then path upto /content/en/oak:index should already exist in repository
• If this option is used with online indexing then do ensure that oak-run version matches with the Oak version used by target repository

You can also use the json file generated from Oakutils. It needs to be modified to confirm to above structure i.e. enclose the whole definition under the intended index path key.

In general the index definitions does not need any special encoding of values as Index definitions in Oak use only String, Long and Double types mostly. However if the index refers to binary config like Tika config then the binary data would need to encoded. Refer to next section for more details.

This option is supported in both online and out-of-band indexing.

For more details refer to OAK-6471

JSON File Format

Some of the standard types used in Oak are not supported directly by JSON like names, blobs etc. Those would need to be encoded in a specific format.

Below are the encoding rules

LONG

No encoding required

“compatVersion”: 2

BOOLEAN

No encoding required

“propertyIndex”: true,

DOUBLE

No encoding required

“weight”: 1.5

STRING

Prefix the value with str:

Generally the value need not be encoded. Encoding is only required if the string starts with 3 letters and then colon

“pathPropertyName”: “str:jcr:path”

DATE

Prefix the value with dat:. The value is ISO8601 formatted date string

“created”: “dat:2017-07-20T13:23:21.196+05:30”

NAME

Prefix the value with nam:.

For jcr:primaryType and jcr:mixins no encoding is required. Any property with these names would be converted to NAME type

“nodetype”: “nam:nt:base”

PATH

Prefix the value with pat:

“imagePath”: “pat:/content/assets/book.jpg”

URI

Prefix the value with uri:

“serverURI”: “uri:http://foo.example.com”

BINARY

By default the binary values are encoded as Base64 string if the binary is less than 1 MB size. The encoded value is prefixed with :blobId:

“jcr:data”: “:blobId:axygz”

Tika Setup

If the indexes being reindex have fulltext indexing enabled then you need to include Tika library in classpath. This is required even if pre extraction is used so as to ensure that any new binary added after pre-extraction is done can be indexed.

First download the tika-app jar from Tika downloads. You should be able to use 1.15 version with Oak 1.7.4 jar.

Then modify the index command like below. The rest of arguments remain same as documented before.

java -cp oak-run.jar:tika-app-1.15.jar org.apache.jackrabbit.oak.run.Main index