Programmatic Operation

A certain subset of ManifoldCF users want to think of ManifoldCF as an engine that they can poke from whatever other system they are developing. While ManifoldCF is not precisely a document indexing engine per se, it can certainly be controlled programmatically. Right now, there are three principle ways of achieving this control.

Control by Servlet API

ManifoldCF provides a servlet-based JSON API that gives you the complete ability to define connections and jobs, and control job execution. You can read about JSON here. The API is designed to be RESTful in character. Thus, it makes full use of the HTTP verbs GET, PUT, POST, and DELETE, and represents objects as URLs.

URL format

The basic format of the JSON servlet resource URLs is as follows:

http[s]://<server_and_port>/mcf-api-service/json/<resource>

The servlet ignores request data, except when the PUT or POST verb is used. In that case, the request data is presumed to be a JSON object. The servlet responds either with an error response code (either 400 or 500) with an appropriate explanatory message, or with a 200 (OK), 201 (CREATED), 401 (UNAUTHORIZED), or 404 (NOT FOUND) response code along with a response JSON object.

JSON equivalents for ManifoldCF

ManifoldCF treats certain JSON forms as equivalent, for the purposes of readability. For example, the array form "foo" : [ { ... } ] is treated equivalently to "foo" : { }, whenever there is only one array element. This gives a coder some flexibility as to how s/he encodes JSON in requests. Please also be aware that similar compressions will occur in the JSON responses from the API servlet, and your code must be able to deal with this possibility. The following table describes some of the equivalences:

Resources and commands

The actual available resources and commands are as follows:

History query parameters

The history query parameters and their meanings are as follows:

Each report type has different return columns, as listed below:

Queue query parameters

The queue query parameters and their meanings are as follows:

Each report type has different return columns, as listed below:

Authorization domain objects

The JSON fields an authorization domain object has are as follows:

Output connector objects

The JSON fields an output connector object has are as follows:

Transformation connector objects

The JSON fields a transformation connector object has are as follows:

Mapping connector objects

The JSON fields a mapping connector object has are as follows:

Authority connector objects

The JSON fields an authority connector object has are as follows:

Repository connector objects

The JSON fields a repository connector object has are as follows:

Notification connector objects

The JSON fields a repository connector object has are as follows:

Authority group objects

Authority group names, when they are part of a URL, should be encoded as follows:

1. All instances of '.' should be replaced by '..'.
2. All instances of '/' should be replaced by '.+'.
3. The URL should be encoded using standard URL utf-8-based %-encoding.

The JSON fields an authority group object has are as follows:

Output connection objects

Output connection names, when they are part of a URL, should be encoded as follows:

1. All instances of '.' should be replaced by '..'.
2. All instances of '/' should be replaced by '.+'.
3. The URL should be encoded using standard URL utf-8-based %-encoding.

The JSON fields an output connection object has are as follows:

Transformation connection objects

Transformation connection names, when they are part of a URL, should be encoded as follows:

1. All instances of '.' should be replaced by '..'.
2. All instances of '/' should be replaced by '.+'.
3. The URL should be encoded using standard URL utf-8-based %-encoding.

The JSON fields an output connection object has are as follows:

Mapping connection objects

Mapping connection names, when they are part of a URL, should be encoded as follows:

1. All instances of '.' should be replaced by '..'.
2. All instances of '/' should be replaced by '.+'.
3. The URL should be encoded using standard URL utf-8-based %-encoding.

The JSON fields for a mapping connection object are as follows:

Authority connection objects

Authority connection names, when they are part of a URL, should be encoded as follows:

1. All instances of '.' should be replaced by '..'.
2. All instances of '/' should be replaced by '.+'.
3. The URL should be encoded using standard URL utf-8-based %-encoding.

The JSON fields for an authority connection object are as follows:

Repository connection objects

Repository connection names, when they are part of a URL, should be encoded as follows:

1. All instances of '.' should be replaced by '..'.
2. All instances of '/' should be replaced by '.+'.
3. The URL should be encoded using standard URL utf-8-based %-encoding.

The JSON fields for a repository connection object are as follows:

Each throttle object has the following fields:

Notification connection objects

Notification connection names, when they are part of a URL, should be encoded as follows:

1. All instances of '.' should be replaced by '..'.
2. All instances of '/' should be replaced by '.+'.
3. The URL should be encoded using standard URL utf-8-based %-encoding.

The JSON fields for a notification connection object are as follows:

Job objects

The JSON fields for a job are is as follows:

Each pipelinestage object has the following fields:

Each notificationstage object has the following fields:

Each hopcount object has the following fields:

Each schedule object has the following fields:

Each enumeration object describes an array of integers using the form:

{"value":[<integer_list>]}

Each integer is a zero-based index describing which entity is being specified. For example, for "dayofweek", 0 corresponds to Sunday, etc., and thus "dayofweek":{"value":[0,6]} would describe Saturdays and Sundays.

Job status objects

The JSON fields of a job status object are as follows:

Connection-type-specific objects

As you may note when trying to use the above JSON API methods, you cannot get very far in defining connections or jobs without knowing the JSON format of a connection's configuration information, or a job's connection-specific document specification and output specification information. The form of these objects is controlled by the Java implementation of the underlying connector, and is translated directly into JSON, so if you write your own connector you should be able to figure out what it will be in the API. For connectors already part of ManifoldCF, it remains an ongoing task to document these connector-specific objects. This task is not yet underway.

Luckily, it is pretty easy to learn a lot about the objects in question by simply creating connections and jobs in the ManifoldCF crawler UI, and then inspecting the resulting JSON objects through the API. In this way, it should be possible to do a decent job of coding most API-based integrations. The one place where difficulties will certainly occur will be if you try to completely replace the ManifoldCF crawler UI with one of your own. This is because most connectors have methods that communicate with their respective back-ends in order to allow the user to select appropriate values. For example, the path drill-down that is presented by the LiveLink connector requires that the connector interrogate the appropriate LiveLink repository in order to populate its path selection pull-downs. There is, at this time, only one sanctioned way to accomplish the same job using the API, which is to use the appropriate "connection_type/execute/type-specific_command" command to perform the necessary functions. Some set of useful functions has been coded for every appropriate connector, but the exact commands for every connector, and their JSON syntax, remains undocumented for now.

File system connector

The file system connector has no configuration information, and no connector-specific commands. However, it does have document specification information. The information looks something like this:

{"startpoint":[{"_attribute_path":"c:\path_to_files","include":[{"_attribute_type":"file","_attribute_match":"*.txt"},{"_attribute_type":"file","_attribute_match":"*.doc"\,"_attribute_type":"directory","_attribute_match":"*"],"exclude":["*.mov"]]}

As you can see, multiple starting paths are possible, and the inclusion and exclusion rules also can be one or multiple.

Control via Commands

For script writers, there currently exist a number of ManifoldCF execution commands. These commands are primarily rich in the area of definition of connections and jobs, controlling jobs, and running reports. The following table lists the current suite.

Control by direct code

Control by direct java code is quite a reasonable thing to do. The sources of the above commands should give a pretty clear idea how to proceed, if that's what you want to do.

Caveats

The above commands know nothing about the differences between connection types. Instead, they deal with configuration and specification information in the form of XML documents. Normally, these XML documents are hidden from a system integrator, unless they happen to look into the database with a tool such as psql. But the API commands above often will require such XML documents to be included as part of the command execution.

This has one major consequence. Any application that would manipulate connections and jobs directly cannot be connection-type independent - these applications must know the proper form of XML to submit to the command. So, it is not possible to use these command APIs to write one's own UI wrapper, without sacrificing some of the repository independence that ManifoldCF by itself maintains.