% Licensed to the Apache Software Foundation (ASF) under one or more
% contributor license agreements. See the NOTICE file distributed with
% this work for additional information regarding copyright ownership.
% The ASF licenses this file to You under the Apache License, Version 2.0
% (the ``License''); you may not use this file except in compliance with
% the License. You may obtain a copy of the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an ``AS IS'' BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.

\begin{changemargin}{1.5in}{0in}

\section{Overview}

The MetaCarta GTS appliance indexes documents and allows users
to search these documents based on both keywords and geographic
references. The MetaCarta Connector Framework allows system
administrators to configure connections to repositories and define
jobs to maintain synchronization between the repositories and the GTS
index. The Connector Framework includes five different connectors. The
Documentum Connector creates connections to EMC\circler\linebreak
Documentum\circler~repositories. The JDBC Connector creates connections
to SQL repositories, with support for Oracle\circler, Postgres SQL,
and Microsoft\circler~SQL Server\circler\linebreak (Version greater
than 6.5) repositories.  The Livelink Connector creates connections to
Open Text\circler~Livelink\circler~repositories. The Share Connector
creates connections to network shares, with full support for Microsoft
Windows\circler~shares and Samba shares, and limited support for
NetApp\circler~shares. The RSS Connector creates connections to RSS feeds.

This document specifies the means for connecting to these repositories,
indexing files from these repositories, and maintaining connections to
these repositories.

\subsection{Assumptions}

This document assumes you have a basic level of familiarity with GTS
appliance administration. This document also assumes that you have a
basic understanding of the repositories to which you are trying to
connect. If you need more information about the MetaCarta GTS
appliance, please read the \documentref{MetaCarta GTS Administrator's
Guide} stored on the appliance at
\dirpath{/usr/share/doc/metacarta/AdminGuide.pdf}. For more
information about your repositories, please see your repository
documentation or repository administrator.

Throughout this document, we assume that your appliance is named \\
\url{metacarta.example.com}. 


%What is the name of the product package? I am just calling it
% Connector Addon for now.

\section{Installation}

The Connector Framework is part of the Connector Addon available from
Metacarta. \input{install-input}

\section{Configuration}

\subsection{Access to the Connector}

To use the Connector Framework, you \input{access-input}

\subsection{Initial Repository Configuration}

Some repositories and security models require additional configuration
to the GTS appliance. In most cases you will need to perform these
setup steps before using the crawler interface. You may need the help
of your GTS appliance administrator to complete these steps.

\subsubsection{Initial Active Directory Configuration}

The Documentum Connector, Livelink Connector, and Share Connector all
support the Active Directory security model. \input{active-directory}

\subsubsection{Initial Documentum Configuration}

\input{Docu-setup}

\section{Collecting Documents From Repositories} % Retitle this, yo.

The Connector Framework manages retrieving documents from
different repositories through \emph{jobs}. Jobs can be scheduled
to run regularly; each job connects to a single repository using a
particular set of credentials. Each job is tied to a \emph{repository
connection}. Repository connections contain information allowing the
connector framework to connect to a given repository. A repository
connection may also be tied to a repository-specific \emph{authority
connection}. These authority connections manage document security,
making sure that when files have been indexed on your GTS appliance,
only authorized users are able to view them as search results. Before
you can create a job, you must create a repository connection for the job
to use. Each repository connection should be set up with an appropriate
authority. A standard authority connection is available for use with any
connector, while it is possible to create specific authority connections
for use with the Documentum Connector and Livelink Connector.

\subsection{Creating Authority Connections}

If you are creating a connection to a Documentum or Livelink
repository, you may wish to create an authority connection. A
Documentum or Livelink authority connection allows the GTS appliance
to connect to the security authority for that repository and enables
the appliance to enforce the security model of that repository.


If you wish to customize the security model for ingested documents on
the GTS appliance or are configuring a connection to a SQL database,
RSS feed, or network share, you may skip this section and use the
standard (Kerberos) authority connection when creating your repository
connection. When the standard connection is selected, the GTS appliance
will attempt to look up Active Directory authorization information for all
users of the search interface. This is appropriate when security should
be disabled for all crawl jobs using a given repository connection,
as in the case of a SQL database connection, or when the repository
itself uses Active Directory for security, as in the case of a network
share connection.

To create an authority connection, first go to the
main Connector Framework Administration interface at
\url{metacarta.example.com/crawler/}.  By default, your username and
password are the Ingestion Basic Auth username and password defined
earlier. 

You will see a sidebar like the one to the left. Click on ``List
Authority Connections'' and then you will be presented with the
list of authority connections. Click ``Add a new connection.''
You will see the following:

\includegraphics[width=300pt]{Generic-edit-authority-tab1}

This is the first tab of the tabbed interface you will use to edit
authority connections. The next tab is as follows:

\includegraphics[width=300pt]{Generic-edit-authority-tab2}

\input{sidebar}

In the first two tabs you must provide a name, description, and authority
type for your new authority connection. The name should be unique on
this appliance, as you will use it to select this connection later when
making repository connections.

\note{If this appliance will be used for Aggregated Search, the authority
name for this authority must be the same on every appliance aggregating
search results. This may require coordinating authority names with other
Connector Framework administrators at other sites. For more information
on Aggregated Search, please see the \documentref{MetaCarta Appliance
Administrator's Guide}.}

The description should explain the authority connection to you or another
administrator. The authority type is the type of authority to which you
will connect.

Once you have filled in those tabs, click ``Continue'' to move on to
repository-specific options.

\input{Docu-edit-auth}

\input{Live-edit-auth}

\subsection{Creating Repository Connections}

Once you have created an authority connection, you need to create a
repository connection.
To do so, click ``List Repository Connections'' on the sidebar menu. Then,
when presented with the list of repository connections, click ``Add a
new connection.'' You will see the following two tabs:

\includegraphics[width=300pt]{Docu-edit-repository-tab1}

\includegraphics[width=300pt]{Docu-edit-repository-tab2}

Now you must provide a name, description, connector type, and authority
type for your new repository connection. The name should be unique,
as you will use it to select this connection later when defining
jobs. The description should explain the repository connection to you
or another administrator.  The connector type is the type of repository
from which you will get documents. The authority type is the type of
authority from which you will get authorization information. In the case
of a Documentum Connector or Livelink Connector, you may have set up an
authority connection. If you select that authority connection here, the
permissions associated with the ingested documents will be the same as the
permissions of the original documents in the repository. If you intend
to force AD ACLs on your documents or are setting up a JDBC Connector,
RSS Connector, or Share Connector, then you should select ``Standard
(Kerberos)'' here.

Once you have filled in those tabs and click continue to move on to
the repository-specific options.

\input{Docu-edit-repo}

\input{JDBC-edit-repo}

\input{Live-edit-repo}

\input{Share-edit-repo}

\input{RSS-edit-repo}

\subsection{Creating and Running Jobs}

To run a job, click ``Status and Job Management'' on the sidebar menu.
You can run or edit existing jobs from this menu.

To create a new job, click ``List All Jobs'' on the sidebar menu. Then, when
presented with the list of current jobs, click ``Add a new job.'' You
will be presented with two tabs, in which you must fill in the following
information:

\includegraphics[width=300pt]{Generic-edit-job-tab1}

\begin{itemize}

\item \textbf{Name:} The name of the job. You will use this to identify
the job later.

\item \textbf{Collection:} The collection name metadata for all
documents in this job. Users of the MetaCarta Web Search Interface or
other search tools can use this name to select the set of documents in
this job. For more information on collection name metadata, please see
the \documentref{MetaCarta Appliance Administrator's Guide}.

\item \textbf{Document Template:} The document template with which you
would like all documents in this job ingested.  Document Templates allow
you to define what parts of a document should be indexed, what parts
should be ignored, and what date to use as the principal date of the
document. For more information, please see the \documentref{MetaCarta
Document Templates Integrator's Guide}.


\end{itemize}

\includegraphics[width=300pt]{Generic-edit-job-tab2}

\begin{itemize}

\item \textbf{Connection:} The name of the repository connection you
wish to use for this job. You select this from the list of repository
connections you have already made. You may have more than one job use
the same repository connection, but if you have two jobs crawl the same
documents, the documents will have the metadata and collection name
associated with whatever job crawled the document most recently. This
will cause unpredictable results when searching those collections,
searching for those documents, or trying to delete those collections.
We recommend never crawling the same document in two different jobs.

\item \textbf{Start method:} When you want the job to start
automatically. The job can automatically start the next time it is
scheduled to run (``Start when schedule window starts''), or not at
all (``Don't automatically start this job''). If you select ``Start
even inside a schedule window'', the job will start whenever its
schedule window is open and it is not running. Thus, if the job
finishes during its one of its schedule windows, it will restart
immediately. If it is created during one if its schedule windows, it
will start immediately. Scheduling windows will be explained in the
``Scheduling'' section.

\item \textbf{Priority:} From 1 (highest) to 10 (lowest), the priority
this crawl should have if it must compete for resources with other
crawls on the appliance. You should not need to change this unless you
are running more than one crawl at the same time; if you are, assign a
higher priority to the crawls whose documents you want to be processed
preferentially before documents from other jobs.

\end{itemize}

After filling in those options, click ``Continue,'' and you will be
presented with additional repository-specific tabs. 

\input{Docu-edit-job}

\input{JDBC-edit-job}

\input{Live-edit-job}

\input{Share-edit-job}

\input{RSS-edit-job}

\subsection{Status and Job Management}\label{ManageJobs}

You can then look at the status of your job by clicking ``Status and 
Job Management'' on the sidebar. You will see a list of one or more jobs
much like this one:

\includegraphics[width=300pt]{Generic-jobs-list}

You can start any crawl you like immediately from this interface by
clicking ``Start'' next to the crawl's name. This interface also allows
you to see how many documents have been crawled; this information may help
you structure and plan future crawls. ``Active'' documents are currently
known documents that will need to be processed during the current job run.
``Processed'' documents have been processed at least once, in the current
run or previous runs.

\note{Refresh this page by clicking the ``Refresh'' link at the bottom
of the page, not by clicking your browser's reload button.}

\section{Reports}

The Connector Framework interface can generate two types of status
reports, or reports on the current crawl status, and four types of
history reports, reports on crawls already going on. The six types are:

\begin{itemize}

\item Document Status, which lets you find information on individual 
documents currently part of a job 

\item Queue Status, which lets you aggregate information about groups
of documents currently part of a job 

\item Simple History, which lets you list an ordered set of log events
based on chosen criteria

\item Maximum Activity, which lets you see the period of time in
which a certain event happened most often

\item Maximum Bandwith, which lets you see the period of time in
which the most bandwidth was used 

\item Result Histogram, which provides log information that would be
appropriate for constructing a histogram or other diagram

\end{itemize}

\subsection{Status Reports}


\subsubsection{Document Status}

This report was generated by selecting ``MetaCarta Sample,'' selecting
the document state ``Documents processed at least once,'' selecting all 
possible document statuses, clicking continue, selecting ``Sample Job,''
and clicking Go.

\includegraphics[width=300pt]{Generic-document-report}

\begin{itemize}

\item \textbf{Connection:} The repository connection from which to 
generate a report. You must select the repository connection and click
Continue to see all repository-specific options. %Are there any?

\item \textbf{Time offset from now:} Defaults to zero. Allows you to
see estimates of future status or, with negative numbers, a record of
past status.

\item \textbf{Document state:} Allows you to select documents that
have not yet been processed or documents that have been processed
at least once.

\item \textbf{Document status:} Allows you to choose one or more 
statuses of document to report on. The statuses you can choose are:

\begin{itemize}

\item Documents that are no longer active

\item Documents currently in progress

\item Documents currently being expired

\item Documents currently being deleted

\item Documents currently available for processing

\item Documents currently available for expiration

\item Documents not yet processable

\item Documents not yet expirable

\end{itemize}

\item \textbf{Document identifier match:} A regular expression allowing
you to see only certain document identifiers.

\item \textbf{Jobs:} The job or jobs for which you want to generate
a report.

\end{itemize}

You can sort this report by any of the returned fields; to do so,
click the field names.

\subsubsection{Queue Status}

This report was generated by selecting ``MetaCarta Sample,'' selecting
the document state ``Documents processed at least once,'' selecting all
possible document statuses, clicking continue, selecting ``Sample Job,''
setting the identifier class description to (/9/9/9), and clicking Go.

\includegraphics[width=300pt]{Generic-queue-report}

This form offers the same fields as the Document Status report with
one addition, the \textbf{Identifier class description}, which allows
you to group results based on a regular expression. In this case, any
document whose path contains ``/9/9/9'' is collected into a group, and
that group's documents are analyzed  separately from the other documents
selected. The ungrouped documents are all analyzed together in the first
row of the results table. If the identifier class description were just
``.*'' then each document would be shown individually as a group of one.

\subsection{History Reports}

Each of these reports allows you to specify a connection, one or more
activities, a start time, an end time, an entity match, and a result code
match.  Some also allow you to specify an identifier class description
and a sliding window size. This section will show sample results for
each type of report and an explanation of the fields selected.

\subsubsection{Simple History}

This report was generated by selecting ``Sample Repository Connector,'' 
clicking Continue, selecting ``document ingest,'' and clicking Go.

\includegraphics[width=300pt]{Generic-simple-history-report}

\begin{itemize}

\item \textbf{Connection:} The repository connection from which to
generate a report. You must select the repository connection and click
Continue to see all repository-specific options.

\item \textbf{Activities:} What crawler activities you would like to
see.  The basic set of activites that can be documented in reports for
all repository connections are document ingest, document remove, job
abort, job continue, job end, job start, and job wait. Some activities
are repository connection-specific:

\begin{itemize}

\item \textbf{Documentum:} fetch

\item \textbf{Livelink:} fetch document, find documents

\item \textbf{Share:} access

\end{itemize}

\item \textbf{Start time:} The earliest time in the crawler logs to be
considered for this query.  Choose ``Not specified'' for any field to
start at the beginning of the crawler's logs.

\item \textbf{End time:} The latest time in the crawler logs to be
considered for this query. Choose ``Not specified'' for any field 
to end at the current time.

\item \textbf{Entity match:} A regular expression (see page
\pageref{regex}) to limit the Identifier field. If the entity match
field in the example above had been \texttt{\^{}17}, only document
fetches with Identifiers starting with 17 would be shown.

\item \textbf{Result code match:} A regular expression to limit the
ResultCode field.

\end{itemize}

You can sort the history report by any of the returned fields; to do so,
click the field names.


\subsubsection{Maximum Activity}

This report was generated by selecting ``Sample Repository Connector,''
clicking Continue, selecting ``document ingest,'' changing the Identifier
class description, and clicking Go.

\includegraphics[width=300pt]{Docu-maximum-activity-report}

This form offers two more fields than the previous form:

\begin{itemize}

\item \textbf{Identifier class description:} A regular expression
that determines how to group identifiers together. If this were set to
\texttt{(.*)}, there would be no grouping, and so there would be only one
ingestion event per document. If this were set to \texttt{(17)},
then all documents with identifiers beginning with 17 would be grouped
together. The setting in the example, \texttt{()}, groups all
documents together. Some other possibilities:

\begin{itemize}

\item \texttt{1(.)}: (up to) Ten groups of documents whose identifier starts
with 1, labeled 0-9, grouped by the second digit of their identifier.

\item \texttt{()}: One group of documents containing all documents 
regardless of identifier.

\item \texttt{17}: One group of documents whose identifier contains
the string ``17.''

\item \texttt{\^{}17}: One group of documents whose identifier \emph{starts}
with the string ``17.''

\end{itemize}

Document identifiers vary from repository to repository. For Documentum
and Livelink repositories, the unique part of the document identifier will
simply be a number. Documentum uses a hexadecimal document identifier,
while Livelink uses standard base 10 integers as document identifiers. For
network shares accessed using the Share Connector, a document identifier
is typically the document's path. For SQL databases accessed using
the JDBC Connector, the document identifier is set in the database and
can take any form as specified in the database.

\item \textbf{Sliding window size}: The search interval in minutes.

\end{itemize}

The report returned will have only one result per group with one or more
documents in it, if there is a clear highest activity rate, or a list of
all the results tied for highest activity rate if there are more than one.

\subsubsection{Maximum Bandwidth}

This report was generated by selecting ``Sample Repository Connector,''
clicking Continue, selecting ``document ingest,'' changing the Identifier
class description, and clicking Go.

\includegraphics[width=300pt]{Generic-maximum-bandwidth-report}

This form offers the same fields as the maximum activity form, and
returns similar results; instead of tracking events per time window,
it tracks the window with the highest average bandwith usage, measured
in bytes per second. Again, the identifier class description has been
changed to a regular expression that will match all identifiers (and
thus in this case documents).

\subsubsection{Result Histogram}

This report was generated by selecting ``Sample Repository Connector,''
clicking Continue, selecting ``document ingest,'' and clicking Go.

\includegraphics[width=300pt]{Generic-activity-result-report}

This form adds one new field:

\begin{itemize}

\item \textbf{Result code class description:} A regular expression that
determines how to group result classes together; like Identifier class
descriptions but for result classes.

\end{itemize}

This report does not produce an actual histogram, but provides data that
could be used to generate histograms.  

\section{Troubleshooting}

If you see the following error message, or a similar error message:

\includegraphics[width=300pt]{no-mucking-error}

then it is likely that your appliance is part of a Standby Appliance
pair. If your appliance is a standby appliance, this error is expected;
you should make configuration changes only on the primary appliance.
If your appliance is a primary appliance, this message should mean either
that your standby appliance is down and your primary appliance will
not allow the ingestion of new documents or changing of configuration,
or that your primary appliance is in the process of being paired to
a standby appliance. Contact your appliance administrator, or read
the Standby Appliance section of the \documentref{MetaCarta Appliance
Administrator's Guide}, for more information.

If you see this error message while not part of a Standby
Appliance pair, please contact Customer Support (see page \pageref{SupportContact}).

\end{changemargin}
