% 
% Licensed to the Apache Software Foundation (ASF) under one
% or more contributor license agreements.  See the NOTICE file
% distributed with this work for additional information
% regarding copyright ownership.  The ASF licenses this file
% to you under the Apache License, Version 2.0 (the
% "License"); you may not use this file except in compliance
% with the License.  You may obtain a copy of the License at
% 
%   http://www.apache.org/licenses/LICENSE-2.0
% 
% Unless required by applicable law or agreed to in writing,
% software distributed under the License is distributed on an
% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
% KIND, either express or implied.  See the License for the
% specific language governing permissions and limitations
% under the License.
% 

      \section{Overview.} 
      A DUCC service is defined by the following two criteria:
      \begin{itemize}
          \item A service is one or more long-running processes that await requests from
            UIMA pipeline components and return something in response.  These processes
            are usually managed by DUCC but need not be.
          \item A service is accompanied by a small program called a ``pinger'' that
            the DUCC Service Manager uses to gauge the availability and health of the
            service.  This pinger must always be be present; however, DUCC will
            supply a default pinger for UIMA-AS services.
      \end{itemize}

      A service is usually a UIMA-AS service, but DUCC supports any arbitrary process
      as a service.

      The DUCC service manager implements several high-level functions:
      
      \begin{itemize}
          \item Ensure services are available for jobs before allowing the jobs to start. This is a fail-fast
            to prevent unnecessary allocation of resources (with potential eviction of healthy processes)
            for jobs that can't run, as well as quick feedback to users that something is amiss.
      
          \item Manage the start-up and management of services: allocate resources, spawn the
            processes, manage the pingers, ensure the processes stay alive, handle errors, etc.
      
          \item Report on the state and availability of services.
       \end{itemize}


    \section{Service Types.}
    \label{sec:services.types}
      DUCC supports two types of services: UIMA-AS and CUSTOM:
      
      \begin{description}
          \item[UIMA-AS] This is a "normal" UIMA-AS service. DUCC fully supports all aspects of UIMA-AS
            services with minimal effort from developers.  A default ``pinger'' is supplied by DUCC
            for UIMA-AS services.  (It is legal to define a CUSTOM pinger for a UIMA-AS service,
            however.)
            
          \item[CUSTOM] This is any arbitrary service.  Developers must provide a CUSTOM pinger
            and declare it in the service registration.            
      \end{description}

      DUCC also supports the concept of a service that is not managed by DUCC.  For example, a
      database or a search engine may be better managed with other facilities.  In order to manage
      dependencies by jobs on this type of service, DUCC supports a CUSTOM service that supplies
      only a ``pinger'' and no other process.  This is known as a ``ping-only'' service.

      \section{Service References and Endpoints} 
      \label{sec:service.endpoints}
      Services are referenced by an entity called a service
      endpoint. The service endpoint is a formatted string used to uniquely identify each
      service and to supply contact information to the pingers.  A service endpoint
      is of the form 
\begin{verbatim}
      <service-type>:<unique id and contact information>
\end{verbatim}
      
      The {\em service-type} must be either UIMA-AS or CUSTOM.
      
      The {\em unique id and contact information} is any string needed to ensure the service is
      uniquely named.  This string is passed to the service pinger and may contain 
      information for the pinger to contact the service.  For UIMA-AS services, service endpoint is
      inferred by the CLI by inspection of the UIMA-AS service's DD XML descriptor.  The UIMA-AS
      service endpoint is always of the form:
\begin{verbatim}
      UIMA-AS:queue-name:broker-url
\end{verbatim}
      where {\em queue-name} is the name of the ActiveMQ queue used by the service, and {\em broker-url}
      is the ActiveMQ broker URL.

      Jobs or other services may register dependencies on specific services by listing one or more
      service endpoints int their specifications. See the 
      \hyperref[sec:cli.ducc-submit]{\em job } and 
      \hyperref[sec:cli.ducc-services]{\em services } CLI descriptions for details.
           
      All services must be registered with DUCC.  It is
      possible to register a ``ping-only'' service that has no process managed by DUCC,
      consisting only of a pinger.  
            
      A service is registered with DUCC using the
      \hyperref[sec:cli.ducc-services]{ducc\_services} CLI. Service registrations are persisted by
      DUCC and last over DUCC and cluster restarts.

      An incoming job which references a service that is not registered is marked ``Services
      Unavailable'' and canceled by the system.

      There are several variants on services:
      \begin{description}

        \item[Autostarted Services] An autostarted service is a  service that is
          automatically started when the DUCC system is started. When DUCC is started, the SM checks the
          service registry for all service that are marked for automatic start-up.  If registered for autostart,
          the DUCC Service Manager submits the registered number of instances
          on behalf of the registering user.  If an instance should die, DUCC automatically restarts
          the instance.  Short of massive failures, DUCC will ensure the service is always running
          and immediately ready for use with no manual intervention.
          
        \item[On-Demand Services] An on-demand service is a registered service that is started only
          when referenced by the service-dependency of another job or service. If the service is
          already started (e.g. by reference from some other job), the dependent job/service is
          marked ready to schedule as indicated above. If not, the service registry is checked and
          if a matching service is found, DUCC starts it. When the service has completed
          initialization a pinger is started and all jobs waiting on it are then started.
          
          When the last job or service that references the on-demand service exits, a timer is
          established to keep the service alive for a while (in anticipation that it will be needed
          again soon.)  When the keep-alive timer expires, and there are no more dependent
          jobs or services, the on-demand service is automatically stopped to free up its resources for
          other work.

        \item[Ping-Only Services] 
          \phantomsection\label{subsub:services.ping-only}
          Ping-only services consist of only
          a ping thread.  The service itself is not managed in any way by DUCC.  This is useful for
          managing dependencies on services that are not under DUCC control: DUCC can detect and
          report on the health of these services and take appropriate actions on dependent jobs if
          the services are not responsive.
      \end{description}
          
      \section{Service Pingers}
      \label{sec:service.pingers}
      A service pinger is a small program that queries a service on behalf of the
      DUCC Service Manager to:
      \begin{itemize}
        \item Report on the availability of the service, and
        \item Report on the health of the service.
      \end{itemize}
      
      Service pingers are always written in Java and must implement an abstract class,
\begin{verbatim}
      org.apache.uima.ducc.common.AServicePing
\end{verbatim}
      When a service is deployed by
      DUCC, the Service Manager spawns a DUCC process that instantiates the pinger for
      the service.  On a regular basis, the Service Manager sends a request to the pinger
      to query the service health.  The pinger is expected to respond within a reasonable
      period of time and if it fails to do so, the service is marked unresponsive.

      \subsection{Declaring a Pinger in A Service}

      If your service is a UIMA-AS service, there is no need to create or declare a pinger as  DUCC
      provides a default pinger.  If a CUSTOM pinger is required, it must be declared in the service
      descriptor, and the service must be registered.  See
      \hyperref[sec:cli.ducc-services]{ducc\_services} for details on service registration specifying
      ping directives.

      \subsection{Implementing a Pinger}
      Pingers must implement the class {\tt org.apache.uima.ducc.cli.AServicePing}.  The abstract class
      is shown below.
      \begin{figure}[H]
\begin{verbatim}
import org.apache.uima.ducc.common.IServiceStatistics;

public abstract class AServicePing
{
    /**
     * Called by the ping driver, to pass in useful things the pinger may want.
     * @param arguments This is passed in from the service specification's
     *                  service_ping_arguments string.
     *
     * @param endpoint This is the name of the service endpoint, as passed in
     *                 at service registration.
     */
    public abstract void init(String arguments, String endpoint)  throws Exception;

    /**
     * Stop is called by the ping wrapper when it is being killed.  Implementors may optionally
     * override this method with conenction shutdown code.
     */
    public abstract void stop();

    /**
     * Returns the object with application-derived health and statistics.
     * @return This object contains the informaton the service manager and web server require
     *     for correct management and display of the service.
     */
    public abstract IServiceStatistics getStatistics();    
}

\end{verbatim}
        \caption{Service Ping Abstract Class}
        \label{fig:service.abstract.pinger}

      \end{figure}
 
      Here we show the class {\em org.apache.uima.ducc.common.IServiceStatistics}.  Custom pingers must
      implement this class and return an instance in response to AServicePing.getStatistics().  A default
      ServiceStatistics class is provide and may be used by custom pingers as
      {\em org org.apache.uima.ducc.cliServiceStatistics}.  See the Javadoc for more complete details
      of these classes.

      \begin{figure}[H]
\begin{verbatim}
public interface  IServiceStatistics
    extends Serializable
{

    /**
     * Query whether the service is alive. This is used internally by the Service Manager.
     *
     * @return "true" if the service is responsive, "false" otherwise.
     */
    public boolean isAlive();

    /**
     * Query wether the service is "healthy". This is used internally by the Service Manager.
     * @return "true" if the service is healthy, "false" otherwise.
     */
    public boolean isHealthy();

    /**
     * Return service statistics, if any. This is used internally by the Service Manager.
     * @return A string containing information regarding the service. 
     */
    public String  getInfo();

    /**
     * Set the "aliveness" of the service.  This is called by each pinger for each service.  Set
     *  this to return "true" if the service is responsive.  Otherwise return "false" so the Service
     *  Manager can reject jobs dependent on this service.
     * @param alive Set to "true" if the service is responseve, "false" otherwise.
     */
    public void setAlive(boolean alive);
 
    /**
     * Set the "health" of the service.  This is called by each pinger for each service.  This is a
     * subject judgement made by the service owner on behalf of his own service.  This is used only
     * to reflect status in the DUCC Web Server.
     * @param healthy Set to "true" if the service is healthy, "false" otherwise.
     */
    public void setHealthy(boolean healthy);

    /**
     * Set the monitor statistics for the service. This is any arbitray string describing critical
     * or useful characteristics of the service.  This string is presented as a "hover" in the
     * webserver over the "health" field.
     * @param info This is an arbitrary string summarizing the service's performance. 
     */
    public void setInfo(String info);

}
\end{verbatim}
        \caption{IServiceStatistics Interface}
        \label{fig:service.iservicestatistics}

      \end{figure}

      Below is a sample CUSTOM pinger for a hypothetical service that returns four integers in
      response to a ping.
      \begin{figure}[H]
\begin{verbatim}
import java.io.DataInputStream;
import java.io.InputStream;
import java.net.Socket;
import org.apache.uima.ducc.cli.AServicePing;
import org.apache.uima.ducc.cli.ServiceStatistics;

public class CustomPing
    extends AServicePing
{
    String host;
    String port;
    public void init(String args, String endpoint) throws Exception {
        // Parse the service endpoint, which is a String of the form 
        //    host:port
        String[] parts = endpoint.split(":");
        host = parts[1];
        port = parts[2];
    }

    public void stop()  {  }

    private long readLong(DataInputStream dis) throws Exception {
        return Long.reverseBytes(dis.readLong());
    }

    public ServiceStatistics getStatistics() {
        // Contact the service, interpret the results, and return a state
        // object for the service.
        ServiceStatistics stats = new ServiceStatistics(false, false,"<NA>");
        try {
            Socket sock = new Socket(host, Integer.parseInt(port));
            DataInputStream dis = new DataInputStream(sock.getInputStream());

            long stat1 = readLong(dis); long stat2 = readLong(dis); 
            long stat3 = readLong(dis); long stat4 = readLong(dis);

            stats.setAlive(true);  stats.setHealthy(true);
            stats.setInfo(  "S1[" + stat1 + "] S2[" + stat2 + 
                            "] S3[" + stat3 + "] S4[" + stat4 + "]" );
        } catch ( Throwable t) {
        	t.printStackTrace();
            stats.setInfo(t.getMessage());
        }
        return stats;        
    }
}
\end{verbatim}
        \caption{Sample UIMA-AS Service Pinger}
        \label{fig:service.custom.pinger}

      \end{figure}
      
      \subsection{Building And Testing Your Pinger}
      This section provides the information needed to use the pinger API and build a
      custom pinger. 

      \paragraph{1. Establish compile CLASSPATH} One DUCC jar is required in the CLASSPATH to build your pinger:
\begin{verbatim}
     $DUCC_HOME/lib/uima-ducc-cli.jar
\end{verbatim}      
      This provides the definition for the {\em AServicePing} and {\em ServiceStatistics} classes.

      \paragraph{2. Create a registration}Next, create a service registration for the pinger.  While
      debugging, set the directive
\begin{verbatim}
     service_ping_dolog = true
\end{verbatim}
      This will log any output from  {\tt System.out.println()} to your home directory in
\begin{verbatim}
     $HOME/ducc/logs/services
\end{verbatim}

      Once the pinger is debugged you may want to turn logging off:
\begin{verbatim}
     service_ping_dolog = false
\end{verbatim}
      
      A sample service registration may look something like the following:
\begin{verbatim}
     bash-3.2$ cat myping.svc

     description              = Ping-only service
     service_request_endpoint = CUSTOM:localhost:7175
     service_ping_class       = CustomPing
     service_ping_classpath   = /myhome/CustomPing.class:$DUCC_HOME/lib/uima_ducc_cli.jar
     service_ping_dolog       = true
     service_ping_timeout     = 500
     service_ping_aruments    = Arg1 Arg2
     service_ping_jvm_args   = -DXmx50M
\end{verbatim}
       
      \paragraph{3. Register and start the pinger} Start up your custom service so the pinger has something to contact, then start
      the pinger.  It may be easier to debug the pinger if you initially start the service outside of DUCC. Once
      the pinger is working it is straightforward to integrate it into the pinger's service registration by merging
      the registration for the pinger with the registration for the service.

      That's it!  Check the web server to make sure the service ``comes alive''.  Check your pinger's
      debugging log if it doesn't.  Once registered, you can stop and start the pinger at will using
\begin{verbatim}
     ducc_services --start <serviceid>
     ducc_services --stop <serviceid>
\end{verbatim}
     where $<$serviceid$>$ is the id returned when you registered the pinger.

     \paragraph{4. If all else fails ...}
     If your pinger does not work and you cannot determine the reason, be sure you enable {\em service\_ping\_dolog} and
     look in your log directory, as most problems with pingers are reflected there.  As a last resort, you can
     inspect the the Service Manager's log in
\begin{verbatim}
     $DUCC_HOME/logs/sm.log
\end{verbatim}
     
