Copyright © 2012 The Apache Software Foundation
Copyright © 2012 International Business Machines Corporation
License and Disclaimer The ASF licenses this documentation to you under the Apache License, Version 2.0 (the ”License”); you may not use this documentation except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, this documentation and its contents are distributed under the License on an ”AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Trademarks All terms mentioned in the text that are known to be trademarks or service marks have been appropriately capitalized. Use of such terms in this book should not be regarded as affecting the validity of the the trademark or service mark.
Publication date: August 2015
DUCC is a multi-user, multi-system distributed application. The instuctions below follow a staged installation/verification methodology, roughly as follows:
DUCC is distributed as a compressed tar file. The instructions below assume installation from one of this distribution media. If building from source, the build creates this file in your svn trunk/target directory. The distribution file is in the form
where [version] is the DUCC version; for example, uima-ducc-VERSION-bin.tar.gz (where VERSION is the current DUCC version). This document will refer to the distribution file as the “<distribution.file>”.
Single system installation:
Note: On some systems the default user limits for max user processes (ulimit -u) and nfiles (ulimit -n) are defined too low for DUCC. The shell login profile for user ducc should set the soft limit for max user processes to be the same as the hard limit (ulimit -u ‘ulimit -Hu‘), and the nfiles limit raised above 1024 to at least twice the number of user processes running on the cluster.
Additional requirements for multiple system installation:
Additional requirements for running multiple user processes with their own credentials.
Additional requirements for CGroup containers:
In order to build DUCC from source the following software is also required:
The DUCC webserver server optionally supports direct “jconsole” attach to DUCC job processes. To install this, the following is required:
To (optionally) build the documentation, the following is also required:
More detailed one-time setup instructions for source-level builds via subversion can be found here: http://uima.apache.org/one-time-setup.html\#svn-setup
To build from source, ensure you have Subversion and Maven installed. Extract the source from the SVN repository named above.
Then from your extract directory into the root directory (usually current-directory/trunk), and run the command
or
if you have LaTeX insalled and wish to do the optional build of documentation.
If this is your first Maven build it may take quite a while as Maven downloads all the open-source pre-requisites. (The pre-requisites are stored in the Maven repository, usually your $HOME/.m2).
When build is complete, a tarball is placed in your current-directory/trunk/target directory.
After installation the DUCC documentation is found (in both PDF and HTML format) in the directory ducc_runtime/docs. As well, the DUCC webserver contains a link to the full documentation on each major page. The API is documented only via JavaDoc, distributed in the webserver’s root directory $DUCC_HOME/webserver/root/doc/api.
If building from source, Maven places the documentation in
Although any user ID can be used to run DUCC, it is recommended to create user “ducc” to later enable use of cgroups as well as running processes with the credentials of the submitting user.
If multiple nodes are going to be added later, it is recommended to install the ducc runtime tree on a shared filesystem so that it can be mounted on the additional nodes.
Verification submits a very simple UIMA pipeline for execution under DUCC. Once this is shown to be working, one may proceed installing additional features.
Please note: DUCC is intended for scaling out memory-intensive UIMA applications over computing clusters consisting of multiple nodes with large (16GB-256GB or more) memory. The minimal requirements are for initial test and evaluation purposes, but will not be sufficient to run actual workloads.
This creates a directory with a name of the form “apache-uima-ducc-[version]”.
This directory contains the full DUCC runtime which you may use “in place” but it is highly recommended that you move it into a standard location on a shared filesystem; for example, under ducc’s HOME directory:
We refer to this directory, regardless of its location, as $DUCC_HOME. For simplicity, some of the examples in this document assume it has been moved to /home/ducc/ducc_runtime.
If this script fails, correct any problems it identifies and run it again.
Note that ducc_post_install initializes various default parameters which may be changed later by the system administrator. Therefore it usually should be run only during this first installation step.
This step may be run at any time if you wish to defer it.
That’s it, DUCC is installed and ready to run. (If errors were displayed during ducc_post_install they must be corrected before continuing.)
The post-installation script performs these tasks:
Here we verify the system configuration, start DUCC, run a test Job, and then shutdown DUCC.
To run the verification, issue these commands.
Examine the output of check_ducc. If any errors are shown, correct the errors and rerun check_ducc until there are no errors.
Start_ducc will first perform a number of consistency checks. It then starts the ActiveMQ broker, the DUCC control processes, and a single DUCC agent on the local node.
You will see some startup messages similar to the following:
Now open a browser and go to the DUCC webserver’s url, http://<hostname>:42133 where <hostname> is the name of the host where DUCC is started. Navigate to the Reservations page via the links in the upper-left corner. You should see the DUCC JobDriver reservation in state WaitingForResources. In a few minutes this should change to Assigned. Now jobs can be submitted.
To submit a job,
Open the browser in the DUCC jobs page. You should see the job progress through a series of transitions: Waiting For Driver, Waiting For Services, Waiting For Resources, Initializing, and finally, Running. You’ll see the number of work items submitted (15) and the number of work items completed grow from 0 to 15. Finally, the job will move into Completing and then Completed..
Since this example does not specify a log directory DUCC will create a log directory in your HOME directory under
In this directory, you will find a log for the sample job’s JobDriver (JD), JobProcess (JP), and a number of other files relating to the job.
This is a good time to explore the DUCC web pages. Notice that the job id is a link to a set of pages with details about the execution of the job.
Notice also, in the upper-right corner is a link to the full DUCC documentation, the “DuccBook”.
Finally, stop DUCC:
Additional nodes must meet all prerequisites (listed above).
$DUCC_HOME must be on a shared filesystem and mounted at the same location on all DUCC nodes.
If user’s home directories are on local filesystems the location for user logfiles should be specified to be on a shared filesystem.
Addional nodes are normally added to a worker node group. Note that the DUCC head node does not have to be a worker node. In addition, the webserver node can be separate from the DUCC head node (see webserver configuration options in ducc.properties).
For worker nodes DUCC needs to know what node group each machine belongs to, and what nodes need an Agent process to be started on.
The configuration shipped with DUCC have all nodes in the same ”default” node pool. Worker nodes are listed in the file
During initial installation, this file was initialized with the node DUCC is installed on. Additional nodes may be added to the file using a text editor to increase the size of the DUCC cluster.
DUCC launches user processes through ducc_ling, a small native C application. By default the resultant process runs with the credentials of the user ID of the DUCC application. It is possible for multiple users to submit work to DUCC in this configuration, but it requires that the user ID running DUCC has write access to all directories to which the user process outputs data. By configuring the ducc user ID and ducc_ling correctly, work submitted by all users will run with their own credentials.
Before proceeding with this step, please note:
ducc_ling is designed to be a setuid-root program whose function is to run user processes with the identity of the submitting user. This must be installed correctly; incorrect installation can prevent jobs from running as their submitters, and in the worse case, can introduce security problems into the system.
ducc_ling can either be installed on a local disk on every system in the DUCC cluster, or on a shared-filesystem that does not suppress setuid-root permissions on client nodes. The path to ducc_ling must be the same on each DUCC node. The default path configuration is $DUCC_HOME/admin/${os.arch}/ in order to handle clusters with mixed OS platforms. ${os.arch} is the architecture specific value of the Java system property with that name; examples are amd64 and ppc64.
The steps are: build ducc_ling for each node architecture to be added to the cluster, copy ducc_ling to the desired location, and then configure ducc_ling to give user ducc the ability to spawn a process as a different user.
In the example below ducc_ling is left under $DUCC_HOME, where it is built.
As user ducc, build ducc_ling for necessary architectures (this is done automatically for the DUCC head machine by the ducc_post_install script). For each unique OS platform:
Then, as user root on the shared filesystem, cd $DUCC_HOME/admin, and for each unique OS architecture:
If these steps are correctly performed, ONLY user ducc may use the ducc_ling program in a privileged way. ducc_ling contains checks to prevent even user root from using it for privileged operations.
If a different location is chosen for ducc_ling the new path needs to be specified for ducc.agent.launcher.ducc_spawn_path in $DUCC_HOME/resources/site.ducc.properties. See more info at see properties merging in the duccbook.
DUCC’s CGroup configuration also allocates CPU resources to managed processes based on relative memory size. A process with 50% of a machine’s RAM will be guaranteed at least 50% of the machine’s CPU resources as well.
The steps in this task must be done as user root and user ducc.
To install and configure CGroups for DUCC:
Note: if CGroups is not installed on a machine the DUCC Agent will detect this and not attempt to use the feature. CGroups can also be disabled for all machines (see ducc.agent.launcher.cgroups.enable in ducc.properties, described in the Duccbook.) or it can be disabled for individual machines (see ducc.agent.exclusion.file in ducc.properties, described in the Duccbook.)
This is identical to initial verification, with the one difference that the job “1.job” should be submitted as any user other than ducc. Watch the webserver and check that the job executes under the correct identity. Once this completes, DUCC is installed and verified.
This step is optional. As shipped, the webserver is disabled for logins. This can be seen by hovering over the Login text located in the upper right of most webserver pages:
To enable logins, a Java-based authenticator must be plugged-in and the login feature must be enabled in the ducc.properties file by the DUCC administrator. Also, ducc_ling should be properly deployed (see Ducc_ling Installation section above).
A beta version of a Linux-based authentication plug-in is shipped with DUCC. It can be found in the source tree:
The Linux-based authentication plug-in will attempt to validate webserver login requests by appealing to the host OS. The user who wishes to login provides a userid and password to the webserver via https, which in-turn are handed-off to the OS for a success/failure reply.
To have the webserver employ the beta Linux-based authentication plug-in, the DUCC administrator should perform the following as user ducc:
Note: The beta Linux-based authentication plug-in has limited testing. In particular, it was tested using:
Alternatively, you can provide your own authentication plug-in. To do so: