Apache Slider AppPackage¶
Slider AppPackages are a declarative definition of an application for application management. AppPackage is not a packaging scheme for application binaries and artifacts. Tarball, zip files, rpms etc. are available for that purpose. Instead AppPackage includes the application binaries along with other artifacts necessary for application management.
An application instance consists of several active component such as one or more masters and several slaves. There may be a number of accompanying processes in addition to the basic master/slave processes - lets refer to all processes as app component instances. When run in the context of Yarn, the application specific processes are activated within individual Yarn Container. If you pry into an Yarn container (created through Slider) it will be apparent as to what is the role of Slider-Agent and the actual application components. The following image provides an high-level view. Within a container there are at least two processes - and instance of a slider agent and an instance of an application component. The application can itself spawn more procsses if needed.
Figure 1 - High-level view of a container
For example:
yarn 8849 -- python ./infra/agent/slider-agent/agent/main.py --label container_1397675825552_0011_01_000003___HBASE_REGIONSERVER --host AM_HOST --port 47830 yarn 9085 -- bash /hadoop/yarn/local/usercache/yarn/appcache/application_1397675825552_0011/ ... internal_start regionserver yarn 9114 -- /usr/jdk64/jdk1.7.0_45/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=...
The above list shows three processes, the Slider-Agent process, the bash script to start HBase Region Server and the HBase Region server itself. Three of these together constitute the container.
Using an AppPackage¶
The following command creates an HBase application using the AppPackage for HBase.
./slider create cl1 --template /work/appConf.json --resources /work/resources.json
Lets analyze various parameters from the perspective of app creation:
--template: app configuration--resources: yarn resource requests- … other parameters are described in accompanying docs.
AppPackage¶
The structure of an AppPackage is described at AppPackage.
In the enlistment, there are several example AppPackages:
app-packages/memcachedapp-packages/memcached-winapp-packages/hbase...
The above folders, with minor edits, can be packaged as zip files to get the corresponding AppPackages. The application tarball file, containing the binaries/artifacts of the application itself is a component within the AppPackage. Refer to the README file in the individual folders on how to package an application.
For example, for Accumulo, package creation is automated in maven
mvn clean package -DskipTests -Paccumulo-app-package
Additional configuration parameters are described in app-packages/accumulo/README.txt.
Looking at the content through unzip -l "$@"
Archive: hbase-v098.zip Length Date Time Name --------- ---------- ----- ---- 3158 06-19-2014 22:08 appConfig.json 0 06-19-2014 22:15 configuration/ 5077 06-19-2014 22:15 configuration/global.xml 5248 06-19-2014 22:15 configuration/hbase-log4j.xml 2250 06-19-2014 22:15 configuration/hbase-policy.xml 14705 06-19-2014 22:15 configuration/hbase-site.xml 2142 06-19-2014 22:17 jmx_metrics.json 3991 06-19-2014 22:45 metainfo.xml 0 06-19-2014 22:33 package/ 0 06-19-2014 22:31 package/files/ 83278591 06-19-2014 22:31 package/files/hbase-0.98.3-hadoop2-bin.tar.gz 0 06-19-2014 22:29 package/scripts/ 787 06-19-2014 22:27 package/scripts/__init__.py 1378 06-19-2014 22:27 package/scripts/functions.py 3738 06-19-2014 22:27 package/scripts/hbase.py 1205 06-19-2014 22:27 package/scripts/hbase_client.py 1640 06-19-2014 22:27 package/scripts/hbase_master.py 1764 06-19-2014 22:27 package/scripts/hbase_regionserver.py 1482 06-19-2014 22:27 package/scripts/hbase_service.py 5089 06-19-2014 22:27 package/scripts/params.py 973 06-19-2014 22:27 package/scripts/status_params.py 0 06-19-2014 22:34 package/templates/ 2799 06-19-2014 22:34 package/templates/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2 2799 06-19-2014 22:34 package/templates/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2 3878 06-19-2014 22:34 package/templates/hbase-env.sh.j2 909 06-19-2014 22:34 package/templates/hbase_client_jaas.conf.j2 989 06-19-2014 22:34 package/templates/hbase_master_jaas.conf.j2 1001 06-19-2014 22:34 package/templates/hbase_regionserver_jaas.conf.j2 837 06-19-2014 22:34 package/templates/regionservers.j2 3486 06-19-2014 22:39 pom.xml 1921 06-19-2014 21:53 README.txt 358 06-19-2014 22:36 resources.json 0 06-19-2014 22:40 src/ 0 06-19-2014 23:02 src/assembly/ 2400 06-19-2014 22:45 src/assembly/hbase.xml --------- ------- 83354595 35 files
Sample resources-default.json and appConfig-default.json files are also included in the enlistment. These are samples and are typically tested on one node test installations. These files are not used during the create commmand, rather the files provided as input parameter are the ones that are used. So you can leave these files as is in the package.
--template appConfig.json¶
An appConfig.json contains the application configuration. See Specifications InstanceConfiguration for details on how to create a template config file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
--resources resources.json¶
Resource specification is an input to Slider to specify the Yarn resource needs for each component type that belong to the application. Specification of Resources describes how to write a resource config json file. The enlistment includes sample config files for HBase, Accumulo, and Storm.
Scripting for AppPackage¶
Refer to App Command Scripts for details on how to write scripts for a AppPackage. These scripts are in the package/script folder within the AppPackage. Use the checked in samples for HBase/Storm/Accumulo as reference for script development.
