Apache Slider App Package¶

All application artifacts, app definition, app configuration, scripts are packaged into a structured single package that can be handed off to any YARN application deployment tool including Slider

Overall Structure¶

App package is a zip package containing all application artifacts. App package contains the following items:

app definition file application structure, content, definition, supported platforms, version, etc.
default configurations folder various configurations and configuration files associated with the application
cmd_impl folder management operations for the application/component
scripts folder various scripts that implement management operations
templates folder various templates used by the application
files folder other scripts, txt files, tarballs, etc.

The example above shows a semi-expanded view of an application "HBASE-YARN-APP" and the package structure for OOZIE command scripts.

app definition¶

App definition is a file named "metainfo.xml". The file contains application definition as described in Application Definition.

default configurations¶

This folder consists of various config files containing default configuration as described in App Configuration.

package folder¶

package includes the "implementation" of all management operations. The folders within are divided into scripts, templates, and files.

scripts folder¶

Scripts are the implementation of management operations. There are five default operations and a composite operation. "restart" can be redefined to have a custom implementation.

install
configure
start
stop
status
restart (by default calls stop + start)

The script specified in the metainfo is expected to understand the command. It can choose to call other scripts based on how the application author organizes the code base. For example:

class HbaseMaster(Script):
  def install(self, env):
    self.install_packages(env)

  def configure(self, env):
    import params
    env.set_params(params)

    hbase(name='master')

  def start(self, env):
    import params
    env.set_params(params)
    self.configure(env) # for security

    hbase_service( 'master',
      action = 'start'
    )
...

How to write scripts discusses how the scripts are developed and the structure of the JSON_FILE containing the parameters.

templates folder¶

templates are configurable text files that are NOT regular config files. A library has been developed that can materialize a complete site configuration file from a property bag and therefore are not generated from templates. Other files such as env sh files, log4j properties file, etc. may be derived from a template. Again, the implementor can choose to create these files from scratch and not use templates. The following operations are allowed during template expansion:

variable expansion
if condition
for loop
...

Sample template file for dfs.exclude file to list excluded/decommissioned hosts. hdfs_exclude_files in the property defined in params.py which is populated from config parameters defined in JSON_FILE.

{% if hdfs_exclude_file %} 
{% for host in hdfs_exclude_file %}
{{host}}
{% endfor %}
{% endif %}

files folder¶

files is a directory to store any other files that are needed for management operations. Sample files stored here are tarballs used to install the application, shell scripts used by various operations.

Apache Slider (incubating)