Simple Slider Demo¶

These instructions cover how to give a simple demo of slider, showing creating an application instance starting and stopping an instance publishing information in the registry flexing an instance recovery from component failure recovery from AM failure

It does not cover Configuring the application instance via app_conf.json Functional use of the deployed application other than viewing its web pages * Security

Preparation¶

Before starting the demo you need

An Apache Hadoop Hadoop 2.6+ cluster, one node or greater
A local installation of the slider client (unzipped into a directory is sufficient)
A slider-client.xml file containing the binding information for the Hadoop cluster

Hadoop cluster setup¶

The cluster must be set up with no checks on memory use, and no minimum size on container allocations. This ensures that all requested VMs will be created, irrespective of how much memory there is available on the VM.

  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>64</value>
  </property>

  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
  </property>

  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>

Local environment and path setup¶

Bash¶

 export HADOOP_CONF_DIR=~/clusters/demo/hadoop-conf

 export SLIDER_CONF_DIR=~/clusters/demo/slider;

 export PATH=~/slider/slider-assembly/target/slider-0.80.0-incubating-all/slider-0.80.0-incubating/bin

Fish¶

 set -gx HADOOP_CONF_DIR ~/clusters/demo/hadoop-conf

 set -gx  SLIDER_CONF_DIR ~/clusters/demo/slider;

 set -gx SLIDER_VERSION slider-0.80.0-incubating
 set -gx PROJECTS /Users/stevel/Projects/Hortonworks/Projects

 set -gx  PATH $PATH $PROJECTS/slider/slider-assembly/target/$SLIDER_VERSION-all/$SLIDER_VERSION/bin

 set -gx SLIDER_CONF_DIR $PROJECTS/clusterconfigs/clusters/structor-1/slider
 set -gx HADOOP_CONF_DIR $PROJECTS/clusterconfigs/clusters/structor-1/hadoop-conf

Tests¶

slider version
slider list

Creating the HDFS paths¶

HDFS needs to have the user's path created, with the username of the user giving the demo, here demo. This username MUST be that of the local user.

hdfs dfs -chmod 777 /
hdfs dfs -mkdir -p /user/demo
hdfs dfs -chown demo /user/demo
hdfs dfs -chmod 777 /user

The YARN user account needs to be present too (in a reformatted VM this may be lost)

hdfs dfs -mkdir -p /user/yarn
hdfs dfs -chown yarn /user/yarn

Building and installing the application¶

This demo uses HBase, so must have the slider hbase application package installed in the cluster.

Download the hbase bin package, such as hbase-0.99.1-bin.tar.gz
rename it hbase-0.99.1.tar.gz, so that the name of the file matches the path to hbase inside it
Untar it somewhere too (e.g. /Users/demo/hbase-0.99.1)

Build the HBase application package

mvn clean package -Phbase-app-package -Dpkg.version=0.99.1 \
  -Dpkg.name=hbase-0.99.1.tar.gz  \
  -Dpkg.src=/Users/demo/hbase-0.99.1

Upload it

slider install-package --name HBASE --package target/slider-hbase-app-package-0.99.1.zip

Preparing the resource definitions¶

Take the template app's resources-default.json and appConf-default.json FROM the target/ subdirectory. They must be the ones with package versions fully expanded.

In appConfig.json

cut the heap size down to "256m" in all components
make sure any needed path to the JDK base directory is valid.

In resources.json

set "yarn.memory": "128" everywhere,
Turn set the number of instances the THRIFT and REST components to 0.

slider

slider resolve --path / --list

slider create hbase1 --resources resources.json --template appConfig.json

slider status hbase1 slider stop hbase1 slider start hbase1

slider flex hbase1 --component HBASE_REGIONSERVER 2

slider destroy hbase1

failures

kill -9 HBase master kill -9 AM

Apache Slider (incubating)