Apache Slider Examples¶
Note: These examples use HBase providers written for Slider.
To use the HBase application package refer to Getting Started.
Setup¶
Setting up a YARN cluster¶
For simple local demos, a pseudo-distributed Apache Hadoop cluster will suffice -if on a VM then its configuration should be changed to use a public (machine public) IP.
The examples below all assume there is a cluster node called 'master', which¶
hosts the HDFS NameNode and the YARN Resource Manager
preamble¶
export HADOOP_CONF_DIR=~/conf export PATH=~/hadoop/bin:/~/hadoop/sbin:~/zookeeper-3.4.5/bin:$PATH hdfs namenode -format master
start all the services¶
nohup hdfs --config $HADOOP_CONF_DIR namenode & nohup hdfs --config $HADOOP_CONF_DIR datanode & nohup yarn --config $HADOOP_CONF_DIR resourcemanager & nohup yarn --config $HADOOP_CONF_DIR nodemanager &
using hadoop/sbin service launchers¶
hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager ~/zookeeper/bin/zkServer.sh start
stop them¶
hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager
NN up on http://master:50070/dfshealth.jsp RM yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
~/zookeeper/bin/zkServer.sh start # shutdown ~/zookeeper/bin/zkServer.sh stop
Tip: after a successful run on a local cluster, do a quick rm -rf $HADOOP_HOME/logs
to keep the log bloat under control.
get hbase in¶
copy to local
get hbase-0.98.0-bin.tar on hdfs dfs -rm hdfs://master:9090/hbase.tar hdfs dfs -copyFromLocal hbase-0.98.0-bin.tar hdfs://master:9090/hbase.tar
or
hdfs dfs -copyFromLocal hbase-0.96.0-bin.tar hdfs://master:9090/hbase.tar hdfs dfs -ls hdfs://master:9090/
Optional: point bin/slider at your chosen cluster configuration¶
export SLIDER_CONF_DIR=~/Projects/slider/slider-core/src/test/configs/ubuntu-secure/slider
Optional: Clean up any existing slider cluster details¶
This is for demos only, otherwise you lose the clusters and their databases.
hdfs dfs -rm -r hdfs://master:9090/user/home/stevel/.slider
Create a Slider Cluster¶
slider create cl1 \ --component worker 1 --component master 1 \ --manager master:8032 --filesystem hdfs://master:9090 \ --zkhosts localhost:2181 --image hdfs://master:9090/hbase.tar # create the cluster // TODO # stop a cluster slider stop cl1 \ --manager master:8032 --filesystem hdfs://master:9090 # start a cluster slider start cl1 \ --manager master:8032 --filesystem hdfs://master:9090 # destroy the cluster slider destroy cl1 \ --manager master:8032 --filesystem hdfs://master:9090 # list clusters slider list cl1 \ --manager master:8032 --filesystem hdfs://master:9090 slider flex cl1 --component worker 2 --manager master:8032 --filesystem hdfs://master:9090 \ --component worker 5
