Using HDFS with SINGA

Quick start using Docker

We provide a Docker container built on top of singa/mesos (see the guide on building SINGA on Docker).

git clone https://github.com/ug93tad/incubator-singa
cd incubator-singa
git checkout SINGA-97-docker
cd tool/docker/hdfs
sudo docker build -t singa/hdfs .

Once built, the container image singa/hdfs contains the installation of HDFS C++ library (libhdfs3) and the latest SINGA code. Many distributed nodes can be launched, and HDFS be set up, by following the guide for running distributed SINGA on Mesos.

In the following, we assume the HDFS setup with node0 being the namenode, and nodei (i>0) being the datanodes.

Setup HDFS

There are at least 2 C/C++ client libraries for interacting with HDFS. One is from Hadoop (libhdfs), which is a JNI-based library, meaning that communication will go through JVM. The other is libhdfs3 which is a native C++ library developed by Pivotal, in which the client communicate directly with HDFS via RPC. The current implementation uses the second one.

1. Install libhdfs3: follow the official guide.

2. Additional setup: recent versions of Hadoop (>2.4.x) support short-circuit local reads which bypass network communications (TCP sockets) when retrieving data at the local nodes. libhdfs3 will throws errors (but will still continue to work) when it finds that short-circuit read is not set. To deal with this complaints, and improve performance, add the following configuration to hdfs-site.xml and to hdfs-client.xml

     <property>
   <name>dfs.client.read.shortcircuit</name>
   <value>true</value>
     </property>
     <property>
   <name>dfs.domain.socket.path</name>
   <value>/var/lib/hadoop-hdfs/dn_socket</value>
     </property>
   

   Next, at each client, set LIBHDFS3_CONF variable to point to hdfs-client.xml file:

     export LIBHDFS3_CONF=$HADOOP_HOME/etc/hadoop/hdfs-client.xml
   

Examples

We explain how to run CIFAR10 and MNIST examples. Before training, the data must be uploaded to HDFS.