This module provides a BookKeeper backend for HFDS Namenode write ahead logging. BookKeeper is a highly available distributed write ahead logging system. For more details, see http://zookeeper.apache.org/bookkeeper ------------------------------------------------------------------------------- How do I build? To generate the distribution packages for BK journal, do the following. $ mvn clean package -Pdist This will generate a jar with all the dependencies needed by the journal manager, target/hadoop-hdfs-bkjournal-.jar Note that the -Pdist part of the build command is important, as otherwise the dependencies would not be packaged in the jar. ------------------------------------------------------------------------------- How do I use the BookKeeper Journal? To run a HDFS namenode using BookKeeper as a backend, copy the bkjournal jar, generated above, into the lib directory of hdfs. In the standard distribution of HDFS, this is at $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/ cp target/hadoop-hdfs-bkjournal-.jar \ $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/ Then, in hdfs-site.xml, set the following properties. dfs.namenode.edits.dir bookkeeper://localhost:2181/bkjournal,file:///path/for/edits dfs.namenode.edits.journal-plugin.bookkeeper org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager In this example, the namenode is configured to use 2 write ahead logging devices. One writes to BookKeeper and the other to a local file system. At the moment is is not possible to only write to BookKeeper, as the resource checker explicitly checked for local disks currently. The given example, configures the namenode to look for the journal metadata at the path /bkjournal on the a standalone zookeeper ensemble at localhost:2181. To configure a multiple host zookeeper ensemble, separate the hosts with semicolons. For example, if you have 3 zookeeper servers, zk1, zk2 & zk3, each listening on port 2181, you would specify this with bookkeeper://zk1:2181;zk2:2181;zk3:2181/bkjournal The final part /bkjournal specifies the znode in zookeeper where ledger metadata will be store. Administrators can set this to anything they wish.