Bookie Internals

Bookie server stores its data in multiple ledger directories and its journal files in a journal directory. Ideally, storing journal files in a separate directory than data files would increase throughput and decrease latency

The Bookie Journal

Journal directory has one kind of file in it:

• {timestamp}.txn - holds transactions executed in the bookie server.

Before persisting ledger index and data to disk, a bookie ensures that the transaction that represents the update is written to a journal in non-volatile storage. A new journal file is created using current timestamp when a bookie starts or an old journal file reaches its maximum size.

A bookie supports journal rolling to remove old journal files. In order to remove old journal files safely, bookie server records LastLogMark in Ledger Device, which indicates all updates (including index and data) before LastLogMark has been persisted to the Ledger Device.

LastLogMark contains two parts:

• LastLogId - indicates which journal file the transaction persisted.
• LastLogPos - indicates the position the transaction persisted in LastLogId journal file.

You may use following settings to further fine tune the behavior of journalling on bookies:

journalMaxSizeMB	journal file size limitation. when a journal reaches this limitation, it will be closed and new journal file be created.
journalMaxBackups	how many old journal files whose id is less than LastLogMark 's journal id.

NOTE: keeping number of old journal files would be useful for manually recovery in special case.

ZooKeeper Metadata

For BookKeeper, we require a ZooKeeper installation to store metadata, and to pass the list of ZooKeeper servers as parameter to the constructor of the BookKeeper class (org.apache.bookkeeper.client.BookKeeper). To setup ZooKeeper, please check the ZooKeeper documentation.

BookKeeper provides two mechanisms to organize its metadata in ZooKeeper. By default, the FlatLedgerManager is used, and 99% of users should never need to look at anything else. However, in cases where there are a lot of active ledgers concurrently, (> 50,000), HierarchicalLedgerManager should be used. For so many ledgers, a hierarchical approach is needed due to a limit ZooKeeper places on packet sizes JIRA Issue.

FlatLedgerManager	All ledger metadata are placed as children in a single zookeeper path.
HierarchicalLedgerManager	All ledger metadata are partitioned into 2-level znodes.

Flat Ledger Manager

All ledgers' metadata are put in a single zookeeper path, created using zookeeper sequential node, which can ensure uniqueness of ledger id. Each ledger node is prefixed with 'L'.

Bookie server manages its owned active ledgers in a hash map. So it is easy for bookie server to find what ledgers are deleted from zookeeper and garbage collect them. And its garbage collection flow is described as below:

• Fetch all existing ledgers from zookeeper (zkActiveLedgers).
• Fetch all ledgers currently active within the Bookie (bkActiveLedgers).
• Loop over bkActiveLedgers to find those ledgers which do not exist in zkActiveLedgers and garbage collect them.

Hierarchical Ledger Manager

HierarchicalLedgerManager first obtains a global unique id from ZooKeeper using a EPHEMERAL_SEQUENTIAL znode.

Since ZooKeeper sequential counter has a format of %10d -- that is 10 digits with 0 (zero) padding, i.e. "<path>0000000001", HierarchicalLedgerManager splits the generated id into 3 parts :

{level1 (2 digits)}{level2 (4 digits)}{level3 (4 digits)}

These 3 parts are used to form the actual ledger node path used to store ledger metadata:

{ledgers_root_path}/{level1}/{level2}/L{level3}

E.g. Ledger 0000000001 is split into 3 parts 00, 0000, 00001, which is stored in znode /{ledgers_root_path}/00/0000/L0001. So each znode could have at most 10000 ledgers, which avoids the problem of the child list being larger than the maximum ZooKeeper packet size.

Bookie server manages its active ledgers in a sorted map, which simplifies access to active ledgers in a particular (level1, level2) partition.

Garbage collection in bookie server is processed node by node as follows:

• Fetching all level1 nodes, by calling zk#getChildren(ledgerRootPath).
  • For each level1 nodes, fetching their level2 nodes :
  • For each partition (level1, level2) :
    • Fetch all existed ledgers from zookeeper belonging to partition (level1, level2) (zkActiveLedgers).
    • Fetch all ledgers currently active in the bookie which belong to partition (level1, level2) (bkActiveLedgers).
    • Loop over bkActiveLedgers to find those ledgers which do not exist in zkActiveLedgers, and garbage collect them.

NOTE: Hierarchical Ledger Manager is more suitable to manage large number of ledgers existed in BookKeeper.