Release Notes - Bookkeeper - Version 4.2.0

Sub-task

• [BOOKKEEPER-2] - bookkeeper does not put enough meta-data in to do recovery properly
• [BOOKKEEPER-203] - improve ledger manager interface to remove zookeeper dependency on metadata operations.
• [BOOKKEEPER-204] - Provide a MetaStore interface, and a mock implementation.
• [BOOKKEEPER-205] - implement a MetaStore based ledger manager for bookkeeper client.
• [BOOKKEEPER-246] - Recording of underreplication of ledger entries
• [BOOKKEEPER-247] - Detection of under replication
• [BOOKKEEPER-248] - Rereplicating of under replicated data
• [BOOKKEEPER-250] - Need a ledger manager like interface to manage metadata operations in Hedwig
• [BOOKKEEPER-259] - Create a topic manager using versioned write for leader election
• [BOOKKEEPER-262] - Implement a meta store based hedwig metadata manager.
• [BOOKKEEPER-272] - Provide automatic mechanism to know bookie failures
• [BOOKKEEPER-278] - Ability to disable auto recovery temporarily
• [BOOKKEEPER-283] - Improve Hedwig Console to use Hedwig Metadata Manager.
• [BOOKKEEPER-293] - Periodic checking of ledger replication status
• [BOOKKEEPER-299] - Provide LedgerFragmentReplicator which should replicate the fragments found from LedgerChecker
• [BOOKKEEPER-304] - Prepare bookie vs ledgers cache and will be used by the Auditor
• [BOOKKEEPER-310] - Changes in hedwig server to support JMS spec
• [BOOKKEEPER-315] - Ledger entries should be replicated sequentially instead of parallel.
• [BOOKKEEPER-317] - Exceptions for replication
• [BOOKKEEPER-319] - Manage auditing and replication processes
• [BOOKKEEPER-325] - Delay the replication of a ledger if RW found that its last fragment is in underReplication.
• [BOOKKEEPER-331] - Let hedwig support returning message seq id for publish requests.
• [BOOKKEEPER-332] - Add SubscriptionPreferences to record all preferences for a subscription
• [BOOKKEEPER-333] - server-side message filter
• [BOOKKEEPER-334] - client-side message filter for java client.
• [BOOKKEEPER-335] - client-side message filter for cpp client.
• [BOOKKEEPER-341] - add documentation for bookkeeper ledger manager interface.
• [BOOKKEEPER-342] - add documentation for hedwig metadata manager interface.
• [BOOKKEEPER-345] - Detect IOExceptions on entrylogger and bookie should consider next ledger dir(if any)
• [BOOKKEEPER-346] - Detect IOExceptions in LedgerCache and bookie should look at next ledger dir(if any)
• [BOOKKEEPER-347] - Provide mechanism to detect r-o bookie by the bookie clients
• [BOOKKEEPER-364] - re-factor hedwig java client to support both one-subscription-per-channel and multiplex-subscriptions-per-channel.
• [BOOKKEEPER-367] - Server-Side Message Delivery Flow Control
• [BOOKKEEPER-368] - Implementing multiplexing java client.
• [BOOKKEEPER-369] - re-factor hedwig cpp client to provide better interface to support both one-subscription-per-channel and multiple-subscriptions-per-channel.
• [BOOKKEEPER-370] - implement multiplexing cpp client.
• [BOOKKEEPER-375] - Document about Auto replication service in BK
• [BOOKKEEPER-376] - LedgerManagers should consider 'underreplication' node as a special Znode
• [BOOKKEEPER-378] - ReplicationWorker may not get ZK watcher notification on UnderReplication ledger lock deletion.
• [BOOKKEEPER-380] - ZkLedgerUnderreplicationManager.markLedgerUnderreplicated() is adding duplicate missingReplicas while multiple bk failed for the same ledger
• [BOOKKEEPER-384] - Clean up LedgerManagerFactory and LedgerManager usage in tests
• [BOOKKEEPER-385] - replicateLedgerFragment should throw Exceptions in error conditions
• [BOOKKEEPER-386] - It should not be possible to replicate a ledger fragment which is at the end of an open ledger
• [BOOKKEEPER-389] - add documentation for message filter.
• [BOOKKEEPER-405] - Let's add Thread name for ReplicationWorker thread.
• [BOOKKEEPER-409] - Integration Test - Perform bookie rereplication cycle by Auditor-RW processes
• [BOOKKEEPER-411] - Add CloseSubscription Request for multiplexing support
• [BOOKKEEPER-416] - LedgerChecker returns underreplicated fragments for an closed ledger with no entries
• [BOOKKEEPER-417] - Hierarchical zk underreplication manager should clean up its hierarchy when done to allow for fast acquisition of underreplicated entries
• [BOOKKEEPER-418] - Store hostname of locker in replication lock
• [BOOKKEEPER-426] - Make auditor Vote znode store a protobuf containing the host that voted
• [BOOKKEEPER-428] - Expose command options in bookie scripts to disable/enable auto recovery temporarily
• [BOOKKEEPER-435] - Create SubscriptionChannelManager to manage all subscription channel
• [BOOKKEEPER-440] - Make Write/Delete SubscriptionData Restricted to Version
• [BOOKKEEPER-444] - Refactor pending read op to make speculative reads possible
• [BOOKKEEPER-452] - Rename ClientConfiguration multiplexing_enabled to subscription_connection_sharing_enabled
• [BOOKKEEPER-453] - Extract commonality from MultiplexSubscribeResponseHandler and SimpleSubscribeResponseHandler and put into an abstract class
• [BOOKKEEPER-459] - Rename metastore mock implementation to InMemory implementation
• [BOOKKEEPER-463] - Refactor garbage collection code for ease to plugin different GC algorithm.
• [BOOKKEEPER-472] - Provide an option to start Autorecovery along with Bookie Servers
• [BOOKKEEPER-490] - add documentation for MetaStore interface
• [BOOKKEEPER-496] - Ensure that the auditor and replication worker will shutdown if they lose their ZK session

Bug

• [BOOKKEEPER-55] - SubscribeReconnectRetryTask might retry subscription endlessly when another subscription is already successfully created previously
• [BOOKKEEPER-274] - Hedwig cpp client library should not link to cppunit which is just used for test.
• [BOOKKEEPER-280] - LedgerHandle.addEntry() should return an entryId
• [BOOKKEEPER-291] - BKMBeanRegistry uses log4j directly
• [BOOKKEEPER-292] - Test backward compatibility automatically between versions.
• [BOOKKEEPER-294] - Not able to start the bookkeeper before the ZK session timeout.
• [BOOKKEEPER-296] - It's better provide stop script for bookie
• [BOOKKEEPER-298] - We run with preferIPv4Stack in the scripts but not in the tests
• [BOOKKEEPER-302] - No more messages delivered when hub server scans messages over two ledgers.
• [BOOKKEEPER-303] - LedgerMetadata should serialized using protobufs
• [BOOKKEEPER-306] - Change C++ client to use gtest for testing
• [BOOKKEEPER-307] - BookieShell introduces 4 findbugs warnings
• [BOOKKEEPER-322] - New protobufs generates findbugs errors
• [BOOKKEEPER-324] - Flakeyness in LedgerCreateDeleteTest
• [BOOKKEEPER-326] - DeadLock during ledger recovery
• [BOOKKEEPER-327] - System.currentTimeMillis usage in BookKeeper
• [BOOKKEEPER-329] - provide stop scripts for hub server
• [BOOKKEEPER-330] - System.currentTimeMillis usage in Hedwig
• [BOOKKEEPER-336] - bookie readEntries is taking more time if the ensemble has failed bookie(s)
• [BOOKKEEPER-337] - Add entry fails with MetadataVersionException when last ensemble has morethan one bookie failures
• [BOOKKEEPER-338] - Create Version.NEW and Version.ANY static instances of Version so that were not passing around nulls
• [BOOKKEEPER-343] - Failed to register hedwig JMX beans in test cases
• [BOOKKEEPER-349] - Entry logger should close all the chennels which are there in Map, instead of closing only current channel.
• [BOOKKEEPER-351] - asyncAddEntry should not throw an exception
• [BOOKKEEPER-352] - Should not use static ServerStats/BKStats instance in TestServerStats/TestBKStats
• [BOOKKEEPER-354] - [BOOKKEEPER-296] [Documentation] Modify the bookkeeper start script and document the bookkeeper stop command in bookkeeperConfig.xml
• [BOOKKEEPER-355] - Ledger recovery will mark ledger as closed with -1, in case of slow bookie is added to ensemble during recovery add
• [BOOKKEEPER-365] - Ledger will never recover if one of the quorum bookie is down forever and others dont have entry
• [BOOKKEEPER-371] - NPE in hedwig hub client causes hedwig hub to shut down.
• [BOOKKEEPER-372] - Check service name in bookie start/stop script.
• [BOOKKEEPER-381] - ReadLastConfirmedOp's Logger class name is wrong
• [BOOKKEEPER-382] - space missed at concatenations in GarbageCollectorThread logging
• [BOOKKEEPER-383] - NPE in BookieJournalTest
• [BOOKKEEPER-387] - BookKeeper Upgrade is not working.
• [BOOKKEEPER-392] - Racey ConcurrentMap usage in java hedwig-client
• [BOOKKEEPER-395] - HDFS dep transitively depends on a busted pom
• [BOOKKEEPER-396] - Compilation issue in TestClient.java of BenchMark ( showing this in eclipse)
• [BOOKKEEPER-397] - Make the hedwig client in RegionManager configurable.
• [BOOKKEEPER-403] - ReReadMetadataCb is not executed in the thread responsible for that ledger
• [BOOKKEEPER-404] - Deprecate non-SubscriptionOptions Subscriber Apis
• [BOOKKEEPER-408] - BookieReadWriteTest will enter the endless loop and will not leave out
• [BOOKKEEPER-413] - Hedwig C++ client: Rename RUN_AS_SSL_MODE to SSL_ENABLED
• [BOOKKEEPER-415] - Rename DeliveryThrottle to MessageWindowSize
• [BOOKKEEPER-424] - Bookie start is failing intermittently when zkclient connection delays
• [BOOKKEEPER-425] - Cleanup Bookie id generation
• [BOOKKEEPER-427] - TestConcurrentTopicAcquisition hangs every so often
• [BOOKKEEPER-434] - [Hedwig CPP Client] Delay resolving default host until necessary.
• [BOOKKEEPER-436] - Journal#rollLog may leak file handler
• [BOOKKEEPER-439] - No more messages delivered after deleted consumed ledgers.
• [BOOKKEEPER-441] - InMemorySubscriptionManager should back up top2sub2seq before change it
• [BOOKKEEPER-442] - Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
• [BOOKKEEPER-447] - Bookie can fail to recover if index pages flushed before ledger flush acknowledged
• [BOOKKEEPER-454] - hedwig c++ tester script assumes sh is bash
• [BOOKKEEPER-457] - Create a format command for Hedwig to cleanup its metadata.
• [BOOKKEEPER-458] - Annoy BKReadException error when changing ledger.
• [BOOKKEEPER-461] - Delivery throughput degrades when there are lots of publishers w/ high traffic.
• [BOOKKEEPER-465] - CreateNewLog may overwrite lastLogId with smaller value
• [BOOKKEEPER-466] - ZooKeeper test utility sets the port number as the tickTime
• [BOOKKEEPER-467] - Allocate ports for testing dynamically
• [BOOKKEEPER-468] - Remove <echo> from protobuf generation in hedwig
• [BOOKKEEPER-469] - Remove System.out.println from TestLedgerManager
• [BOOKKEEPER-470] - Possible infinite loop in simple.SubscribeReconnectCallback
• [BOOKKEEPER-471] - Add scripts for preCommit testing
• [BOOKKEEPER-474] - BookieReadWriteTest#testShutdown doesn't make sense
• [BOOKKEEPER-475] - BookieRecoveryTest#testSyncBookieRecoveryToRandomBookiesCheckForDupes() iterates too much
• [BOOKKEEPER-477] - In ReadOnlyBookieTest, we should wait for the bookie to die before asserting on it
• [BOOKKEEPER-479] - Fix apache-rat issues in tree
• [BOOKKEEPER-480] - Fix javac warnings
• [BOOKKEEPER-481] - Fix javadoc warnings
• [BOOKKEEPER-482] - Precommit is reporting findbugs errors in trunk
• [BOOKKEEPER-483] - precommit tests only check toplevel rat file, not the one for submodules.
• [BOOKKEEPER-484] - Misc fixes for test scripts
• [BOOKKEEPER-485] - TestFencing hung
• [BOOKKEEPER-493] - moveLedgerIndexFile might have chance pickup same directory
• [BOOKKEEPER-497] - GcLedgersTest has a potential race
• [BOOKKEEPER-498] - BookieRecoveryTest.tearDown NPE
• [BOOKKEEPER-500] - Fencing doesn't work when restarting bookies.
• [BOOKKEEPER-503] - The test case of TestThrottlingDelivery#testServerSideThrottle failed sometimes
• [BOOKKEEPER-504] - Fix findbugs warning in PendingReadOp
• [BOOKKEEPER-507] - Race condition happens if closeSubscription and subscribe happened at the same time (in multiplexed client).
• [BOOKKEEPER-509] - TestBookKeeperPersistenceManager failed on latest trunk
• [BOOKKEEPER-511] - BookieShell is very noisy
• [BOOKKEEPER-512] - BookieZkExpireTest fails periodically
• [BOOKKEEPER-513] - TestMessageFilter fails periodically
• [BOOKKEEPER-514] - TestDeadLock hanging sometimes
• [BOOKKEEPER-520] - BookieFailureTest hangs on precommit build
• [BOOKKEEPER-522] - TestHedwigHub is failing silently on Jenkins
• [BOOKKEEPER-524] - Bookie journal filesystem gets full after SyncThread is terminated with exception
• [BOOKKEEPER-530] - data might be lost during compaction.
• [BOOKKEEPER-531] - Cache thread should wait until old entries are collected
• [BOOKKEEPER-532] - AbstractSubscriptionManager#AcquireOp read subscriptions every time even it already owned the topic
• [BOOKKEEPER-533] - TestSubAfterCloseSub fails strangely in tests
• [BOOKKEEPER-534] - Flakeyness in AuditorBookieTest
• [BOOKKEEPER-538] - Race condition in BookKeeper#close
• [BOOKKEEPER-539] - ClientNotSubscribedException & doesn't receive enough messages in TestThrottlingDelivery#testServerSideThrottle
• [BOOKKEEPER-543] - Read zk host list in a wrong way in hedwig server

Improvement

• [BOOKKEEPER-32] - Clean up LOG.debug statements
• [BOOKKEEPER-143] - Add SSL support for hedwig cpp client
• [BOOKKEEPER-191] - Hub server should change ledger to write, so consumed messages have chance to be garbage collected.
• [BOOKKEEPER-289] - mvn clean doesn't remove test output files
• [BOOKKEEPER-318] - Spelling mistake in MultiCallback log message.
• [BOOKKEEPER-320] - Let hedwig cpp client could publish messages using Message object instead of string.
• [BOOKKEEPER-328] - Bookie DeathWatcher is missing thread name
• [BOOKKEEPER-339] - Let hedwig cpp client support returning message seq id for publish requests.
• [BOOKKEEPER-340] - Test backward compatibility for hedwig between different versions.
• [BOOKKEEPER-388] - Document bookie format command
• [BOOKKEEPER-394] - CompositeException message is not useful
• [BOOKKEEPER-422] - Simplify AbstractSubscriptionManager
• [BOOKKEEPER-430] - Remove manual bookie registration from overview
• [BOOKKEEPER-431] - Duplicate definition of COOKIES_NODE
• [BOOKKEEPER-476] - Log to file during tests
• [BOOKKEEPER-491] - Hedwig doc for configuration
• [BOOKKEEPER-495] - Revise BK config doc
• [BOOKKEEPER-523] - Every test should have a timeout
• [BOOKKEEPER-529] - stopServingSubscriber in delivery manager should remove stub callbacks in ReadAheadCache
• [BOOKKEEPER-540] - #stopServingSubscriber when channel is disconnected.
• [BOOKKEEPER-541] - Add guava to notice file
• [BOOKKEEPER-542] - Remove trailing spaces in IndexCorruptionTest

New Feature

• [BOOKKEEPER-183] - Provide tools to read/check data files in bookie server
• [BOOKKEEPER-208] - Separate write quorum from ack quorum
• [BOOKKEEPER-252] - Hedwig: provide a subscription mode to kill other subscription channel when hedwig client is used as a proxy-style server.
• [BOOKKEEPER-300] - Create Bookie format command
• [BOOKKEEPER-399] - Let hub server configure write quorum and ack quorum.

Task

• [BOOKKEEPER-487] - Add existed hub server settings to configuration template file

Test

• [BOOKKEEPER-460] - LedgerDeleteTest checks wrong place for log file