=============== Version 1.6.0 =============== .. rubric:: Status of this release Apache Flume 1.6.0 is the ninth release of Flume as an Apache top-level project (TLP). Apache Flume 1.6.0 is production-ready software. .. rubric:: Release Documentation * `Flume 1.6.0 User Guide `__ (also in `pdf `__) * `Flume 1.6.0 Developer Guide `__ (also in `pdf `__) * `Flume 1.6.0 API Documentation `__ .. rubric:: Changes Release Notes - Flume - Version v1.6.0 ** Sub-task * [`FLUME-2250 `__] - Add support for Kafka Source * [`FLUME-2251 `__] - Add support for Kafka Sink * [`FLUME-2677 `__] - Update versions in 1.6.0 branch * [`FLUME-2686 `__] - Update KEYS file for 1.6 release ** Bug * [`FLUME-1793 `__] - Unit test TestElasticSearchLogStashEventSerializer fails with IBM JDK * [`FLUME-1934 `__] - Spoolingdir source exception when reading multiple zero size files * [`FLUME-2024 `__] - Add ExecSource flush timeout to Flume User Guide * [`FLUME-2126 `__] - Problem in elasticsearch sink when the event body is a complex field * [`FLUME-2148 `__] - Windows : Add flume-env.ps1 * [`FLUME-2214 `__] - FileChannelIntegrityTool needs documentation in user guide * [`FLUME-2245 `__] - HDFS files with errors unable to close * [`FLUME-2337 `__] - export JAVA_HOME in flume-env.sh.template and increase heap size * [`FLUME-2346 `__] - idLogFileMap in Log can lose track of file ids * [`FLUME-2404 `__] - Default maxReadBufferBytes might cause OOM and cause scribe source exit * [`FLUME-2408 `__] - Remove FLUME-1899 from Flume 1.5.0 change log. * [`FLUME-2416 `__] - Use CodecPool in compressed stream to prevent leak of direct buffers * [`FLUME-2420 `__] - HDFSEventSink#stop might throw ConcurrentModificationException * [`FLUME-2425 `__] - FileChannel should trim data and checkpoint directories * [`FLUME-2432 `__] - Update Kite dependency to 0.15.0 * [`FLUME-2438 `__] - Make Syslog source message body configurable * [`FLUME-2450 `__] - Improve replay index insertion speed. * [`FLUME-2466 `__] - Embedded agent name cannot contain spaces * [`FLUME-2470 `__] - Kafka Sink and Source must use camel case for all configs. * [`FLUME-2472 `__] - SyslogUtils fails when system locale is not English * [`FLUME-2475 `__] - toLowerCase/toUpperCase used without Locale parameter to manipulate Enum * [`FLUME-2479 `__] - Kafka property auto.commit.enable is incorrect for KafkaSource * [`FLUME-2480 `__] - Typo in file channel exception: "Usable space exhaused" * [`FLUME-2481 `__] - TestFileChannelEncryption fails with System locale other than en_US * [`FLUME-2482 `__] - Race condition in File Channels' Log.removeOldLogs * [`FLUME-2486 `__] - TestExecSource fails on some environments * [`FLUME-2487 `__] - SyslogParser has rounding errors in timestamp parsing * [`FLUME-2488 `__] - TestElasticSearchRestClient fails on Oracle JDK 8 * [`FLUME-2489 `__] - Upgrade Apache Curator to 2.6.0 * [`FLUME-2492 `__] - Flume's Kafka Source doesn't account time correctly * [`FLUME-2495 `__] - Kafka Source may miss events when channel is not available * [`FLUME-2497 `__] - TCP and UDP syslog sources parsing the timestamp incorrectly * [`FLUME-2500 `__] - Add a channel that uses Kafka * [`FLUME-2505 `__] - Test added in FLUME-2502 is flaky * [`FLUME-2517 `__] - Performance issue: SimpleDateFormat constructor takes 30% of HDFSEventSink.process() * [`FLUME-2525 `__] - flume should handle a zero byte .flumespool-main.meta file for the spooldir source * [`FLUME-2538 `__] - TestResettableFileInputStream fails on JDK 8 * [`FLUME-2556 `__] - TestBucketPath.testDateFormatHours fails intermittently * [`FLUME-2557 `__] - DatasetSink doesn't work with Kerberos * [`FLUME-2560 `__] - Kafka channel entries missing for output jar generation * [`FLUME-2568 `__] - Additional fix for TestReliableSpoolingFileEventReader * [`FLUME-2578 `__] - Kafka source throws NPE if Kafka record has null key * [`FLUME-2586 `__] - HDFS Sink should have an option to try rename even if close fails * [`FLUME-2592 `__] - Specify main manifest attribute in flume tools jar * [`FLUME-2594 `__] - Close Async HBase Client if there are large number of consecutive timeouts * [`FLUME-2595 `__] - Add option to checkpoint on file channel shutdown * [`FLUME-2624 `__] - Improve Hive Sink performance * [`FLUME-2626 `__] - Remove trustmanager-type from Thrift RPC client * [`FLUME-2633 `__] - Update Kite dependency to 1.0.0 * [`FLUME-2639 `__] - SecureThriftRpcClient client privileged calls throw IllegalAccessError * [`FLUME-2645 `__] - ipFilter.rules property name is wrong * [`FLUME-2654 `__] - Memory channel error message related to byteCapacity needs refining * [`FLUME-2657 `__] - Upgrade to Hive 1.0 * [`FLUME-2658 `__] - Upgrade to Hbase 1.0 * [`FLUME-2664 `__] - Site profile fails due to dependency issues * [`FLUME-2665 `__] - Update documentation for hdfs.closeTries based on FLUME-2586 * [`FLUME-2670 `__] - Modify dependencies to not pull hadoop/hive related jars * [`FLUME-2673 `__] - Remove unused import in TestLog * [`FLUME-2678 `__] - Upgrade xalan to 2.7.2 to take care of CVE-2014-0107 vulnerability * [`FLUME-2679 `__] - Make hbase-1 the default profile ** Dependency upgrade * [`FLUME-2443 `__] - org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 ** Documentation * [`FLUME-1594 `__] - Document JMX options * [`FLUME-2509 `__] - Add description for properties in Failover Sink Processor * [`FLUME-2630 `__] - Update documentation for Thrift Src/Sink SSL support and Kerberos support * [`FLUME-2635 `__] - Documentation for zookeeper dynamic configuration in flume * [`FLUME-2668 `__] - Document SecureThriftRpcClient/SecureRpcClientFactory in Flume Developer Guide ** Improvement * [`FLUME-1334 `__] - Write an startscript for flume agents on Windows * [`FLUME-1491 `__] - Dynamic configuration from Zookeeper watcher * [`FLUME-1521 `__] - Document the StressSource * [`FLUME-1710 `__] - JSONEvent.getBody should not return null * [`FLUME-1920 `__] - Test case TestFileChannel fails when flume is built from paths containing the string "hadoop" * [`FLUME-2237 `__] - Example config in User Guide to use same notation for Agent name * [`FLUME-2246 `__] - event body data size can make it configurable for logger sinker * [`FLUME-2273 `__] - ElasticSearchSink: Add handling for header substitution in indexName * [`FLUME-2373 `__] - Enable configuration to switch thrift source serialization protocol * [`FLUME-2385 `__] - Flume spans log file with "Spooling Directory Source runner has shutdown" messages at INFO level * [`FLUME-2401 `__] - Optionally Compress Backup Checkpoint * [`FLUME-2439 `__] - Update DatasetSink for Kite 0.15.0 * [`FLUME-2462 `__] - Remove use of deprecated methods in DatasetSink * [`FLUME-2463 `__] - Add support for Hive and HBase datasets to DatasetSink * [`FLUME-2469 `__] - DatasetSink should load dataset when needed, not at startup * [`FLUME-2499 `__] - Include Kafka Message Key in Event Header, Updated Comments * [`FLUME-2502 `__] - Spool source's directory listing is inefficient * [`FLUME-2558 `__] - Update javadoc for StressSource * [`FLUME-2562 `__] - Metrics for Flafka components * [`FLUME-2591 `__] - Kite DatasetSink 2.0 * [`FLUME-2613 `__] - Tool/script for deleting individual message from queue * [`FLUME-2642 `__] - Limit kerberos relogin attempt ** New Feature * [`FLUME-1734 `__] - Create a Hive Sink based on the new Hive Streaming support * [`FLUME-2242 `__] - Flume Sink and Source for Apache Kafka * [`FLUME-2426 `__] - Support interceptors in the Embedded Agent * [`FLUME-2431 `__] - Add simple regex search-and-replace interceptor * [`FLUME-2570 `__] - Add option to not pad date fields * [`FLUME-2574 `__] - SSL Support for Thrift Rpc * [`FLUME-2631 `__] - End to End authentication in Flume ** Task * [`FLUME-2365 `__] - Please create a DOAP file for your TLP * [`FLUME-2454 `__] - Support batchSize to allow multiple events per transaction to the Kafka Sink * [`FLUME-2455 `__] - Documentation update for Kafka Sink * [`FLUME-2523 `__] - Document Kafka channel * [`FLUME-2612 `__] - Update kite to 0.17.1 ** Test * [`FLUME-1501 `__] - Flume Scribe Source needs unit tests.