============= Version 1.5.0 ============= .. rubric:: Status of this release Apache Flume 1.5.0 is the fifth release of Flume as an Apache top-level project (TLP). Apache Flume 1.5.0 is production-ready software. .. rubric:: Release Documentation * `Flume 1.5.0 User Guide `__ (also in `pdf `__) * `Flume 1.5.0 Developer Guide `__ (also in `pdf `__) * `Flume 1.5.0 API Documentation `__ .. rubric:: Changes Release Notes - Flume - Version v1.5.0 ** New Feature * [`FLUME-1227 `__] - Introduce some sort of SpillableChannel * [`FLUME-2056 `__] - Allow SpoolDir to pass just the filename that is the source of an event * [`FLUME-2071 `__] - Flume Context doesn't support float or double configuration values. * [`FLUME-2185 `__] - Upgrade morphlines to 0.7.0 * [`FLUME-2188 `__] - flume-ng-log4jappender Support user supplied headers * [`FLUME-2225 `__] - Elasticsearch Sink for ES HTTP API * [`FLUME-2294 `__] - Add a sink for Kite Datasets * [`FLUME-2309 `__] - Spooling directory should not always consume the oldest file first. ** Improvement * [`FLUME-1542 `__] - Flume User Guide sample configurations need fixing * [`FLUME-1899 `__] - Make SpoolDir work with Sub-Directories * [`FLUME-2052 `__] - Spooling directory source should be able to replace or ignore malformed characters * [`FLUME-2130 `__] - Handle larger payloads via SyslogUDPSource * [`FLUME-2139 `__] - upgrade morphline library dependency to a 0.5.0 release * [`FLUME-2154 `__] - Reducing duplicate events caused by reset-connection-interval * [`FLUME-2155 `__] - Improve replay time * [`FLUME-2181 `__] - Optionally disable File Channel fsyncs * [`FLUME-2202 `__] - AsyncHBaseSink should coalesce increments to reduce RPC roundtrips * [`FLUME-2206 `__] - ElasticSearchSink ttl field modification to mimic Elasticsearch way of specifying TTL * [`FLUME-2207 `__] - HDFS file suffix style index suffix in ElasticSearchSink * [`FLUME-2212 `__] - upgrade to Morphlines-0.8.0 * [`FLUME-2213 `__] - MorphlineInterceptor should share metric registry across threads for better (aggregate) reporting * [`FLUME-2217 `__] - Preserve priority, timestamp and hostname fields in MultiportSyslogTcp and Udp sources * [`FLUME-2231 `__] - Add details in Flume Ganglia config in User Guide * [`FLUME-2243 `__] - AvroSource to use TransceiverThreadFactory for Thread naming while initializing NioServerSocketChannelFactory * [`FLUME-2267 `__] - Increase default transactionCapacity for FileChannel to 10000 from 1000 * [`FLUME-2275 `__] - Improve scalability of MorphlineInterceptor under contention * [`FLUME-2292 `__] - Upgrade mapdb to 0.9.8 * [`FLUME-2316 `__] - Upgrade MorphlineSolrSink to kite-0.12.0 * [`FLUME-2340 `__] - Refactor to make room for Morphlines Elasticsearch Sink * [`FLUME-2343 `__] - Add user impersonation to DatasetSink * [`FLUME-2351 `__] - Ability to override any parameter from the configuration file * [`FLUME-2352 `__] - HDFSCompressedDataStream should support appendBatch ** Bug * [`FLUME-1666 `__] - Syslog source strips timestamp and hostname from log message body * [`FLUME-1679 `__] - Add dependency on Guava to flume-ng-elasticsearch-sink POM * [`FLUME-1892 `__] - IRC Sink NPE * [`FLUME-1951 `__] - Remove unused future from FlumeEventQueue * [`FLUME-2007 `__] - HDFS Sink should check if file is closed and retry if it is not. * [`FLUME-2088 `__] - Minor typo in Flume User Guide JSON Reporting section * [`FLUME-2109 `__] - HTTPS support in HTTP Source * [`FLUME-2110 `__] - Scribe Source must check if category is null before inserting the headers * [`FLUME-2119 `__] - duplicate files cause flume to enter irrecoverable state * [`FLUME-2121 `__] - Upgrade Flume to log4j 1.2.17 to be in sync with HDFS/HBase * [`FLUME-2122 `__] - Minor cleanups of User guide * [`FLUME-2123 `__] - Morphline Solr sink missing short type name * [`FLUME-2124 `__] - Upgrade Morphline Solr Sink to CDK 0.4.1 * [`FLUME-2127 `__] - JMX shutdown command for Flume * [`FLUME-2134 `__] - AsyncHbase Sink bugfix plus tests errors on Windows * [`FLUME-2135 `__] - Add zip to the build distribution for Windows support * [`FLUME-2136 `__] - Windows - Fix intermitent test failure in TestMonitoredCounterGroup.java * [`FLUME-2137 `__] - Fix StagedInstall.java to invoke the correct startup script on Windows * [`FLUME-2142 `__] - HTTPS tests for http source * [`FLUME-2145 `__] - TestCheckpointRebuilder.testFastReplay fails on Windows due to checkpoint file being memory mapped * [`FLUME-2151 `__] - Windows: Update TestExecSource to use native commands on Windows * [`FLUME-2152 `__] - Flume user guide says "Ganglia support" where it should say "JSON support" * [`FLUME-2156 `__] - Unregister then re-register MonitoredCounterGroup JMX MBeans on reconfigure * [`FLUME-2157 `__] - Spool directory source does not shut down correctly when Flume is reconfigured * [`FLUME-2159 `__] - Sporadic failures in TestNettyAvroRpcClient.spinThreadsCrazily() * [`FLUME-2161 `__] - Flume does not support spaces in -X java-opt command line args * [`FLUME-2172 `__] - Update protocol buffer from 2.4.1 to 2.5.0 * [`FLUME-2176 `__] - SpoolDir Source, get 'File has changed' exception but actually there is no change on the file * [`FLUME-2182 `__] - Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer * [`FLUME-2184 `__] - flume-ng-morphline-solr-sink Build failing due to incorrect hadoop-common dependency declaration * [`FLUME-2191 `__] - HDFS Minicluster tests failing after protobuf upgrade. * [`FLUME-2192 `__] - AbstractSinkProcessor stop incorrectly calls start * [`FLUME-2198 `__] - Avro Source should disable itself if ipFilterRules contains invalid rules * [`FLUME-2199 `__] - Flume builds with new version require mvn install before site can be generated * [`FLUME-2200 `__] - HTTP Source should be able to use "port" parameter if SSL is enabled * [`FLUME-2208 `__] - Jetty's default SocketSelector leaks File descriptors * [`FLUME-2209 `__] - AsyncHBaseSink will never recover if the column family does not exists for the first start * [`FLUME-2210 `__] - UnresolvedAddressException when using multiple hostNames in Elasticsearch sink configuration * [`FLUME-2220 `__] - ElasticSearch sink - duplicate fields in indexed document * [`FLUME-2229 `__] - Backoff period gets reset too often in OrderSelector * [`FLUME-2233 `__] - MemoryChannel lock contention on every put due to bytesRemaining Semaphore * [`FLUME-2235 `__] - idleFuture should be cancelled at the start of append * [`FLUME-2238 `__] - Provide option to configure worker threads in NettyAvroRpcClient * [`FLUME-2239 `__] - Clarify File Channel's dataDirs setting in User Guide * [`FLUME-2252 `__] - Add null check before closing table in HbaseSink * [`FLUME-2253 `__] - Please delete old releases from mirroring system * [`FLUME-2255 `__] - Spooling Directory Source cannot handle channel exceptions * [`FLUME-2259 `__] - transaction closure not happening for all the scenario in hbasesink * [`FLUME-2262 `__] - Log4j Appender should use timeStamp field not getTimestamp * [`FLUME-2263 `__] - Bump Hadoop 2 version to 2.3.0 * [`FLUME-2264 `__] - Log4j Appender + Avro Reflection on string results in an invalid avro schema * [`FLUME-2265 `__] - Closed bucket writers should be removed from sfwriters map * [`FLUME-2266 `__] - Update Morphline Sink to kite-0.10.0 * [`FLUME-2270 `__] - Twitter Source Documentation Does not load properly * [`FLUME-2272 `__] - Getting start page returning 503 error * [`FLUME-2283 `__] - Spool Dir source must check interrupt flag before writing to channel * [`FLUME-2289 `__] - Disable maxUnderReplication test which is extremely flakey * [`FLUME-2301 `__] - Update HBaseSink tests to reflect sink returning backoff only on empty batches * [`FLUME-2302 `__] - TestHDFS Sink fails with Can't get Kerberos realm * [`FLUME-2303 `__] - HBaseSink tests can fail based on order of execution * [`FLUME-2304 `__] - DatasetSink test fails unexpectedly * [`FLUME-2305 `__] - BucketWriter#close must cancel idleFuture * [`FLUME-2307 `__] - Remove Log writetimeout * [`FLUME-2311 `__] - Use standard way of finding queue/topic * [`FLUME-2312 `__] - Add utility for adorning HTTP contexts in Jetty * [`FLUME-2314 `__] - Upgrade to Mapdb 0.9.9 * [`FLUME-2320 `__] - Deadlock in DatasetSink * [`FLUME-2323 `__] - Morphline sink must increment eventDrainAttemptCount when it takes event from channel * [`FLUME-2324 `__] - Support writing to multiple HBase clusters using HBaseSink * [`FLUME-2325 `__] - BucketWriter might throw BucketClosedException incorrectly * [`FLUME-2328 `__] - FileChannel Dual Checkpoint Backup Thread not released on Application stop * [`FLUME-2329 `__] - Add an alias for the Morphline Solr Sink * [`FLUME-2330 `__] - Remove the MorphlineHandlerImpl configuration option from MorphlineSink * [`FLUME-2334 `__] - Upgrade Asynchbase to 1.5.0 as this include hbase 0.96.x support * [`FLUME-2335 `__] - TestHBaseSink#testWithoutConfigurationObject() must delete the table at the end of the test * [`FLUME-2336 `__] - HBase tests that pass in ZK configs must use a new context object * [`FLUME-2338 `__] - Support coalescing increments in HBaseSink * [`FLUME-2345 `__] - Update to Kite 0.12.0 dependency * [`FLUME-2347 `__] - Add FLUME_JAVA_OPTS which allows users to inject java properties from cmd line * [`FLUME-2350 `__] - Consume Order tests need to space out file creation * [`FLUME-2357 `__] - HDFS sink should retry closing files that previously had close errors * [`FLUME-2381 `__] - Upgrade Hadoop version in Hadoop 2 profile to 2.4.0 * [`FLUME-2379 `__] - Flume's pom files are invalid and not parseable by non-maven build tools ** Documentation * [`FLUME-1223 `__] - Userguide improvement rolling file sink * [`FLUME-1678 `__] - Incorrect documentation for HBase sink * [`FLUME-1851 `__] - User Guide grammar mistake * [`FLUME-2064 `__] - Typo/Grammar in flume main user doc under Scribe * [`FLUME-2065 `__] - Regex Extractor Interceptor config agent name inconsistent with rest of docs * [`FLUME-2183 `__] - Add "Other Resources" page to wiki * [`FLUME-2278 `__] - Incorrect documentation for write-timeout of File Channel * [`FLUME-2319 `__] - Incorrect property name of Ganglia Reporting documentation