Hadoop 2.6.5 Release Notes

• YARN-5483. Major improvement reported by sandflee and fixed by sandflee
  Optimize RMAppAttempt#pullJustFinishedContainers
  

• YARN-5462. Major bug reported by Eric Badger and fixed by Eric Badger
  TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently
  

• YARN-5353. Critical bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  ResourceManager can leak delegation tokens when they are shared across apps
  

• YARN-5262. Major bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
  Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat
  

• YARN-5206. Minor bug reported by Steve Loughran and fixed by Steve Loughran (client , security)
  RegistrySecurity includes id:pass in exception text if considered invalid
  

• YARN-5197. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  RM leaks containers if running container disappears from node update
  

• YARN-5009. Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  NMLeveldbStateStoreService database can grow substantially leading to longer recovery times
  

• YARN-4794. Critical bug reported by Sumana Sathish and fixed by Jian He
  Deadlock in NMClientImpl
  

• YARN-4785. Major bug reported by Jayesh and fixed by Varun Vasudev (webapp)
  inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API - cluster/scheduler
  

• YARN-4773. Minor bug reported by Jason Lowe and fixed by Jun Gong (nodemanager)
  Log aggregation performs extraneous filesystem operations when rolling log aggregation is disabled
  

• YARN-4761. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (fairscheduler)
  NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations on fair scheduler
  

• YARN-4722. Major bug reported by Jason Lowe and fixed by Jason Lowe
  AsyncDispatcher logs redundant event queue sizes
  

• YARN-4690. Major improvement reported by Ming Ma and fixed by Ming Ma
  Skip object allocation in FSAppAttempt#getResourceUsage when possible
  

• YARN-4573. Major sub-task reported by Takashi Ohnishi and fixed by Takashi Ohnishi (resourcemanager , test)
  TestRMAppTransitions.testAppRunningKill and testAppKilledKilled fail on trunk
  

• YARN-4459. Major bug reported by Jun Gong and fixed by Jun Gong (nodemanager)
  container-executor should only kill process groups
  

• YARN-4393. Major sub-task reported by Varun Saxena and fixed by Varun Saxena (test)
  TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently
  

• YARN-2046. Major bug reported by Jason Lowe and fixed by Ming Ma (nodemanager)
  Out of band heartbeats are sent only on container kill and possibly too early
  

• MAPREDUCE-6689. Blocker bug reported by Wangda Tan and fixed by Wangda Tan
  MapReduce job can infinitely increase number of reducer resource requests
  

• MAPREDUCE-6656. Blocker bug reported by J.Andreina and fixed by J.Andreina
  [NNBench] OP_DELETE operation isn't working after MAPREDUCE-6363
  

• MAPREDUCE-6637. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (test)
  Testcase Failure : TestFileInputFormat.testSplitLocationInfo
  

• MAPREDUCE-6635. Critical bug reported by Sergey Shelukhin and fixed by Junping Du
  Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException
  

• MAPREDUCE-6622. Critical improvement reported by Ray Chiang and fixed by Ray Chiang (jobhistoryserver)
  Add capability to set JHS job cache to a task-based limit
  

  Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property: 1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB. 2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth.

• MAPREDUCE-6558. Major bug reported by Wilfred Spiegelenburg and fixed by Wilfred Spiegelenburg (mrv1 , mrv2)
  multibyte delimiters with compressed input files generate duplicate records
  

• MAPREDUCE-6514. Blocker bug reported by Varun Saxena and fixed by Varun Saxena (applicationmaster)
  Job hangs as ask is not updated after ramping down of all reducers
  

• MAPREDUCE-6413. Major bug reported by Jason Lowe and fixed by zhihai xu (test)
  TestLocalJobSubmission is failing with unknown host
  

• MAPREDUCE-6302. Critical bug reported by mai shurong and fixed by Karthik Kambatla
  Preempt reducers after a configurable timeout irrespective of headroom
  

• MAPREDUCE-6191. Minor test reported by sam liu and fixed by sam liu (test)
  TestJavaSerialization fails with getting incorrect MR job result
  

• MAPREDUCE-5817. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (applicationmaster)
  Mappers get rescheduled on node transition even after all reducers are completed
  

• HDFS-10870. Trivial bug reported by John Zhuge and fixed by John Zhuge (documentation)
  Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm
  

• HDFS-10763. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Open files can leak permanently due to inconsistent lease update
  

• HDFS-10653. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Optimize conversion from path string to components
  

• HDFS-10544. Major bug reported by Zhe Zhang and fixed by Zhe Zhang (balancer & mover , ha)
  Balancer doesn't work with IPFailoverProxyProvider
  

• HDFS-10458. Major bug reported by Zhe Zhang and fixed by Zhe Zhang (encryption , namenode)
  getFileEncryptionInfo should return quickly for non-encrypted cluster
  

• HDFS-10377. Major improvement reported by Konstantin Shvachko and fixed by Yiqun Lin (logging , namenode)
  CacheReplicationMonitor shutdown log message should use INFO level.
  

• HDFS-10271. Critical bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Extra bytes are getting released from reservedSpace for append
  

• HDFS-10264. Major improvement reported by Konstantin Shvachko and fixed by Xiaobing Zhou (namenode)
  Logging improvements in FSImageFormatProtobuf.Saver
  

• HDFS-10182. Major bug reported by zhouyingchao and fixed by zhouyingchao
  Hedged read might overwrite user's buf
  

• HDFS-10178. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Permanent write failures can happen if pipeline recoveries occur for the first packet
  

• HDFS-9752. Critical bug reported by Kihwal Lee and fixed by Walter Su
  Permanent write failures may happen to slow writers during datanode rolling upgrades
  

• HDFS-9696. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Garbage snapshot records lingering forever
  

• HDFS-9688. Major test reported by Zhe Zhang and fixed by Zhe Zhang (encryption , test)
  Test the effect of nested encryption zones in HDFS downgrade
  

• HDFS-9669. Major improvement reported by Elliott Clark and fixed by Elliott Clark
  TcpPeerServer should respect ipc.server.listen.queue.size
  

• HDFS-9629. Major improvement reported by Xiao Chen and fixed by Xiao Chen
  Update the footer of Web UI to show year 2016
  

• HDFS-9530. Critical bug reported by Fei Hui and fixed by Brahma Reddy Battula (datanode)
  ReservedSpace is not cleared for abandoned Blocks
  

• HDFS-9365. Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Balancer does not work with the HDFS-6376 HA setup
  

• HDFS-9347. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (test)
  Invariant assumption in TestQuorumJournalManager.shutdown() is wrong
  

• HDFS-8845. Major bug reported by Chang Li and fixed by Chang Li
  DiskChecker should not traverse the entire tree
  

• HDFS-8581. Minor bug reported by tongshiquan and fixed by J.Andreina (namenode)
  ContentSummary on / skips further counts on yielding lock
  

• HDFS-7530. Minor bug reported by Charles Lamb and fixed by Charles Lamb (namenode)
  Allow renaming of encryption zone roots
  

• HDFS-7258. Minor bug reported by Xiaoyu Yao and fixed by Xiaoyu Yao (namenode)
  CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level
  

• HDFS-2580. Minor improvement reported by Harsh J and fixed by Harsh J (namenode)
  NameNode#main(...) can make use of GenericOptionsParser.
  

• HADOOP-13579. Blocker bug reported by Akira Ajisaka and fixed by Tsuyoshi Ozawa
  Fix source-level compatibility after HADOOP-11252
  

• HADOOP-13494. Major bug reported by Sean Mackrory and fixed by Sean Mackrory (security)
  ReconfigurableBase can log sensitive information
  

• HADOOP-13434. Major bug reported by Owen O'Malley and fixed by Owen O'Malley
  Add quoting to Shell class
  

• HADOOP-13350. Blocker bug reported by Xiao Chen and fixed by Xiao Chen (build)
  Additional fix to LICENSE and NOTICE
  

• HADOOP-13298. Minor improvement reported by Xiao Chen and fixed by Tsuyoshi Ozawa
  Fix the leftover L&N files in hadoop-build-tools/src/main/resources/META-INF/
  

• HADOOP-13297. Major bug reported by Akira Ajisaka and fixed by Sean Busbey (build)
  Add missing dependency in setting maven-remote-resource-plugin to fix builds
  

• HADOOP-13290. Major improvement reported by Konstantin Shvachko and fixed by Jonathan Hung (ipc)
  Appropriate use of generics in FairCallQueue
  

• HADOOP-13255. Major bug reported by Xiao Chen and fixed by Xiao Chen (kms)
  KMSClientProvider should check and renew tgt when doing delegation token operations.
  

• HADOOP-13192. Critical bug reported by binde and fixed by binde (util)
  org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly
  

• HADOOP-13189. Major bug reported by Konstantin Shvachko and fixed by Vinitha Reddy Gankidi (ipc)
  FairCallQueue makes callQueue larger than the configured capacity.
  

• HADOOP-13052. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  ChecksumFileSystem mishandles crc file permissions
  

• HADOOP-13043. Major bug reported by Andrew Wang and fixed by Andrew Wang
  Add LICENSE.txt entries for bundled javascript dependencies
  

• HADOOP-13042. Major bug reported by Andrew Wang and fixed by Andrew Wang
  Restore lost leveldbjni LICENSE and NOTICE changes
  

• HADOOP-12958. Major bug reported by Jason Lowe and fixed by Sangjin Lee
  PhantomReference for filesystem statistics can trigger OOM
  

• HADOOP-12893. Blocker bug reported by Allen Wittenauer and fixed by Xiao Chen (build)
  Verify LICENSE.txt and NOTICE.txt
  

• HADOOP-12810. Major bug reported by Ryan Blue and fixed by Ryan Blue (fs , fs/s3)
  FileSystem#listLocatedStatus causes unnecessary RPC calls
  

• HADOOP-12805. Major improvement reported by Ted Yu and fixed by Ted Yu
  Annotate CanUnbuffer with @InterfaceAudience.Public
  

  Made CanBuffer interface public for use in client applications.

• HADOOP-12800. Major improvement reported by Zhe Zhang and fixed by Zhe Zhang (build , yetus)
  Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs
  

• HADOOP-12789. Minor improvement reported by Sangjin Lee and fixed by Sangjin Lee (util)
  log classpath of ApplicationClassLoader at INFO level
  

• HADOOP-12773. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (util)
  HBase classes fail to load with client/job classloader enabled
  

• HADOOP-12682. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang
  Fix TestKMS#testKMSRestart* failure
  

• HADOOP-12589. Major bug reported by Tsuyoshi Ozawa and fixed by Masatake Iwasaki (test)
  Fix intermittent test failure of TestCopyPreserveFlag
  

• HADOOP-12559. Major bug reported by Zhe Zhang and fixed by Zhe Zhang (security)
  KMS connection failures should trigger TGT renewal
  

• HADOOP-12482. Major bug reported by Tony Wu and fixed by Tony Wu
  Race condition in JMX cache update
  

• HADOOP-12348. Major bug reported by zhihai xu and fixed by zhihai xu (metrics)
  MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.
  

• HADOOP-11361. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Fix a race condition in MetricsSourceAdapter.updateJmxCache
  

• HADOOP-11301. Major improvement reported by Maysam Yabandeh and fixed by Maysam Yabandeh
  [optionally] update jmx cache to drop old metrics
  

• HADOOP-7817. Major bug reported by Kristofer Tomasette and fixed by Kanaka Kumar Avvaru (fs)
  RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos()