Hadoop 2.7.4 Release Notes

• YARN-6818. Major bug reported by Jonathan Hung and fixed by Jonathan Hung
  User limit per partition is not honored in branch-2.7 >=
  

• YARN-6719. Major bug reported by Akira Ajisaka and fixed by Akira Ajisaka
  Fix findbugs warnings in SLSCapacityScheduler.java
  

• YARN-6615. Major bug reported by Wilfred Spiegelenburg and fixed by Wilfred Spiegelenburg
  AmIpFilter drops query parameters on redirect
  

• YARN-6493. Major improvement reported by Jonathan Hung and fixed by Jonathan Hung
  Print requested node partition in assignContainer logs
  

• YARN-6457. Major improvement reported by Sanjay M Pujare and fixed by Sanjay M Pujare (webapp , yarn)
  Allow custom SSL configuration to be supplied in WebApps
  

• YARN-6310. Major bug reported by Haibo Chen and fixed by Haibo Chen (yarn)
  OutputStreams in AggregatedLogFormat.LogWriter can be left open upon exceptions
  

• YARN-6304. Major bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
  Skip rm.transitionToActive call to RM if RM is already active.
  

• YARN-6274. Trivial task reported by Charles Zhang and fixed by Weiwei Yang (documentation)
  Documentation refers to incorrect nodemanager health checker interval property
  

• YARN-6152. Major bug reported by Jonathan Hung and fixed by Jonathan Hung
  Used queue percentage not accurate in UI for 2.7 and below when using DominantResourceCalculator
  

• YARN-6024. Major bug reported by Wangda Tan and fixed by Wangda Tan
  Capacity Scheduler 'continuous reservation looking' doesn't work when sum of queue's used and reserved resources is equal to max
  

• YARN-5988. Blocker bug reported by Ajith S and fixed by Ajith S
  RM unable to start in secure setup
  

• YARN-5837. Major bug reported by Robert Kanter and fixed by Robert Kanter
  NPE when getting node status of a decommissioned node after an RM restart
  

• YARN-5694. Critical bug reported by Daniel Templeton and fixed by Daniel Templeton (resourcemanager)
  ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable
  

• YARN-5550. Minor improvement reported by Jonathan Hung and fixed by Jonathan Hung (client , test)
  TestYarnCLI#testGetContainers should format according to CONTAINER_PATTERN
  

• YARN-5543. Major bug reported by Min Shen and fixed by Min Shen (capacityscheduler , resourcemanager)
  ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread
  

• YARN-5540. Major improvement reported by Nathan Roberts and fixed by Jason Lowe (capacity scheduler , fairscheduler , resourcemanager)
  scheduler spends too much time looking at empty priorities
  

• YARN-5483. Major improvement reported by sandflee and fixed by sandflee
  Optimize RMAppAttempt#pullJustFinishedContainers
  

• YARN-5469. Minor bug reported by Eric Badger and fixed by Eric Badger
  Increase timeout of TestAmFilter.testFilter
  

• YARN-5462. Major bug reported by Eric Badger and fixed by Eric Badger
  TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently
  

• YARN-5382. Major bug reported by Jason Lowe and fixed by Vrushali C (resourcemanager)
  RM does not audit log kill request for active applications
  

• YARN-5353. Critical bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  ResourceManager can leak delegation tokens when they are shared across apps
  

• YARN-5333. Major bug reported by Jun Gong and fixed by Jun Gong
  Some recovered apps are put into default queue when RM HA
  

• YARN-5262. Major bug reported by Rohith Sharma K S and fixed by Rohith Sharma K S (resourcemanager)
  Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat
  

• YARN-5197. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  RM leaks containers if running container disappears from node update
  

• YARN-5121. Blocker bug reported by Allen Wittenauer and fixed by Allen Wittenauer (nodemanager , security)
  fix some container-executor portability issues
  

• YARN-5092. Major test reported by Rohith Sharma K S and fixed by Jason Lowe (test)
  TestRMDelegationTokens fails intermittently
  

• YARN-5001. Major bug reported by Haibo Chen and fixed by Haibo Chen (log-aggregation , nodemanager , security)
  Aggregated Logs root directory is created with wrong group if nonexistent
  

• YARN-4927. Major bug reported by Karthik Kambatla and fixed by Bibin A Chundatt (test)
  TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler
  

• YARN-4925. Major bug reported by Bibin A Chundatt and fixed by Bibin A Chundatt
  ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
  

• YARN-4717. Minor test reported by Daniel Templeton and fixed by Daniel Templeton (nodemanager)
  TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup
  

• YARN-4612. Major bug reported by Ming Ma and fixed by Ming Ma
  Fix rumen and scheduler load simulator handle killed tasks properly
  

• YARN-4573. Major sub-task reported by Takashi Ohnishi and fixed by Takashi Ohnishi (resourcemanager , test)
  TestRMAppTransitions.testAppRunningKill and testAppKilledKilled fail on trunk
  

• YARN-4562. Major bug reported by Sergey Shelukhin and fixed by Sergey Shelukhin
  YARN WebApp ignores the configuration passed to it for keystore settings
  

• YARN-4393. Major sub-task reported by Varun Saxena and fixed by Varun Saxena (test)
  TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently
  

• YARN-4367. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (scheduler-load-simulator)
  SLS webapp doesn't load
  

• YARN-4355. Major bug reported by Jason Lowe and fixed by Varun Saxena (nodemanager)
  NPE while processing localizer heartbeat
  

• YARN-4328. Minor bug reported by Varun Saxena and fixed by Akira Ajisaka (resourcemanager)
  Findbugs warning in resourcemanager in branch-2.7 and branch-2.6
  

• YARN-4302. Major bug reported by Bibin A Chundatt and fixed by Bibin A Chundatt
  SLS not able start due to NPE in SchedulerApplicationAttempt#getResourceUsageReport
  

• YARN-4250. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (resourcemanager , scheduler)
  NPE in AppSchedulingInfo#isRequestLabelChanged
  

• YARN-4140. Major sub-task reported by Bibin A Chundatt and fixed by Bibin A Chundatt (scheduler)
  RM container allocation delayed incase of app submitted to Nodelabel partition
  

• YARN-4109. Minor bug reported by Bibin A Chundatt and fixed by Mohammad Shahid Khan
  Exception on RM scheduler page loading with labels
  

• YARN-4017. Major bug reported by Allen Wittenauer and fixed by Sidharta Seethana (nodemanager)
  container-executor overuses PATH_MAX
  

• YARN-3877. Minor improvement reported by Steve Loughran and fixed by Varun Saxena (client)
  YarnClientImpl.submitApplication swallows exceptions
  

• YARN-3707. Blocker bug reported by Wangda Tan and fixed by Wangda Tan
  RM Web UI queue filter doesn't work
  

• YARN-3432. Major bug reported by Thomas Graves and fixed by Brahma Reddy Battula (capacityscheduler , resourcemanager)
  Cluster metrics have wrong Total Memory when there is reserved memory on CS
  

• YARN-3269. Major bug reported by Xuan Gong and fixed by Xuan Gong
  Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path
  

• YARN-3260. Critical bug reported by Jason Lowe and fixed by Bibin A Chundatt (resourcemanager)
  AM attempt fail to register before RM processes launch event
  

• YARN-1728. Major bug reported by Abraham Elmahrek and fixed by Yuanbo Liu
  Workaround guice3x-undecoded pathInfo in YARN WebApp
  

• YARN-1471. Minor bug reported by Carlo Curino and fixed by Carlo Curino
  The SLS simulator is not running the preemption policy for CapacityScheduler
  

• MAPREDUCE-6873. Minor bug reported by Erik Krogen and fixed by Erik Krogen (mrv2)
  MR Job Submission Fails if MR framework application path not on defaultFS
  

• MAPREDUCE-6771. Major bug reported by Haibo Chen and fixed by Haibo Chen (mrv2)
  RMContainerAllocator sends container diagnostics event after corresponding completion event
  

• MAPREDUCE-6768. Major bug reported by Haibo Chen and fixed by Haibo Chen (mrv2)
  TestRecovery.testSpeculative failed with NPE
  

• MAPREDUCE-6763. Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  Shuffle server listen queue is too small
  

• MAPREDUCE-6744. Major bug reported by Eric Badger and fixed by Eric Badger
  Increase timeout on TestDFSIO tests
  

• MAPREDUCE-6741. Major improvement reported by Haibo Chen and fixed by Haibo Chen (mrv2)
  add MR support to redact job conf properties
  

• MAPREDUCE-6724. Major bug reported by Haibo Chen and fixed by Haibo Chen (mrv2)
  Single shuffle to memory must not exceed Integer#MAX_VALUE
  

• MAPREDUCE-6711. Major bug reported by Li Lu and fixed by Prabhu Joseph
  JobImpl fails to handle preemption events on state COMMITTING
  

• MAPREDUCE-6697. Major bug reported by Jason Lowe and fixed by Nathan Roberts (mrv2)
  Concurrent task limits should only be applied when necessary
  

• MAPREDUCE-6433. Major bug reported by Allen Wittenauer and fixed by zhihai xu (jobhistoryserver , mrv2)
  launchTime may be negative
  

• MAPREDUCE-6304. Major new feature reported by Jian Fang and fixed by Naganarasimha G R
  Specifying node labels when submitting MR jobs
  

• MAPREDUCE-4784. Major bug reported by Jason Lowe and fixed by Haibo Chen (mrv2 , test)
  TestRecovery occasionally fails
  

• HDFS-12177. Major bug reported by Jiandan Yang and fixed by Jiandan Yang (block placement)
  NameNode exits due to setting BlockPlacementPolicy loglevel to Debug
  

• HDFS-11896. Blocker bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Non-dfsUsed will be doubled on dead node re-registration
  

• HDFS-11893. Major bug reported by Konstantin Shvachko and fixed by Brahma Reddy Battula (test)
  Fix TestDFSShell.testMoveWithTargetPortEmpty failure.
  

• HDFS-11864. Major bug reported by Brahma Reddy Battula and fixed by Yiqun Lin (documentation)
  Document Metrics to track usage of memory for writes
  

• HDFS-11856. Major bug reported by Vinayakumar B and fixed by Vinayakumar B (hdfs-client , rolling upgrades)
  Ability to re-add Upgrading Nodes (remote) to pipeline for future pipeline updates
  

• HDFS-11849. Major bug reported by Surendra Singh Lilhore and fixed by Surendra Singh Lilhore (journal-node)
  JournalNode startup failure exception should be logged in log file
  

• HDFS-11795. Major bug reported by Yiqun Lin and fixed by Yiqun Lin
  Fix ASF Licence warnings in branch-2.7
  

• HDFS-11766. Major bug reported by Akira Ajisaka and fixed by Chen Liang
  Fix findbugs warning in branch-2.7
  

• HDFS-11743. Blocker bug reported by Zhe Zhang and fixed by Zhe Zhang (namenode)
  Revert the incompatible fsck reporting output in HDFS-7933 from branch-2.7
  

• HDFS-11742. Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee
  Improve balancer usability after HDFS-8818
  

• HDFS-11741. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (balancer & mover)
  Long running balancer may fail due to expired DataEncryptionKey
  

• HDFS-11736. Major bug reported by Konstantin Shvachko and fixed by Yiqun Lin
  OIV tests should not write outside 'target' directory.
  

• HDFS-11717. Minor task reported by Erik Krogen and fixed by Erik Krogen (ha , namenode)
  Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully
  

• HDFS-11714. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Newly added NN storage directory won't get initialized and cause space exhaustion
  

• HDFS-11709. Critical bug reported by Zhe Zhang and fixed by Erik Krogen (ha , namenode)
  StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully
  

• HDFS-11708. Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (hdfs-client)
  Positional read will fail if replicas moved to different DNs after stream is opened
  

• HDFS-11674. Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (datanode)
  reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered
  

• HDFS-11648. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp
  Lazy construct the IIP pathname
  

• HDFS-11634. Major improvement reported by Konstantin Shvachko and fixed by Konstantin Shvachko
  Optimize BlockIterator when iterating starts in the middle.
  

• HDFS-11628. Major improvement reported by Xiaobing Zhou and fixed by Xiaobing Zhou (documentation)
  Clarify the behavior of HDFS Mover in documentation
  

• HDFS-11615. Major bug reported by Erik Krogen and fixed by Erik Krogen (hdfs)
  FSNamesystemLock metrics can be inaccurate due to millisecond precision
  

• HDFS-11609. Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  Some blocks can be permanently lost if nodes are decommissioned while dead
  

• HDFS-11608. Critical bug reported by Xiaobing Zhou and fixed by Xiaobing Zhou (hdfs-client)
  HDFS write crashed with block size greater than 2 GB
  

• HDFS-11583. Major bug reported by Karan Mehta and fixed by Masatake Iwasaki (tracing)
  Parent spans are not initialized to NullScope for every DFSPacket
  

• HDFS-11499. Major bug reported by Lukas Majercak and fixed by Lukas Majercak (hdfs , namenode)
  Decommissioning stuck because of failing recovery
  

  Allow a block to complete if the number of replicas on live nodes, decommissioning nodes and nodes in maintenance mode satisfies minimum replication factor. The fix prevents block recovery failure if replica of last block is being decommissioned. Vice versa, the decommissioning will be stuck, waiting for the last block to be completed. In addition, file close() operation will not fail due to last block being decommissioned.

• HDFS-11486. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang
  Client close() should not fail fast if the last block is being decommissioned
  

• HDFS-11472. Critical bug reported by Wei-Chiu Chuang and fixed by Erik Krogen (datanode)
  Fix inconsistent replica size after a data pipeline failure
  

• HDFS-11466. Major improvement reported by Andrew Wang and fixed by Andrew Wang (namenode)
  Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms
  

• HDFS-11445. Critical bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  FSCK shows overall health stauts as corrupt even one replica is corrupt
  

• HDFS-11384. Major improvement reported by yunjiong zhao and fixed by Konstantin Shvachko (balancer & mover)
  Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
  

• HDFS-11379. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs-client)
  DFSInputStream may infinite loop requesting block locations
  

• HDFS-11377. Major bug reported by yunjiong zhao and fixed by yunjiong zhao (balancer & mover)
  Balancer hung due to no available mover threads
  

• HDFS-11373. Critical bug reported by Akira Ajisaka and fixed by Akira Ajisaka
  Backport HDFS-11258 and HDFS-11272 to branch-2.7
  

• HDFS-11352. Critical bug reported by Erik Krogen and fixed by Erik Krogen (namenode)
  Potential deadlock in NN when failing over
  

• HDFS-11345. Minor improvement reported by Zhe Zhang and fixed by Erik Krogen (documentation , namenode)
  Document the configuration key for FSNamesystem lock fairness
  

• HDFS-11333. Minor improvement reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (namenode)
  Print a user friendly error message when plugins are not found
  

• HDFS-11290. Major test reported by Akira Ajisaka and fixed by Erik Krogen (test)
  TestFSNameSystemMBean should wait until JMX cache is cleared
  

• HDFS-11280. Major bug reported by Zheng Shao and fixed by Zheng Shao (hdfs)
  Allow WebHDFS to reuse HTTP connections to NN
  

• HDFS-11263. Critical bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  ClassCastException when we use Bzipcodec for Fsimage compression
  

• HDFS-11229. Blocker bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (datanode)
  HDFS-11056 failed to close meta file
  

  The fix for HDFS-11056 reads meta file to load last partial chunk checksum when a block is converted from finalized/temporary to rbw. However, it did not close the file explicitly, which may cause number of open files reaching system limit. This jira fixes it by closing the file explicitly after the meta file is read.

• HDFS-11180. Blocker bug reported by Abhishek Modi and fixed by Akira Ajisaka (namenode)
  Intermittent deadlock in NameNode when failover happens.
  

• HDFS-11174. Minor bug reported by John Zhuge and fixed by John Zhuge (documentation , httpfs)
  Wrong HttpFS test command in doc
  

• HDFS-11160. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (datanode)
  VolumeScanner reports write-in-progress replicas as corrupt incorrectly
  

  Fixed a race condition that caused VolumeScanner to recognize a good replica as a bad one if the replica is also being written concurrently.

• HDFS-11087. Major bug reported by Konstantin Shvachko and fixed by Erik Krogen (namenode)
  NamenodeFsck should check if the output writer is still writable.
  

• HDFS-11078. Major bug reported by Inigo Goiri and fixed by Inigo Goiri
  Fix NPE in LazyPersistFileScrubber
  

• HDFS-11069. Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (datanode , security)
  Tighten the authorization of datanode RPC
  

• HDFS-11056. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (datanode , httpfs)
  Concurrent append and read operations lead to checksum error
  

  Load last partial chunk checksum properly into memory when converting a finalized/temporary replica to rbw replica. This ensures concurrent reader reads the correct checksum that matches the data before the update.

• HDFS-11053. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode , security)
  Unnecessary superuser check in versionRequest()
  

• HDFS-11015. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (balancer & mover)
  Enforce timeout in balancer
  

• HDFS-11002. Major bug reported by Mingliang Liu and fixed by Mingliang Liu (documentation)
  Fix broken attr/getfattr/setfattr links in ExtendedAttributes.md
  

• HDFS-10991. Blocker bug reported by Surendra Singh Lilhore and fixed by Surendra Singh Lilhore (libhdfs)
  Export hdfsTruncateFile symbol in libhdfs
  

• HDFS-10987. Critical bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Make Decommission less expensive when lot of blocks present.
  

• HDFS-10966. Major bug reported by Zhe Zhang and fixed by Mark Wagner (balancer & mover)
  Enhance Dispatcher logic on deciding when to give up a source DataNode
  

• HDFS-10941. Major improvement reported by Xiaoyu Yao and fixed by Chen Liang (namenode)
  Improve BlockManager#processMisReplicatesAsync log
  

• HDFS-10921. Major bug reported by Eric Badger and fixed by Eric Badger
  TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
  

• HDFS-10915. Minor bug reported by Xiaobing Zhou and fixed by Xiaobing Zhou (test)
  Fix time measurement bug in TestDatanodeRestart#testWaitForRegistrationOnRestart
  

• HDFS-10896. Major sub-task reported by Erik Krogen and fixed by Erik Krogen (namenode)
  Move lock logging logic from FSNamesystem into FSNamesystemLock
  

• HDFS-10889. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (documentation)
  Remove outdated Fault Injection Framework documentaion
  

• HDFS-10879. Major bug reported by Xiao Chen and fixed by Xiao Chen
  TestEncryptionZonesWithKMS#testReadWrite fails intermittently
  

• HDFS-10878. Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah (hdfs-client)
  TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException
  

• HDFS-10872. Major sub-task reported by Erik Krogen and fixed by Erik Krogen (namenode)
  Add MutableRate metrics for FSNamesystemLock operations
  

• HDFS-10843. Major bug reported by Erik Krogen and fixed by Erik Krogen (hdfs , namenode)
  Update space quota when a UC block is completed rather than committed.
  

• HDFS-10817. Major improvement reported by Erik Krogen and fixed by Erik Krogen (logging , namenode)
  Add Logging for Long-held NN Read Locks
  

• HDFS-10809. Major bug reported by Zhe Zhang and fixed by Vinitha Reddy Gankidi (encryption , namenode)
  getNumEncryptionZones causes NPE in branch-2.7
  

• HDFS-10807. Minor improvement reported by Mingliang Liu and fixed by Mingliang Liu (documentation)
  Doc about upgrading to a version of HDFS with snapshots may be confusing
  

• HDFS-10798. Major improvement reported by Zhe Zhang and fixed by Erik Krogen (logging , namenode)
  Make the threshold of reporting FSNamesystem lock contention configurable
  

• HDFS-10763. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Open files can leak permanently due to inconsistent lease update
  

• HDFS-10747. Minor bug reported by Mingliang Liu and fixed by Mingliang Liu (hdfs-client)
  o.a.h.hdfs.tools.DebugAdmin usage message is misleading
  

• HDFS-10745. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Directly resolve paths into INodesInPath
  

• HDFS-10744. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Internally optimize path component resolution
  

• HDFS-10733. Major bug reported by Konstantin Shvachko and fixed by Vinitha Reddy Gankidi (namenode , qjm)
  NameNode terminated after full GC thinking QJM is unresponsive.
  

• HDFS-10716. Minor bug reported by Yiqun Lin and fixed by Yiqun Lin (balancer & mover)
  In Balancer, the target task should be removed when its size < 0.
  

• HDFS-10715. Major bug reported by Guangbin Zhu and fixed by Guangbin Zhu (namenode)
  NPE when applying AvailableSpaceBlockPlacementPolicy
  

• HDFS-10713. Major bug reported by Arpit Agarwal and fixed by Hanisha Koneru (logging , namenode)
  Throttle FsNameSystem lock warnings
  

• HDFS-10712. Major bug reported by Konstantin Shvachko and fixed by Vinitha Reddy Gankidi
  Fix TestDataNodeVolumeFailure on 2.* branches.
  

• HDFS-10694. Major bug reported by Konstantin Shvachko and fixed by Yuanbo Liu (logging , namenode)
  BlockManager.processReport() should print blockReportId in each log message.
  

• HDFS-10693. Major bug reported by Konstantin Shvachko and fixed by Yuanbo Liu (namenode)
  metaSave should print blocks, not LightWeightHashSet
  

• HDFS-10691. Major bug reported by Yiqun Lin and fixed by Yiqun Lin
  FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException
  

• HDFS-10674. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Optimize creating a full path from an inode
  

• HDFS-10673. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Optimize FSPermissionChecker's internal path usage
  

• HDFS-10662. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Optimize UTF8 string/byte conversions
  

• HDFS-10656. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Optimize conversion of byte arrays back to path string
  

• HDFS-10655. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Fix path related byte array conversion bugs
  

• HDFS-10653. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Optimize conversion from path string to components
  

• HDFS-10627. Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah (hdfs)
  Volume Scanner marks a block as "suspect" even if the exception is network-related
  

• HDFS-10625. Major improvement reported by Yongjun Zhang and fixed by Rushabh S Shah (datanode , hdfs)
  VolumeScanner to report why a block is found bad
  

• HDFS-10619. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (hdfs)
  Cache path in InodesInPath
  

• HDFS-10609. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (encryption)
  Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications
  

  If pipeline recovery fails due to expired encryption key, attempt to refresh the key and retry.

• HDFS-10544. Major bug reported by Zhe Zhang and fixed by Zhe Zhang (balancer & mover , ha)
  Balancer doesn't work with IPFailoverProxyProvider
  

• HDFS-10534. Major improvement reported by Zhe Zhang and fixed by Kai Sasaki (namenode , ui)
  NameNode WebUI should display DataNode usage histogram
  

• HDFS-10512. Major bug reported by Wei-Chiu Chuang and fixed by Yiqun Lin (datanode)
  VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks
  

• HDFS-10455. Minor bug reported by Tianyin Xu and fixed by Rakesh R (namenode)
  Logging the username when deny the setOwner operation
  

• HDFS-10396. Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang
  Using -diff option with DistCp may get "Comparison method violates its general contract" exception
  

• HDFS-10336. Major bug reported by Yiqun Lin and fixed by Yiqun Lin (test)
  TestBalancer failing intermittently because of not reseting UserGroupInformation completely
  

• HDFS-10301. Critical bug reported by Konstantin Shvachko and fixed by Vinitha Reddy Gankidi (namenode)
  BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
  

• HDFS-10276. Major bug reported by Kevin Cox and fixed by Yuanbo Liu
  HDFS should not expose path info that user has no permission to see.
  

• HDFS-10270. Minor bug reported by Andras Bokor and fixed by Gergely Novák (test)
  TestJMXGet:testNameNode() fails
  

• HDFS-9902. Major improvement reported by Pan Yuxuan and fixed by Brahma Reddy Battula (datanode)
  Support different values of dfs.datanode.du.reserved per storage type
  

  Reserved space can be configured independently for different storage types for clusters with heterogeneous storage. The 'dfs.datanode.du.reserved' property name can be suffixed with a storage types (i.e. one of ssd, disk, archival or ram_disk). e.g. reserved space for RAM_DISK storage can be configured using the property 'dfs.datanode.du.reserved.ram_disk'. If specific storage type reservation is not configured then the value specified by 'dfs.datanode.du.reserved' will be used for all volumes.

• HDFS-9888. Minor test reported by Xiao Chen and fixed by Xiao Chen
  Allow reseting KerberosName in unit tests
  

• HDFS-9885. Minor bug reported by Archana T and fixed by Surendra Singh Lilhore (distcp)
  Correct the distcp counters name while displaying counters
  

• HDFS-9804. Major new feature reported by Xiao Chen and fixed by Xiao Chen (balancer & mover , security)
  Allow long-running Balancer to login with keytab
  

• HDFS-9745. Minor test reported by Xiao Chen and fixed by Xiao Chen
  TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts
  

• HDFS-9726. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
  Refactor IBR code to a new class
  

• HDFS-9710. Major improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
  Change DN to send block receipt IBRs in batches
  

• HDFS-9696. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Garbage snapshot records lingering forever
  

• HDFS-9621. Critical sub-task reported by Sushmitha Sreenivasan and fixed by Jing Zhao (erasure-coding)
  getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory
  

• HDFS-9601. Major sub-task reported by Masatake Iwasaki and fixed by Masatake Iwasaki (test)
  NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block
  

• HDFS-9503. Major sub-task reported by Konstantin Shvachko and fixed by Mingliang Liu (test)
  Replace -namenode option with -fs for NNThroughputBenchmark
  

  The patch replaces -namenode option with -fs for specifying the remote name node against which the benchmark is running. Before this patch, if '-namenode' was not given, the benchmark would run in standalone mode, ignoring the 'fs.defaultFS' in config file even if it's remote. With this patch, the benchmark, as other tools, will rely on the 'fs.defaultFS' config, which is overridable by -fs command option, to run standalone mode or remote mode.

• HDFS-9500. Major bug reported by Phil Yang and fixed by Erik Krogen
  datanodesSoftwareVersions map may counting wrong when rolling upgrade
  

• HDFS-9467. Major bug reported by Mingliang Liu and fixed by Mingliang Liu (namenode)
  Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
  

• HDFS-9444. Major bug reported by Brahma Reddy Battula and fixed by Masatake Iwasaki
  Add utility to find set of available ephemeral ports to ServerSocketUtil
  

• HDFS-9421. Major sub-task reported by Xiaoyu Yao and fixed by Mingliang Liu (benchmarks)
  NNThroughputBenchmark replication test NPE with -namenode option
  

• HDFS-9412. Major improvement reported by He Tianyi and fixed by He Tianyi
  getBlocks occupies FSLock and takes too long to complete
  

• HDFS-9333. Minor test reported by Kai Zheng and fixed by Masatake Iwasaki (test)
  Some tests using MiniDFSCluster errored complaining port in use
  

• HDFS-9145. Major improvement reported by Jing Zhao and fixed by Mingliang Liu (namenode)
  Tracking methods that hold FSNamesytemLock for too long
  

• HDFS-9038. Major bug reported by Chris Nauroth and fixed by Brahma Reddy Battula (datanode)
  DFS reserved space is erroneously counted towards non-DFS used.
  

• HDFS-9019. Minor improvement reported by Thejas M Nair and fixed by Xiaoyu Yao (security)
  Adding informative message to sticky bit permission denied exception
  

• HDFS-8915. Minor bug reported by Anu Engineer and fixed by Masatake Iwasaki (test)
  TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins
  

• HDFS-8883. Major improvement reported by Anu Engineer and fixed by Anu Engineer (namenode)
  NameNode Metrics : Add FSNameSystem lock Queue Length
  

• HDFS-8873. Major improvement reported by Nathan Roberts and fixed by Daniel Templeton (datanode)
  Allow the directoryScanner to be rate-limited
  

• HDFS-8824. Major sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Do not use small blocks for balancing the cluster
  

• HDFS-8818. Major sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (balancer & mover)
  Allow Balancer to run faster
  

  Add a new conf "dfs.balancer.max-size-to-move" so that Balancer.MAX_SIZE_TO_MOVE becomes configurable.

• HDFS-8780. Major bug reported by J.Andreina and fixed by J.Andreina
  Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node.
  

• HDFS-8721. Major sub-task reported by Rakesh R and fixed by Rakesh R (encryption)
  Add a metric for number of encryption zones
  

• HDFS-8709. Minor improvement reported by Andrew Wang and fixed by Andrew Wang
  Clarify automatic sync in FSEditLog#logEdit
  

• HDFS-8682. Major bug reported by J.Andreina and fixed by J.Andreina
  Should not remove decommissioned node,while calculating the number of live/dead decommissioned node.
  

• HDFS-8549. Major improvement reported by Andrew Wang and fixed by Andrew Wang (balancer & mover)
  Abort the balancer if an upgrade is in progress
  

• HDFS-8498. Critical bug reported by Daryn Sharp and fixed by Jing Zhao (hdfs-client)
  Blocks can be committed with wrong size
  

• HDFS-8312. Critical bug reported by Eric Yang and fixed by Weiwei Yang (fs , security)
  Trash does not descent into child directories to check for permissions
  

  HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang via Eric Yang)

• HDFS-8307. Trivial bug reported by Anu Engineer and fixed by Andres Perez (hdfs-client)
  Spurious DNS Queries from hdfs shell
  

• HDFS-8224. Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah (datanode)
  Schedule a block for scanning if its metadata file is corrupt
  

• HDFS-8200. Major improvement reported by Haohui Mai and fixed by Haohui Mai
  Refactor FSDirStatAndListingOp
  

• HDFS-8131. Minor improvement reported by Liu Shaohui and fixed by Liu Shaohui (namenode)
  Implement a space balanced block placement policy
  

• HDFS-7990. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  IBR delete ack should not be delayed
  

• HDFS-7933. Major improvement reported by Jitendra Nath Pandey and fixed by Xiaoyu Yao (namenode)
  fsck should also report decommissioning replicas.
  

  The output of hdfs fsck now also contains information about decommissioning replicas.

• HDFS-7847. Major bug reported by Colin P. McCabe and fixed by Charles Lamb
  Modify NNThroughputBenchmark to be able to operate on a remote NameNode
  

• HDFS-6291. Minor bug reported by Ted Yu and fixed by Sanghyun Yun (ha)
  FSImage may be left unclosed in BootstrapStandby#doRun()
  

• HDFS-5802. Trivial bug reported by Harsh J and fixed by Xiao Chen (namenode)
  NameNode does not check for inode type before traversing down a path
  

• HDFS-5042. Critical bug reported by Dave Latham and fixed by Vinayakumar B
  Completed files lost after power failure
  

• HADOOP-14686. Blocker bug reported by Sean Busbey and fixed by Sean Busbey (build , precommit)
  Branch-2.7 .gitignore is out of date
  

• HADOOP-14586. Minor bug reported by Uwe Schindler and fixed by Akira Ajisaka (common)
  StringIndexOutOfBoundsException breaks org.apache.hadoop.util.Shell on 2.7.x with Java 9
  

• HADOOP-14474. Major bug reported by Akira Ajisaka and fixed by Akira Ajisaka (build)
  Use OpenJDK 7 instead of Oracle JDK 7 to avoid oracle-java7-installer failures
  

• HADOOP-14440. Major improvement reported by Eric Badger and fixed by Eric Badger
  Add metrics for connections dropped
  

• HADOOP-14434. Major bug reported by Lukas Majercak and fixed by Lukas Majercak (native)
  Use MoveFileEx to allow renaming a file when the destination exists
  

• HADOOP-14376. Major bug reported by Eli Acherkan and fixed by Eli Acherkan (common , io)
  Memory leak when reading a compressed file using the native library
  

• HADOOP-14374. Major bug reported by lixinglong and fixed by lixinglong
  License error in GridmixTestUtils.java
  

• HADOOP-14371. Major bug reported by hu xiaodong and fixed by hu xiaodong
  License error in TestLoadBalancingKMSClientProvider.java
  

• HADOOP-14356. Blocker bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Update CHANGES.txt to reflect all the changes in branch-2.7
  

• HADOOP-14293. Major bug reported by Andrew Wang and fixed by Andrew Wang (test)
  Initialize FakeTimer with a less trivial value
  

• HADOOP-14276. Minor improvement reported by Erik Krogen and fixed by Erik Krogen (util)
  Add a nanosecond API to Time/Timer/FakeTimer
  

• HADOOP-14214. Critical bug reported by Mingliang Liu and fixed by Mingliang Liu (hdfs-client)
  DomainSocketWatcher::add()/delete() should not self interrupt while looping await()
  

• HADOOP-14211. Major bug reported by Erik Krogen and fixed by Erik Krogen (viewfs)
  FilterFs and ChRootedFs are too aggressive about enforcing "authorityNeeded"
  

• HADOOP-14195. Major bug reported by Vihang Karajgaonkar and fixed by Vihang Karajgaonkar (security)
  CredentialProviderFactory$getProviders is not thread-safe
  

• HADOOP-14169. Minor improvement reported by Erik Krogen and fixed by Erik Krogen (viewfs)
  Implement listStatusIterator, listLocatedStatus for ViewFs
  

• HADOOP-14138. Critical sub-task reported by Steve Loughran and fixed by Steve Loughran (fs/s3)
  Remove S3A ref from META-INF service discovery, rely on existing core-default entry
  

  The classpath implementing the s3a filesystem is now defined in core-default.xml. Attempting to instantiate an S3A filesystem instance using a Configuration instance which has not included the default resorts will fail. Applications should not be doing this anyway, as it will lose other critical configuration options needed by the filesystem.

• HADOOP-14100. Critical bug reported by Vinayakumar B and fixed by Vinayakumar B
  Upgrade Jsch jar to latest version to fix vulnerability in old versions
  

• HADOOP-14066. Critical bug reported by Thejas M Nair and fixed by Akira Ajisaka (common)
  VersionInfo should be marked as public API
  

• HADOOP-14044. Major bug reported by Hrishikesh Gadre and fixed by Hrishikesh Gadre
  Synchronization issue in delegation token cancel functionality
  

• HADOOP-14001. Major bug reported by Akira Ajisaka and fixed by Akira Ajisaka
  Improve delegation token validity checking
  

• HADOOP-13958. Blocker bug reported by Junping Du and fixed by Junping Du
  Bump up release year to 2017
  

• HADOOP-13867. Major bug reported by Vinayakumar B and fixed by Vinayakumar B
  FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations
  

• HADOOP-13839. Minor bug reported by Masatake Iwasaki and fixed by Elek, Marton (documentation , tracing)
  Fix outdated tracing documentation
  

• HADOOP-13812. Blocker bug reported by John Zhuge and fixed by John Zhuge (kms)
  Upgrade Tomcat to 6.0.48
  

  Tomcat 6.0.46 starts to filter weak ciphers. Some old SSL clients may be affected. It is recommended to upgrade the SSL client. Run the SSL client against https://www.howsmyssl.com/a/check to find out its TLS version and cipher suites.

• HADOOP-13804. Minor bug reported by Erik Krogen and fixed by Erik Krogen (metrics)
  MutableStat mean loses accuracy if add(long, long) is used
  

• HADOOP-13782. Major improvement reported by Erik Krogen and fixed by Erik Krogen (metrics)
  Make MutableRates metrics thread-local write, aggregate-on-read
  

• HADOOP-13742. Major improvement reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Expose "NumOpenConnectionsPerUser" as a metric
  

• HADOOP-13670. Blocker task reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Update CHANGES.txt to reflect all the changes in branch-2.7
  

• HADOOP-13655. Major sub-task reported by Steve Loughran and fixed by Steve Loughran (documentation , fs , fs/s3)
  document object store use with fs shell and distcp
  

• HADOOP-13601. Trivial bug reported by Mehran Hassani and fixed by Mehran Hassani
  Fix typo in a log messages of AbstractDelegationTokenSecretManager
  

• HADOOP-13579. Blocker bug reported by Akira Ajisaka and fixed by Tsuyoshi Ozawa
  Fix source-level compatibility after HADOOP-11252
  

• HADOOP-13558. Major bug reported by Alejandro Abdelnur and fixed by Xiao Chen (security)
  UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket
  

• HADOOP-13535. Major bug reported by Wei-Chiu Chuang and fixed by Min Shen
  Add jetty6 acceptor startup issue workaround to branch-2
  

• HADOOP-13512. Critical bug reported by Mingliang Liu and fixed by Mingliang Liu (security)
  ReloadingX509TrustManager should keep reloading in case of exception
  

• HADOOP-13494. Major bug reported by Sean Mackrory and fixed by Sean Mackrory (security)
  ReconfigurableBase can log sensitive information
  

• HADOOP-13433. Major bug reported by Duo Zhang and fixed by Duo Zhang (security)
  Race in UGI.reloginFromKeytab
  

• HADOOP-13362. Blocker bug reported by Jason Lowe and fixed by Junping Du (metrics)
  DefaultMetricsSystem leaks the source name when a source unregisters
  

• HADOOP-13290. Major improvement reported by Konstantin Shvachko and fixed by Jonathan Hung (ipc)
  Appropriate use of generics in FairCallQueue
  

• HADOOP-13236. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  truncate will fail when we use viewfilesystem
  

• HADOOP-13202. Major bug reported by zhengbing li and fixed by Kai Sasaki (util)
  Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes
  

• HADOOP-13201. Major bug reported by Tianyin Xu and fixed by Rakesh R (viewfs)
  Print the directory paths when ViewFs denies the rename operation on internal dirs
  

• HADOOP-13119. Major bug reported by Jeffrey E Rodriguez and fixed by Yuanbo Liu
  Add ability to secure log servlet using proxy users
  

• HADOOP-13026. Critical bug reported by Xuan Gong and fixed by Xuan Gong
  Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator
  

• HADOOP-12991. Major bug reported by Kevin Hogeland and fixed by Kai Sasaki (fs)
  Conflicting default ports in DelegateToFileSystem
  

• HADOOP-12975. Major sub-task reported by Elliott Clark and fixed by Elliott Clark
  Add jitter to CachingGetSpaceUsed's thread
  

• HADOOP-12765. Major bug reported by Min Shen and fixed by Min Shen
  HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections
  

• HADOOP-12668. Critical improvement reported by Vijay Singh and fixed by Vijay Singh (security)
  Support excluding weak Ciphers in HttpServer2 through ssl-server.xml
  

  The Code Changes include following: - Modified DFSUtil.java in Apache HDFS project for supplying new parameter ssl.server.exclude.cipher.list - Modified HttpServer2.java in Apache Hadoop-common project to work with new parameter and exclude ciphers using jetty setExcludeCihers method. - Modfied associated test classes to owrk with existing code and also cover the newfunctionality in junit

• HADOOP-12597. Minor bug reported by huangyitian and fixed by Surendra Singh Lilhore (security)
  In kms-site.xml configuration "hadoop.security.keystore.JavaKeyStoreProvider.password" should be updated with new name
  

• HADOOP-12483. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  Maintain wrapped SASL ordering for postponed IPC responses
  

• HADOOP-12418. Major bug reported by Steve Loughran and fixed by Kihwal Lee (test)
  TestRPC.testRPCInterruptedSimple fails intermittently
  

• HADOOP-12325. Major improvement reported by Anu Engineer and fixed by Anu Engineer (ipc , metrics)
  RPC Metrics : Add the ability track and log slow RPCs
  

• HADOOP-12259. Major improvement reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (test , util)
  Utility to Dynamic port allocation
  

• HADOOP-12173. Major bug reported by Inigo Goiri and fixed by Inigo Goiri
  NetworkTopology#add calls NetworkTopology#toString always
  

• HADOOP-11859. Major bug reported by Eugene Koifman and fixed by Eugene Koifman
  PseudoAuthenticationHandler fails with httpcomponents v4.4
  

• HADOOP-11780. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  Prevent IPC reader thread death
  

• HADOOP-11703. Major bug reported by Abin Shahab and fixed by Abin Shahab
  git should ignore .DS_Store files on Mac OS X
  

• HADOOP-11361. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Fix a race condition in MetricsSourceAdapter.updateJmxCache
  

• HADOOP-11149. Major bug reported by Rajat Jain and fixed by Steve Loughran (test)
  Increase the timeout of TestZKFailoverController
  

• HADOOP-10980. Minor test reported by Ted Yu and fixed by Eric Badger
  TestActiveStandbyElector fails occasionally in trunk
  

• HADOOP-10829. Major bug reported by Benoy Antony and fixed by Benoy Antony (security)
  Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe
  

• HADOOP-10597. Major sub-task reported by Ming Ma and fixed by Ming Ma
  RPC Server signals backoff to clients when all request queues are full
  

  This change introduces a new configuration key used by RPC server to decide whether to send backoff signal to RPC Client when RPC call queue is full. When the feature is enabled, RPC server will no longer block on the processing of RPC requests when RPC call queue is full. It helps to improve quality of service when the service is under heavy load. The configuration key is in the format of "ipc.#port#.backoff.enable" where #port# is the port number that RPC server listens on. For example, if you want to enable the feature for the RPC server that listens on 8020, set ipc.8020.backoff.enable to true.

• HADOOP-10300. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  Allowed deferred sending of call responses
  

• HADOOP-9631. Major bug reported by Lohit Vijayarenu and fixed by Erik Krogen (fs , viewfs)
  ViewFs should use underlying FileSystem's server side defaults
  

• HADOOP-7851. Major bug reported by Amar Kamat and fixed by Uma Maheswara Rao G (conf)
  Configuration.getClasses() never returns the default value.
  

  Fixed Configuration.getClasses() API to return the default value if the key is not set.