Apache Hadoop Changelog

Release 2.7.4 - 2017-08-04

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component Reporter Contributor
HDFS-7933 fsck should also report decommissioning replicas. Major namenode Jitendra Nath Pandey Xiaoyu Yao
HADOOP-13812 Upgrade Tomcat to 6.0.48 Blocker kms John Zhuge John Zhuge
HADOOP-14138 Remove S3A ref from META-INF service discovery, rely on existing core-default entry Critical fs/s3 Steve Loughran Steve Loughran

NEW FEATURES:

JIRA Summary Priority Component Reporter Contributor
HDFS-9804 Allow long-running Balancer to login with keytab Major balancer & mover, security Xiao Chen Xiao Chen
MAPREDUCE-6304 Specifying node labels when submitting MR jobs Major job submission Jian Fang Naganarasimha G R

IMPROVEMENTS:

JIRA Summary Priority Component Reporter Contributor
HDFS-8200 Refactor FSDirStatAndListingOp Major . Haohui Mai Haohui Mai
HDFS-8131 Implement a space balanced block placement policy Minor namenode Liu Shaohui Liu Shaohui
HDFS-8549 Abort the balancer if an upgrade is in progress Major balancer & mover Andrew Wang Andrew Wang
HDFS-8709 Clarify automatic sync in FSEditLog#logEdit Minor . Andrew Wang Andrew Wang
HADOOP-12259 Utility to Dynamic port allocation Major test, util Brahma Reddy Battula Brahma Reddy Battula
HDFS-8883 NameNode Metrics : Add FSNameSystem lock Queue Length Major namenode Anu Engineer Anu Engineer
HDFS-9019 Adding informative message to sticky bit permission denied exception Minor security Thejas M Nair Xiaoyu Yao
HDFS-9145 Tracking methods that hold FSNamesytemLock for too long Major namenode Jing Zhao Mingliang Liu
HDFS-9726 Refactor IBR code to a new class Minor datanode Tsz Wo Nicholas Sze Tsz Wo Nicholas Sze
HADOOP-12668 Support excluding weak Ciphers in HttpServer2 through ssl-server.xml Critical security Vijay Singh Vijay Singh
HDFS-9710 Change DN to send block receipt IBRs in batches Major datanode Tsz Wo Nicholas Sze Tsz Wo Nicholas Sze
HDFS-9412 getBlocks occupies FSLock and takes too long to complete Major . He Tianyi He Tianyi
HDFS-9902 Support different values of dfs.datanode.du.reserved per storage type Major datanode Pan Yuxuan Brahma Reddy Battula
HADOOP-13290 Appropriate use of generics in FairCallQueue Major ipc Konstantin Shvachko Jonathan Hung
YARN-5483 Optimize RMAppAttempt#pullJustFinishedContainers Major . sandflee sandflee
HDFS-10798 Make the threshold of reporting FSNamesystem lock contention configurable Major logging, namenode Zhe Zhang Erik Krogen
HDFS-10807 Doc about upgrading to a version of HDFS with snapshots may be confusing Minor documentation Mingliang Liu Mingliang Liu
HDFS-10625 VolumeScanner to report why a block is found bad Major datanode, hdfs Yongjun Zhang Rushabh S Shah
YARN-5550 TestYarnCLI#testGetContainers should format according to CONTAINER_PATTERN Minor client, test Jonathan Hung Jonathan Hung
HDFS-10817 Add Logging for Long-held NN Read Locks Major logging, namenode Erik Krogen Erik Krogen
YARN-5540 scheduler spends too much time looking at empty priorities Major capacity scheduler, fairscheduler, resourcemanager Nathan Roberts Jason Lowe
YARN-3877 YarnClientImpl.submitApplication swallows exceptions Minor client Steve Loughran Varun Saxena
MAPREDUCE-6741 add MR support to redact job conf properties Major mrv2 Haibo Chen Haibo Chen
HDFS-11069 Tighten the authorization of datanode RPC Major datanode, security Kihwal Lee Kihwal Lee
HADOOP-12325 RPC Metrics : Add the ability track and log slow RPCs Major ipc, metrics Anu Engineer Anu Engineer
HADOOP-13782 Make MutableRates metrics thread-local write, aggregate-on-read Major metrics Erik Krogen Erik Krogen
HDFS-10941 Improve BlockManager#processMisReplicatesAsync log Major namenode Xiaoyu Yao Chen Liang
HADOOP-13742 Expose “NumOpenConnectionsPerUser” as a metric Major . Brahma Reddy Battula Brahma Reddy Battula
HDFS-10534 NameNode WebUI should display DataNode usage histogram Major namenode, ui Zhe Zhang Kai Sasaki
HDFS-11333 Print a user friendly error message when plugins are not found Minor namenode Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-11466 Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms Major namenode Andrew Wang Andrew Wang
HADOOP-14169 Implement listStatusIterator, listLocatedStatus for ViewFs Minor viewfs Erik Krogen Erik Krogen
HDFS-11628 Clarify the behavior of HDFS Mover in documentation Major documentation Xiaobing Zhou Xiaobing Zhou
HADOOP-14276 Add a nanosecond API to Time/Timer/FakeTimer Minor util Erik Krogen Erik Krogen
HDFS-11648 Lazy construct the IIP pathname Major . Daryn Sharp Daryn Sharp
HDFS-11634 Optimize BlockIterator when iterating starts in the middle. Major . Konstantin Shvachko Konstantin Shvachko
HDFS-11384 Add option for balancer to disperse getBlocks calls to avoid NameNode’s rpc.CallQueueLength spike Major balancer & mover yunjiong zhao Konstantin Shvachko
HDFS-8873 Allow the directoryScanner to be rate-limited Major datanode Nathan Roberts Daniel Templeton
YARN-6457 Allow custom SSL configuration to be supplied in WebApps Major webapp, yarn Sanjay M Pujare Sanjay M Pujare
YARN-6493 Print requested node partition in assignContainer logs Major . Jonathan Hung Jonathan Hung
HADOOP-14440 Add metrics for connections dropped Major . Eric Badger Eric Badger
HDFS-11345 Document the configuration key for FSNamesystem lock fairness Minor documentation, namenode Zhe Zhang Erik Krogen

BUG FIXES:

JIRA Summary Priority Component Reporter Contributor
HADOOP-7851 Configuration.getClasses() never returns the default value. Major conf Amar Kamat Uma Maheswara Rao G
YARN-1471 The SLS simulator is not running the preemption policy for CapacityScheduler Minor . Carlo Curino Carlo Curino
HADOOP-11703 git should ignore .DS_Store files on Mac OS X Major . Abin Shahab Abin Shahab
YARN-3269 Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path Major . Xuan Gong Xuan Gong
HDFS-7990 IBR delete ack should not be delayed Major namenode Daryn Sharp Daryn Sharp
HADOOP-11859 PseudoAuthenticationHandler fails with httpcomponents v4.4 Major . Eugene Koifman Eugene Koifman
HDFS-7847 Modify NNThroughputBenchmark to be able to operate on a remote NameNode Major . Colin P. McCabe Charles Lamb
HDFS-6291 FSImage may be left unclosed in BootstrapStandby#doRun() Minor ha Ted Yu Sanghyun Yun
YARN-3707 RM Web UI queue filter doesn’t work Blocker . Wangda Tan Wangda Tan
HADOOP-12173 NetworkTopology#add calls NetworkTopology#toString always Major . Íñigo Goiri Íñigo Goiri
HDFS-8682 Should not remove decommissioned node,while calculating the number of live/dead decommissioned node. Major . J.Andreina J.Andreina
HDFS-5802 NameNode does not check for inode type before traversing down a path Trivial namenode Harsh J Xiao Chen
YARN-4017 container-executor overuses PATH_MAX Major nodemanager Allen Wittenauer Sidharta Seethana
YARN-4250 NPE in AppSchedulingInfo#isRequestLabelChanged Major resourcemanager, scheduler Brahma Reddy Battula Brahma Reddy Battula
HADOOP-12483 Maintain wrapped SASL ordering for postponed IPC responses Critical ipc Daryn Sharp Daryn Sharp
HADOOP-12418 TestRPC.testRPCInterruptedSimple fails intermittently Major test Steve Loughran Kihwal Lee
YARN-4302 SLS not able start due to NPE in SchedulerApplicationAttempt#getResourceUsageReport Major . Bibin A Chundatt Bibin A Chundatt
YARN-4367 SLS webapp doesn’t load Major scheduler-load-simulator Karthik Kambatla Karthik Kambatla
HADOOP-11149 Increase the timeout of TestZKFailoverController Major test Rajat Jain Steve Loughran
HDFS-9467 Fix data race accessing writeLockHeldTimeStamp in FSNamesystem Major namenode Mingliang Liu Mingliang Liu
YARN-4109 Exception on RM scheduler page loading with labels Minor . Bibin A Chundatt Mohammad Shahid Khan
YARN-4612 Fix rumen and scheduler load simulator handle killed tasks properly Major . Ming Ma Ming Ma
YARN-4927 TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler Major test Karthik Kambatla Bibin A Chundatt
YARN-4562 YARN WebApp ignores the configuration passed to it for keystore settings Major . Sergey Shelukhin Sergey Shelukhin
HDFS-10270 TestJMXGet:testNameNode() fails Minor test Andras Bokor Gergely Novák
HADOOP-13026 Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator Critical . Xuan Gong Xuan Gong
HDFS-10276 HDFS should not expose path info that user has no permission to see. Major . Kevin Cox Yuanbo Liu
YARN-5197 RM leaks containers if running container disappears from node update Critical resourcemanager Jason Lowe Jason Lowe
YARN-5262 Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat Major resourcemanager Rohith Sharma K S Rohith Sharma K S
HDFS-10396 Using -diff option with DistCp may get “Comparison method violates its general contract” exception Major . Yongjun Zhang Yongjun Zhang
HDFS-10336 TestBalancer failing intermittently because of not reseting UserGroupInformation completely Major test Yiqun Lin Yiqun Lin
HDFS-10512 VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks Major datanode Wei-Chiu Chuang Yiqun Lin
YARN-5353 ResourceManager can leak delegation tokens when they are shared across apps Critical resourcemanager Jason Lowe Jason Lowe
HADOOP-11361 Fix a race condition in MetricsSourceAdapter.updateJmxCache Major . Brahma Reddy Battula Brahma Reddy Battula
HDFS-10544 Balancer doesn’t work with IPFailoverProxyProvider Major balancer & mover, ha Zhe Zhang Zhe Zhang
HADOOP-13202 Avoid possible overflow in org.apache.hadoop.util.bloom.BloomFilter#getNBytes Major util zhengbing li Kai Sasaki
HADOOP-12991 Conflicting default ports in DelegateToFileSystem Major fs Kevin Hogeland Kai Sasaki
MAPREDUCE-6744 Increase timeout on TestDFSIO tests Major . Eric Badger Eric Badger
HDFS-10691 FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException Major . Yiqun Lin Yiqun Lin
YARN-5121 fix some container-executor portability issues Blocker nodemanager, security Allen Wittenauer Allen Wittenauer
MAPREDUCE-6724 Single shuffle to memory must not exceed Integer#MAX_VALUE Major mrv2 Haibo Chen Haibo Chen
HDFS-8780 Fetching live/dead datanode list with arg true for removeDecommissionNode,returns list with decom node. Major . J.Andreina J.Andreina
YARN-5462 TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently Major . Eric Badger Eric Badger
YARN-5469 Increase timeout of TestAmFilter.testFilter Minor . Eric Badger Eric Badger
HDFS-10716 In Balancer, the target task should be removed when its size < 0. Minor balancer & mover Yiqun Lin Yiqun Lin
HDFS-10715 NPE when applying AvailableSpaceBlockPlacementPolicy Major namenode Guangbin Zhu Guangbin Zhu
YARN-5333 Some recovered apps are put into default queue when RM HA Major . Jun Gong Jun Gong
HDFS-10693 metaSave should print blocks, not LightWeightHashSet Major namenode Konstantin Shvachko Yuanbo Liu
HDFS-10694 BlockManager.processReport() should print blockReportId in each log message. Major logging, namenode Konstantin Shvachko Yuanbo Liu
HDFS-8224 Schedule a block for scanning if its metadata file is corrupt Major datanode Rushabh S Shah Rushabh S Shah
YARN-5382 RM does not audit log kill request for active applications Major resourcemanager Jason Lowe Vrushali C
HDFS-9696 Garbage snapshot records lingering forever Critical . Kihwal Lee Kihwal Lee
HDFS-10747 o.a.h.hdfs.tools.DebugAdmin usage message is misleading Minor hdfs-client Mingliang Liu Mingliang Liu
HADOOP-13494 ReconfigurableBase can log sensitive information Major security Sean Mackrory Sean Mackrory
HADOOP-13512 ReloadingX509TrustManager should keep reloading in case of exception Critical security Mingliang Liu Mingliang Liu
HDFS-10763 Open files can leak permanently due to inconsistent lease update Critical . Kihwal Lee Kihwal Lee
MAPREDUCE-6763 Shuffle server listen queue is too small Major mrv2 Jason Lowe Jason Lowe
HDFS-8312 Trash does not descent into child directories to check for permissions Critical fs, security Eric Yang Weiwei Yang
HDFS-8915 TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins Minor test Anu Engineer Masatake Iwasaki
HADOOP-12765 HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections Major . Min Shen Min Shen
MAPREDUCE-6768 TestRecovery.testSpeculative failed with NPE Major mrv2 Haibo Chen Haibo Chen
MAPREDUCE-4784 TestRecovery occasionally fails Major mrv2, test Jason Lowe Haibo Chen
HDFS-10809 getNumEncryptionZones causes NPE in branch-2.7 Major encryption, namenode Zhe Zhang Vinitha Reddy Gankidi
HADOOP-13558 UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket Major security Alejandro Abdelnur Xiao Chen
HDFS-9038 DFS reserved space is erroneously counted towards non-DFS used. Major datanode Chris Nauroth Brahma Reddy Battula
HADOOP-13579 Fix source-level compatibility after HADOOP-11252 Blocker . Akira Ajisaka Tsuyoshi Ozawa
HADOOP-13601 Fix typo in a log messages of AbstractDelegationTokenSecretManager Trivial . Mehran Hassani Mehran Hassani
HDFS-10879 TestEncryptionZonesWithKMS#testReadWrite fails intermittently Major . Xiao Chen Xiao Chen
HDFS-10843 Update space quota when a UC block is completed rather than committed. Major hdfs, namenode Erik Krogen Erik Krogen
HADOOP-13535 Add jetty6 acceptor startup issue workaround to branch-2 Major . Wei-Chiu Chuang Min Shen
HADOOP-12597 In kms-site.xml configuration “hadoop.security.keystore.JavaKeyStoreProvider.password” should be updated with new name Minor security huangyitian Surendra Singh Lilhore
HDFS-9885 Correct the distcp counters name while displaying counters Minor distcp Archana T Surendra Singh Lilhore
HDFS-10889 Remove outdated Fault Injection Framework documentaion Major documentation Brahma Reddy Battula Brahma Reddy Battula
HDFS-10713 Throttle FsNameSystem lock warnings Major logging, namenode Arpit Agarwal Hanisha Koneru
HDFS-10915 Fix time measurement bug in TestDatanodeRestart#testWaitForRegistrationOnRestart Minor test Xiaobing Zhou Xiaobing Zhou
HDFS-9444 Add utility to find set of available ephemeral ports to ServerSocketUtil Major . Brahma Reddy Battula Masatake Iwasaki
HADOOP-11780 Prevent IPC reader thread death Critical ipc Daryn Sharp Daryn Sharp
MAPREDUCE-6771 RMContainerAllocator sends container diagnostics event after corresponding completion event Major mrv2 Haibo Chen Haibo Chen
HDFS-10878 TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException Major hdfs-client Rushabh S Shah Rushabh S Shah
HDFS-10609 Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications Major encryption Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-10991 Export hdfsTruncateFile symbol in libhdfs Blocker libhdfs Surendra Singh Lilhore Surendra Singh Lilhore
HDFS-11002 Fix broken attr/getfattr/setfattr links in ExtendedAttributes.md Major documentation Mingliang Liu Mingliang Liu
HDFS-10987 Make Decommission less expensive when lot of blocks present. Critical . Brahma Reddy Battula Brahma Reddy Battula
HDFS-10301 BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order Critical namenode Konstantin Shvachko Vinitha Reddy Gankidi
HDFS-10712 Fix TestDataNodeVolumeFailure on 2.* branches. Major . Konstantin Shvachko Vinitha Reddy Gankidi
HDFS-10627 Volume Scanner marks a block as “suspect” even if the exception is network-related Major hdfs Rushabh S Shah Rushabh S Shah
HADOOP-13236 truncate will fail when we use viewfilesystem Major . Brahma Reddy Battula Brahma Reddy Battula
HDFS-11015 Enforce timeout in balancer Major balancer & mover Kihwal Lee Kihwal Lee
HDFS-11053 Unnecessary superuser check in versionRequest() Major namenode, security Kihwal Lee Kihwal Lee
HDFS-10921 TestDiskspaceQuotaUpdate doesn’t wait for NN to get out of safe mode Major . Eric Badger Eric Badger
HADOOP-13201 Print the directory paths when ViewFs denies the rename operation on internal dirs Major viewfs Tianyin Xu Rakesh R
YARN-4328 Findbugs warning in resourcemanager in branch-2.7 and branch-2.6 Minor resourcemanager Varun Saxena Akira Ajisaka
YARN-3432 Cluster metrics have wrong Total Memory when there is reserved memory on CS Major capacityscheduler, resourcemanager Thomas Graves Brahma Reddy Battula
HDFS-9500 datanodesSoftwareVersions map may counting wrong when rolling upgrade Major . Phil Yang Erik Krogen
HDFS-10455 Logging the username when deny the setOwner operation Minor namenode Tianyin Xu Rakesh R
YARN-5001 Aggregated Logs root directory is created with wrong group if nonexistent Major log-aggregation, nodemanager, security Haibo Chen Haibo Chen
YARN-5837 NPE when getting node status of a decommissioned node after an RM restart Major . Robert Kanter Robert Kanter
HADOOP-13804 MutableStat mean loses accuracy if add(long, long) is used Minor metrics Erik Krogen Erik Krogen
HDFS-8307 Spurious DNS Queries from hdfs shell Trivial hdfs-client Anu Engineer Andres Perez
HDFS-11087 NamenodeFsck should check if the output writer is still writable. Major namenode Konstantin Shvachko Erik Krogen
HDFS-11056 Concurrent append and read operations lead to checksum error Major datanode, httpfs Wei-Chiu Chuang Wei-Chiu Chuang
YARN-4355 NPE while processing localizer heartbeat Major nodemanager Jason Lowe Varun Saxena
HDFS-10966 Enhance Dispatcher logic on deciding when to give up a source DataNode Major balancer & mover Zhe Zhang Mark Wagner
HDFS-11174 Wrong HttpFS test command in doc Minor documentation, httpfs John Zhuge John Zhuge
YARN-5694 ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable Critical resourcemanager Daniel Templeton Daniel Templeton
HDFS-11180 Intermittent deadlock in NameNode when failover happens. Blocker namenode Abhishek Modi Akira Ajisaka
HADOOP-13867 FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations Major . Vinayakumar B Vinayakumar B
HDFS-11229 HDFS-11056 failed to close meta file Blocker datanode Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-11160 VolumeScanner reports write-in-progress replicas as corrupt incorrectly Major datanode Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-11263 ClassCastException when we use Bzipcodec for Fsimage compression Critical . Brahma Reddy Battula Brahma Reddy Battula
YARN-6024 Capacity Scheduler ‘continuous reservation looking’ doesn’t work when sum of queue’s used and reserved resources is equal to max Major . Wangda Tan Wangda Tan
YARN-5988 RM unable to start in secure setup Blocker . Ajith S Ajith S
HADOOP-13839 Fix outdated tracing documentation Minor documentation, tracing Masatake Iwasaki Elek, Marton
HDFS-11280 Allow WebHDFS to reuse HTTP connections to NN Major hdfs Zheng Shao Zheng Shao
MAPREDUCE-6711 JobImpl fails to handle preemption events on state COMMITTING Major . Li Lu Prabhu Joseph
HADOOP-13958 Bump up release year to 2017 Blocker . Junping Du Junping Du
HDFS-10733 NameNode terminated after full GC thinking QJM is unresponsive. Major namenode, qjm Konstantin Shvachko Vinitha Reddy Gankidi
HADOOP-14001 Improve delegation token validity checking Major . Akira Ajisaka Akira Ajisaka
HDFS-11352 Potential deadlock in NN when failing over Critical namenode Erik Krogen Erik Krogen
HADOOP-14044 Synchronization issue in delegation token cancel functionality Major . Hrishikesh Gadre Hrishikesh Gadre
HDFS-11377 Balancer hung due to no available mover threads Major balancer & mover yunjiong zhao yunjiong zhao
YARN-6152 Used queue percentage not accurate in UI for 2.7 and below when using DominantResourceCalculator Major . Jonathan Hung Jonathan Hung
HADOOP-13433 Race in UGI.reloginFromKeytab Major security Duo Zhang Duo Zhang
HADOOP-13119 Add ability to secure log servlet using proxy users Major . Jeffrey E Rodriguez Yuanbo Liu
HDFS-11379 DFSInputStream may infinite loop requesting block locations Critical hdfs-client Daryn Sharp Daryn Sharp
YARN-1728 Workaround guice3x-undecoded pathInfo in YARN WebApp Major . Abraham Elmahrek Yuanbo Liu
YARN-6310 OutputStreams in AggregatedLogFormat.LogWriter can be left open upon exceptions Major yarn Haibo Chen Haibo Chen
HDFS-11499 Decommissioning stuck because of failing recovery Major hdfs, namenode Lukas Majercak Lukas Majercak
HADOOP-9631 ViewFs should use underlying FileSystem’s server side defaults Major fs, viewfs Lohit Vijayarenu Erik Krogen
HADOOP-14214 DomainSocketWatcher::add()/delete() should not self interrupt while looping await() Critical hdfs-client Mingliang Liu Mingliang Liu
HADOOP-14195 CredentialProviderFactory$getProviders is not thread-safe Major security Vihang Karajgaonkar Vihang Karajgaonkar
HADOOP-14211 FilterFs and ChRootedFs are too aggressive about enforcing “authorityNeeded” Major viewfs Erik Krogen Erik Krogen
HDFS-11486 Client close() should not fail fast if the last block is being decommissioned Major . Wei-Chiu Chuang Wei-Chiu Chuang
HADOOP-13362 DefaultMetricsSystem leaks the source name when a source unregisters Blocker metrics Jason Lowe Junping Du
MAPREDUCE-6873 MR Job Submission Fails if MR framework application path not on defaultFS Minor mrv2 Erik Krogen Erik Krogen
HDFS-11608 HDFS write crashed with block size greater than 2 GB Critical hdfs-client Xiaobing Zhou Xiaobing Zhou
HADOOP-14066 VersionInfo should be marked as public API Critical common Thejas M Nair Akira Ajisaka
HADOOP-14293 Initialize FakeTimer with a less trivial value Major test Andrew Wang Andrew Wang
YARN-6304 Skip rm.transitionToActive call to RM if RM is already active. Major resourcemanager Rohith Sharma K S Rohith Sharma K S
HDFS-11615 FSNamesystemLock metrics can be inaccurate due to millisecond precision Major hdfs Erik Krogen Erik Krogen
HDFS-11709 StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully Critical ha, namenode Zhe Zhang Erik Krogen
HDFS-11609 Some blocks can be permanently lost if nodes are decommissioned while dead Blocker namenode Kihwal Lee Kihwal Lee
HDFS-8498 Blocks can be committed with wrong size Critical hdfs-client Daryn Sharp Jing Zhao
HDFS-11714 Newly added NN storage directory won’t get initialized and cause space exhaustion Critical . Kihwal Lee Kihwal Lee
HADOOP-14371 License error in TestLoadBalancingKMSClientProvider.java Major . hu xiaodong hu xiaodong
HADOOP-14374 License error in GridmixTestUtils.java Major . lixinglong lixinglong
HDFS-11766 Fix findbugs warning in branch-2.7 Major . Akira Ajisaka Chen Liang
HADOOP-14100 Upgrade Jsch jar to latest version to fix vulnerability in old versions Critical . Vinayakumar B Vinayakumar B
HDFS-11373 Backport HDFS-11258 and HDFS-11272 to branch-2.7 Critical . Akira Ajisaka Akira Ajisaka
HDFS-11795 Fix ASF Licence warnings in branch-2.7 Major . Yiqun Lin Yiqun Lin
YARN-5543 ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread Major capacityscheduler, resourcemanager Min Shen Min Shen
HDFS-11674 reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered Critical datanode Vinayakumar B Vinayakumar B
HADOOP-14376 Memory leak when reading a compressed file using the native library Major common, io Eli Acherkan Eli Acherkan
HADOOP-14434 Use MoveFileEx to allow renaming a file when the destination exists Major native Lukas Majercak Lukas Majercak
HDFS-11849 JournalNode startup failure exception should be logged in log file Major journal-node Surendra Singh Lilhore Surendra Singh Lilhore
HDFS-11864 Document Metrics to track usage of memory for writes Major documentation Brahma Reddy Battula Yiqun Lin
YARN-6615 AmIpFilter drops query parameters on redirect Major . Wilfred Spiegelenburg Wilfred Spiegelenburg
HDFS-11445 FSCK shows overall health stauts as corrupt even one replica is corrupt Critical . Brahma Reddy Battula Brahma Reddy Battula
HDFS-11078 Fix NPE in LazyPersistFileScrubber Major . Íñigo Goiri Íñigo Goiri
HDFS-5042 Completed files lost after power failure Critical . Dave Latham Vinayakumar B
HDFS-11893 Fix TestDFSShell.testMoveWithTargetPortEmpty failure. Major test Konstantin Shvachko Brahma Reddy Battula
HDFS-11741 Long running balancer may fail due to expired DataEncryptionKey Major balancer & mover Wei-Chiu Chuang Wei-Chiu Chuang
HDFS-11856 Ability to re-add Upgrading Nodes (remote) to pipeline for future pipeline updates Major hdfs-client, rolling upgrades Vinayakumar B Vinayakumar B
HADOOP-14474 Use OpenJDK 7 instead of Oracle JDK 7 to avoid oracle-java7-installer failures Major build Akira Ajisaka Akira Ajisaka
YARN-4925 ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression Major . Bibin A Chundatt Bibin A Chundatt
HDFS-11708 Positional read will fail if replicas moved to different DNs after stream is opened Critical hdfs-client Vinayakumar B Vinayakumar B
HDFS-11743 Revert the incompatible fsck reporting output in HDFS-7933 from branch-2.7 Blocker namenode Zhe Zhang Zhe Zhang
HDFS-11583 Parent spans are not initialized to NullScope for every DFSPacket Major tracing Karan Mehta Masatake Iwasaki
YARN-6719 Fix findbugs warnings in SLSCapacityScheduler.java Major . Akira Ajisaka Akira Ajisaka
HDFS-11736 OIV tests should not write outside ‘target’ directory. Major . Konstantin Shvachko Yiqun Lin
MAPREDUCE-6433 launchTime may be negative Major jobhistoryserver, mrv2 Allen Wittenauer zhihai xu
HADOOP-10829 Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe Major security Benoy Antony Benoy Antony
HADOOP-14586 StringIndexOutOfBoundsException breaks org.apache.hadoop.util.Shell on 2.7.x with Java 9 Minor common Uwe Schindler Akira Ajisaka
MAPREDUCE-6697 Concurrent task limits should only be applied when necessary Major mrv2 Jason Lowe Nathan Roberts
YARN-3260 AM attempt fail to register before RM processes launch event Critical resourcemanager Jason Lowe Bibin A Chundatt
YARN-6818 User limit per partition is not honored in branch-2.7 >= Major . Jonathan Hung Jonathan Hung
HADOOP-14356 Update CHANGES.txt to reflect all the changes in branch-2.7 Blocker . Brahma Reddy Battula Brahma Reddy Battula
HDFS-11472 Fix inconsistent replica size after a data pipeline failure Critical datanode Wei-Chiu Chuang Erik Krogen
HDFS-12177 NameNode exits due to setting BlockPlacementPolicy loglevel to Debug Major block placement Jiandan Yang Jiandan Yang
HDFS-11742 Improve balancer usability after HDFS-8818 Blocker . Kihwal Lee Kihwal Lee
HADOOP-14686 Branch-2.7 .gitignore is out of date Blocker build, precommit Sean Busbey Sean Busbey
HDFS-11896 Non-dfsUsed will be doubled on dead node re-registration Blocker . Brahma Reddy Battula Brahma Reddy Battula

TESTS:

JIRA Summary Priority Component Reporter Contributor
HDFS-9888 Allow reseting KerberosName in unit tests Minor . Xiao Chen Xiao Chen
YARN-4717 TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup Minor nodemanager Daniel Templeton Daniel Templeton
YARN-5092 TestRMDelegationTokens fails intermittently Major test Rohith Sharma K S Jason Lowe
HADOOP-10980 TestActiveStandbyElector fails occasionally in trunk Minor . Ted Yu Eric Badger
HDFS-9745 TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts Minor . Xiao Chen Xiao Chen
HDFS-9333 Some tests using MiniDFSCluster errored complaining port in use Minor test Kai Zheng Masatake Iwasaki
HDFS-11290 TestFSNameSystemMBean should wait until JMX cache is cleared Major test Akira Ajisaka Erik Krogen

SUB-TASKS:

JIRA Summary Priority Component Reporter Contributor
HDFS-8721 Add a metric for number of encryption zones Major encryption Rakesh R Rakesh R
HDFS-8824 Do not use small blocks for balancing the cluster Major balancer & mover Tsz Wo Nicholas Sze Tsz Wo Nicholas Sze
HDFS-9421 NNThroughputBenchmark replication test NPE with -namenode option Major benchmarks Xiaoyu Yao Mingliang Liu
YARN-4393 TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently Major test Varun Saxena Varun Saxena
HDFS-9621 getListing wrongly associates Erasure Coding policy to pre-existing replicated files under an EC directory Critical erasure-coding Sushmitha Sreenivasan Jing Zhao
HDFS-9601 NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block Major test Masatake Iwasaki Masatake Iwasaki
YARN-4573 TestRMAppTransitions.testAppRunningKill and testAppKilledKilled fail on trunk Major resourcemanager, test Takashi Ohnishi Takashi Ohnishi
HDFS-9503 Replace -namenode option with -fs for NNThroughputBenchmark Major test Konstantin Shvachko Mingliang Liu
HADOOP-12975 Add jitter to CachingGetSpaceUsed’s thread Major . Elliott Clark Elliott Clark
HDFS-10653 Optimize conversion from path string to components Major hdfs Daryn Sharp Daryn Sharp
HDFS-10656 Optimize conversion of byte arrays back to path string Major hdfs Daryn Sharp Daryn Sharp
HDFS-10674 Optimize creating a full path from an inode Major hdfs Daryn Sharp Daryn Sharp
HDFS-10655 Fix path related byte array conversion bugs Major hdfs Daryn Sharp Daryn Sharp
HDFS-10662 Optimize UTF8 string/byte conversions Major hdfs Daryn Sharp Daryn Sharp
HDFS-10673 Optimize FSPermissionChecker’s internal path usage Major hdfs Daryn Sharp Daryn Sharp
HDFS-10744 Internally optimize path component resolution Major hdfs Daryn Sharp Daryn Sharp
HDFS-10619 Cache path in InodesInPath Major hdfs Daryn Sharp Daryn Sharp
HDFS-10896 Move lock logging logic from FSNamesystem into FSNamesystemLock Major namenode Erik Krogen Erik Krogen
HDFS-10745 Directly resolve paths into INodesInPath Major hdfs Daryn Sharp Daryn Sharp
HADOOP-10597 RPC Server signals backoff to clients when all request queues are full Major . Ming Ma Ming Ma
HADOOP-10300 Allowed deferred sending of call responses Major ipc Daryn Sharp Daryn Sharp
HDFS-10872 Add MutableRate metrics for FSNamesystemLock operations Major namenode Erik Krogen Erik Krogen
HADOOP-13655 document object store use with fs shell and distcp Major documentation, fs, fs/s3 Steve Loughran Steve Loughran
HDFS-8818 Allow Balancer to run faster Major balancer & mover Tsz Wo Nicholas Sze Tsz Wo Nicholas Sze
YARN-4140 RM container allocation delayed incase of app submitted to Nodelabel partition Major scheduler Bibin A Chundatt Bibin A Chundatt

OTHER:

JIRA Summary Priority Component Reporter Contributor
HADOOP-13670 Update CHANGES.txt to reflect all the changes in branch-2.7 Blocker . Brahma Reddy Battula Brahma Reddy Battula
YARN-6274 Documentation refers to incorrect nodemanager health checker interval property Trivial documentation Charles Zhang Weiwei Yang
HDFS-11717 Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully Minor ha, namenode Erik Krogen Erik Krogen