Hadoop 2.7.6 Release Notes

• YARN-7661. Major bug reported by Yang Wang and fixed by Yang Wang
  NodeManager metrics return wrong value after update node resource
  

• YARN-7590. Major improvement reported by Eric Yang and fixed by Eric Yang (security , yarn)
  Improve container-executor validation check
  

• YARN-7249. Blocker bug reported by Wangda Tan and fixed by Wangda Tan
  Fix CapacityScheduler NPE issue when a container preempted while the node is being removed
  

• YARN-6633. Major task reported by Íñigo Goiri and fixed by Íñigo Goiri
  Backport YARN-4167 to branch 2.7
  

• YARN-6632. Major bug reported by Íñigo Goiri and fixed by Íñigo Goiri
  Backport YARN-3425 to branch 2.7
  

• YARN-4167. Minor bug reported by Bibin A Chundatt and fixed by Bibin A Chundatt
  NPE on RMActiveServices#serviceStop when store is null
  

• YARN-3425. Minor bug reported by Bibin A Chundatt and fixed by Bibin A Chundatt (resourcemanager)
  NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed
  

• MAPREDUCE-7052. Major bug reported by Peter Bacsko and fixed by Peter Bacsko (client , test)
  TestFixedLengthInputFormat#testFormatCompressedIn is flaky
  

• MAPREDUCE-7048. Major improvement reported by Peter Bacsko and fixed by Peter Bacsko (mr-am)
  Uber AM can crash due to unknown task in statusUpdate
  

• MAPREDUCE-7028. Blocker bug reported by Gergo Repas and fixed by Gergo Repas (mr-am)
  Concurrent task progress updates causing NPE in Application Master
  

• MAPREDUCE-7020. Major bug reported by Akira Ajisaka and fixed by Peter Bacsko (mr-am)
  Task timeout in uber mode can crash AM
  

• MAPREDUCE-5124. Major bug reported by Jason Lowe and fixed by Peter Bacsko (mr-am)
  AM lacks flow control for task events
  

• HDFS-13195. Minor bug reported by maobaolong and fixed by maobaolong (datanode)
  DataNode conf page cannot display the current value after reconfig
  

• HDFS-13126. Major bug reported by Erik Krogen and fixed by Erik Krogen (datanode , webhdfs)
  Backport [HDFS-7959] to branch-2.7 to re-enable HTTP request logging for WebHDFS
  

• HDFS-13120. Major bug reported by Xiaoyu Yao and fixed by Xiaoyu Yao (namenode , snapshots)
  Snapshot diff could be corrupted after concat
  

• HDFS-13112. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  Token expiration edits may cause log corruption or deadlock
  

• HDFS-12884. Major improvement reported by Konstantin Shvachko and fixed by chencan (namenode)
  BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo
  

• HDFS-12881. Major bug reported by Jason Lowe and fixed by Ajay Kumar
  Output streams closed with IOUtils suppressing write errors
  

• HDFS-12371. Major bug reported by Sai Nukavarapu and fixed by Hanisha Koneru (metrics)
  "BlockVerificationFailures" and "BlocksVerified" show up as 0 in Datanode JMX
  

• HDFS-12347. Critical bug reported by Xiao Chen and fixed by Bharat Viswanadham (test)
  TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently
  

• HDFS-12299. Critical bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Race Between update pipeline and DN Re-Registration
  

• HDFS-11187. Major improvement reported by Wei-Chiu Chuang and fixed by Gabor Bota (datanode)
  Optimize disk access for last partial chunk checksum of Finalized replica
  

• HDFS-11003. Major improvement reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
  Expose "XmitsInProgress" through DataNodeMXBean
  

• HDFS-10453. Major bug reported by He Xiaoqiao and fixed by He Xiaoqiao (namenode)
  ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.
  

• HDFS-7959. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  WebHdfs logging is missing on Datanode
  

• HDFS-4210. Trivial bug reported by Damien Hardy and fixed by John Zhuge (ha , journal-node , namenode)
  Throw helpful exception when DNS entry for JournalNode cannot be resolved
  

• HADOOP-15345. Major improvement reported by He Xiaoqiao and fixed by He Xiaoqiao
  Backport HADOOP-12185 to branch-2.7: NetworkTopology is not efficient adding/getting/removing nodes
  

• HADOOP-15283. Major bug reported by Xiao Chen and fixed by Akira Ajisaka
  Upgrade from findbugs 3.0.1 to spotbugs 3.1.2 in branch-2 to fix docker image build
  

• HADOOP-15279. Minor improvement reported by Allen Wittenauer and fixed by Allen Wittenauer (build , documentation , test)
  increase maven heap size recommendations
  

• HADOOP-15212. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
  Add independent secret manager method for logging expired tokens
  

• HADOOP-15206. Major bug reported by Aki Tanaka and fixed by Aki Tanaka
  BZip2 drops and duplicates records when input split size is small
  

• HADOOP-15177. Blocker task reported by Akira Ajisaka and fixed by Bharat Viswanadham (build)
  Update the release year to 2018
  

• HADOOP-15143. Major bug reported by Jitendra Nath Pandey and fixed by Mukul Kumar Singh
  NPE due to Invalid KerberosTicket in UGI
  

• HADOOP-14246. Major improvement reported by Robert Kanter and fixed by Robert Kanter (security)
  Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
  

• HADOOP-13508. Major bug reported by Atul Sikaria and fixed by Atul Sikaria
  FsPermission string constructor does not recognize sticky bit
  

• HADOOP-13375. Major bug reported by Mingliang Liu and fixed by Weiwei Yang (security , test)
  o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky
  

• HADOOP-13263. Major improvement reported by Stephen O'Donnell and fixed by Stephen O'Donnell
  Reload cached groups in background after expiry
  

  hadoop.security.groups.cache.background.reload can be set to true to enable background reload of expired groups cache entries. This setting can improve the performance of services that use Groups.java (e.g. the NameNode) when group lookups are slow. The setting is disabled by default.

• HADOOP-13105. Major improvement reported by Chris Nauroth and fixed by Mingliang Liu (security)
  Support timeouts in LDAP queries in LdapGroupsMapping.
  

  This patch adds two new config keys for supporting timeouts in LDAP query operations. The property "hadoop.security.group.mapping.ldap.connection.timeout.ms" is the connection timeout (in milliseconds), within which period if the LDAP provider doesn't establish a connection, it will abort the connect attempt. The property "hadoop.security.group.mapping.ldap.read.timeout.ms" is the read timeout (in milliseconds), within which period if the LDAP provider doesn't get a LDAP response, it will abort the read attempt.

• HADOOP-12862. Major bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang
  LDAP Group Mapping over SSL can not specify trust store
  

• HADOOP-12751. Critical bug reported by Bolke de Bruin and fixed by Bolke de Bruin (security)
  While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
  

• HADOOP-12611. Major bug reported by Wei-Chiu Chuang and fixed by Eric Badger
  TestZKSignerSecretProvider#testMultipleInit occasionally fail
  

• HADOOP-12568. Minor improvement reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (documentation)
  Update core-default.xml to describe posixGroups support
  

• HADOOP-12472. Minor improvement reported by Steve Loughran and fixed by Steve Loughran (test)
  Make GenericTestUtils.assertExceptionContains robust
  

• HADOOP-12181. Minor bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki
  Fix intermittent test failure of TestZKSignerSecretProvider
  

• HADOOP-12001. Blocker bug reported by Patrick White and fixed by Patrick White (security)
  Limiting LDAP search conflicts with posixGroup addition
  

• HADOOP-9477. Major new feature reported by Kai Zheng and fixed by Dapeng Sun
  Add posixGroups support for LDAP groups mapping service
  

  Add posixGroups support for LDAP groups mapping service. The change in LDAPGroupMapping is compatible with previous scenario. In LDAP, the group mapping between {{posixAccount}} and {{posixGroup}} is different from the general LDAPGroupMapping, one of the differences is the {{"memberUid"}} will be used to mapping {{posixAccount}} and {{posixGroup}}. The feature will handle the mapping in internal when configuration {{hadoop.security.group.mapping.ldap.search.filter.user}} is set as "posixAccount" and {{hadoop.security.group.mapping.ldap.search.filter.group}} is "posixGroup".