Hadoop 0.22.0 Release Notes

These release notes include new developer and user-facing incompatibilities, features, and major improvements.

Changes since Hadoop 0.21.1

HADOOP-7730. Major test reported by cos and fixed by cos (test)
Allow TestCLI to be run against a cluster

Use the same CLI test to test cluster bits (see HDFS-1762 for more info)
HADOOP-7861. Major improvement reported by shv and fixed by shv (documentation)
changes2html.pl should generate links to HADOOP, HDFS, and MAPREDUCE jiras

changes2html.pl correctly generates links to HADOOP jiras only. This hasn't been updated since projects split.
HADOOP-7786. Major improvement reported by eli and fixed by eli
Remove HDFS-specific configuration keys defined in FsConfig

HADOOP-4952 added a couple HDFS-specific configuration values to common (the block size and the replication factor) that conflict with the HDFS values (eg have the wrong defaults, wrong key name), are not used by common or hdfs and should be removed. After removing these I noticed the rest of FsConfig is only used once outside a test, and isn't tagged as a public API, I think we can remove it entirely.
HADOOP-7783. Major test reported by eli and fixed by eli (fs)
Add more symlink tests that cover intermediate links

This covers the tests for HDFS-2514.
HADOOP-7697. Major bug reported by shv and fixed by shv (build)
Remove dependency on different version of slf4j in avro

Avro upgrade led to a mixture of slf4j versions. Hadoop uses slf4j 1.5.11, and avro brings in 1.6.1
HADOOP-7663. Major bug reported by mayank_bansal and fixed by mayank_bansal (test)
TestHDFSTrash failing on 22

Seems to have started failing recently in many commit builds as well as the last two nightly builds of 22:
https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/51/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testTrashEmptier/

https://issues.apache.org/jira/browse/HDFS-1967
HADOOP-7646. Major bug reported by jrottinghuis and fixed by jrottinghuis (io, ipc)
Make hadoop-common use same version of avro as HBase

HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
When building HBase on top of hadoop, this should be consistent.
HADOOP-7577. Minor bug reported by jrottinghuis and fixed by jrottinghuis (metrics)
TT does not start due to backwards compatibility wrt. EventCounter

Between metrics1 and mertrics2 EventCounter was moved from o.a.h.log to o.a.h.metrics.jvm.
On 0.20-security a wrapper marked with @Deprecated was added back to o.a.h.log for compatibility, the same wrapper exists on trunk, but no on 0.22.

Without it the TT will fail to start with a ClassNotFoundException.
Hive configuration also point to this class in the log4j.properties.
HADOOP-7568. Major bug reported by shv and fixed by zero45 (io)
SequenceFile should not print into stdout

The following line in {{SequenceFile.Reader.initialize()}} should be removed:
System.out.println("Setting end to " + end);
HADOOP-7514. Minor bug reported by jrottinghuis and fixed by jrottinghuis (build)
Build fails with ClassCastException when running both mvn-install and mvn-deploy targets

Although this may not be a common use-case, the exception thrown is really confusing and does not clarify what the problem is.
The resulting error is: java.lang.ClassCastException: org.codehaus.plexus.DefaultPlexusContainer cannot be cast to org.codehaus.plexus.PlexusContainer
The error occurs because mvn-init target gets called twice.
HADOOP-7513. Major bug reported by jrottinghuis and fixed by jrottinghuis (build)
mvn-deploy target fails

When executing mvn-deploy target, the build fails.
hadoop-common and hadoop-common-sources deploy, but the test jar does not.

property staging is not set and/or set to false, meaning when you try to deploy a snapshot build.

The error reads:
Invalid reference: 'hadoop.core.test'.
HADOOP-7457. Blocker improvement reported by jghoman and fixed by jghoman (documentation)
Remove out-of-date Chinese language documentation

The Chinese language documentation haven't been updated (other than copyright years and svn moves) since their original contribution several years ago. Worse than no docs is out-of-date, wrong docs. We should delete them from the source tree.
HADOOP-7450. Blocker bug reported by jnp and fixed by cos (build)
Bump jetty to 6.1.26

Bump the jetty version, as previous version has an issue that can cause it to hang at startup.

6.1.14 jetty is also tends to hung on heavy datanode loads.
HADOOP-7390. Minor bug reported by tgraves and fixed by tlipcon (build)
VersionInfo not generated properly in git after unsplit

The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:

@HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")

The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.
HADOOP-7358. Minor improvement reported by tlipcon and fixed by tlipcon (ipc)
Improve log levels when exceptions caught in RPC handler

When a server implementation throws an exception handling an RPC, the Handler thread catches it and logs it before responding with the exception over the IPC channel. This is currently done at INFO level.

I'd like to propose that, if the exception is an unchecked exception, it should be logged at WARN level instead. This can help alert operators when they might be hitting some kind of bug.
HADOOP-7355. Major improvement reported by stack and fixed by stack
Add audience and stability annotations to HttpServer class

HttpServer has at least one subclasser in HBase. Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351
HADOOP-7351. Major bug reported by stack and fixed by stack
Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461

It USED to be protected rather than private but its access was changed by HADOOP-6461. It did the following:

- protected String getWebAppsPath() throws IOException {
- URL url = getClass().getClassLoader().getResource("webapps");
+ private String getWebAppsPath(String appName) throws FileNotFoundException {
+ URL url = getClass().getClassLoader().getResource("webapps/" + appName);
...

HBase subclasses HttpServer providing its UI. This change makes it so we can no longer do so.

This change made it into 0.21. I'd like to get a fix committed to 0.22 as well as TRUNK.
HADOOP-7349. Major bug reported by tlipcon and fixed by tlipcon (ipc, test)
HADOOP-7121 accidentally disabled some tests

When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.
HADOOP-7346. Major improvement reported by tlipcon and fixed by tlipcon (ipc)
Send back nicer error to clients using outdated IPC version

When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".

Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.
HADOOP-7335. Minor improvement reported by tlipcon and fixed by tlipcon (build, test)
Force entropy to come from non-true random for tests

Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.

We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.
HADOOP-7325. Minor improvement reported by brocknoland and fixed by brocknoland (scripts)
hadoop command - do not accept class names starting with a hyphen

The hadoop command does not appear to advertise allowing JVM options before the classname.
HADOOP-7318. Critical bug reported by tlipcon and fixed by tlipcon (io)
MD5Hash factory should reset the digester it returns

Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.
HADOOP-7312. Minor bug reported by tlipcon and fixed by qwertymaniac (conf)
core-default.xml lists configuration version as 0.21

This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.

Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.
HADOOP-7302. Major bug reported by asrabkin and fixed by asrabkin (documentation)
webinterface.private.actions should not be in common

The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker.

I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)
HADOOP-7300. Major bug reported by tlipcon and fixed by tlipcon (conf)
Configuration methods that return collections are inconsistent about mutability

In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.

IMO we should always return mutable collections since these methods by definition are doing copies.
HADOOP-7298. Major test reported by tlipcon and fixed by tlipcon (test)
Add test utility for writing multi-threaded tests

A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.

In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.
HADOOP-7296. Minor bug reported by sseth and fixed by sseth (fs)
The FsPermission(FsPermission) constructor does not use the sticky bit

The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.
HADOOP-7291. Major task reported by eli and fixed by eli
Update Hudson job not to run test-contrib

The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.
HADOOP-7287. Blocker bug reported by tlipcon and fixed by atm (conf)
Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools

For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
- JVM starts
- GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
- DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
- Some class calls conf.get("new key") and sees the default instead of the version set on the command line
HADOOP-7283. Blocker task reported by tomwhite and fixed by tomwhite (build)
Include 32-bit and 64-bit native libraries in Jenkins tarball builds

The job at https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/ is building tarballs, but they do not currently include both 32-bit and 64-bit native libraries. We should update/duplicate hadoop-nighly/hudsonBuildHadoopRelease.sh to support post-split builds.
HADOOP-7252. Minor bug reported by ponyloky and fixed by qwertymaniac (build, conf, test)
JUnit shows up as a compile time dependency
HADOOP-7245. Major bug reported by tomwhite and fixed by tomwhite
FsConfig should use constants in CommonConfigurationKeys

In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.
HADOOP-7244. Blocker improvement reported by tomwhite and fixed by tomwhite (documentation)
Documentation change for updated configuration keys

Common counterpart of HDFS-671.
HADOOP-7241. Minor improvement reported by weiyj and fixed by weiyj (fs, test)
fix typo of command 'hadoop fs -help tail'

Fix the typo of command 'hadoop fs -help tail'.

$ hadoop fs -help tail
-tail [-f] <file>: Show the last 1KB of the file.
The -f option shows apended data as the file grows.

The "apended data" should be "appended data".
HADOOP-7229. Major bug reported by atm and fixed by atm (security)
Absolute path to kinit in auto-renewal thread

In the auto-renewal thread for Kerberos credentials in {{UserGroupInformation}}, the path to {{kinit}} is defaulted to {{/usr/kerberos/bin/kinit}}. This is the default path to {{kinit}} on RHEL/CentOS for MIT krb5, but not on Debian/Ubuntu (and perhaps others OSes.)
HADOOP-7192. Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (documentation)
fs -stat docs aren't updated to reflect the format features

The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).

Like 'fs -help', the docs must also reflect the latest available features.

I shall attach a doc-fix patch shortly.

If anyone has other discrepancies to point out in the web version of the guide, please do so :)
HADOOP-7189. Minor improvement reported by tlipcon and fixed by yuzhihong@gmail.com (security)
Add ability to enable 'debug' property in JAAS configuration

Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.
HADOOP-7187. Major bug reported by umamaheswararao and fixed by umamaheswararao (metrics)
Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext

Init method is creating DatagramSocket. But this is not closed any where.
HADOOP-7184. Major bug reported by tlipcon and fixed by tlipcon (documentation, filecache)
Remove deprecated local.cache.size from core-default.xml

MAPREDUCE-2379 documents the new name of this parameter (mapreduce.tasktracker.cache.local.size) in mapred-default.xml where it belongs.
HADOOP-7172. Critical bug reported by tlipcon and fixed by tlipcon (io, security)
SecureIO should not check owner on non-secure clusters that have no native support

The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.

Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.
HADOOP-7156. Critical bug reported by tlipcon and fixed by tlipcon
getpwuid_r is not thread-safe on RHEL6

Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):

https://fedorahosted.org/sssd/ticket/640

This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:

*** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3575675676]
/lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
/lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]
HADOOP-7154. Minor improvement reported by tlipcon and fixed by tlipcon (scripts)
Should set MALLOC_ARENA_MAX in hadoop-env.sh

New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.

Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.
HADOOP-7146. Major bug reported by tlipcon and fixed by tlipcon
RPC server leaks file descriptors

Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.
HADOOP-7145. Major bug reported by tlipcon and fixed by tlipcon
Configuration.getLocalPath should trim whitespace from the provided directories

MR and HDFS use the Configuration.getTrimmedStrings API for local directory lists, but in a few places also use Configuration.getLocalPath. The former API trims whitespace around each entry in the list, but the latter doesn't. This can cause some subtle problems - the latter API should be fixed to also trim the directory names.
HADOOP-7140. Critical bug reported by tlipcon and fixed by tlipcon
IPC Reader threads do not stop when server stops

After HADOOP-6713, the new IPC "Reader" threads are not properly stopped when the server shuts down. One repercussion of this is that conditions that are supposed to shut down a daemon no longer work (eg the TT doesn't shut itself down if it detects an incompatible build version)
HADOOP-7137. Major task reported by nidaley and fixed by nidaley
Remove hod contrib

As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will
svn remove common/trunk/src/contrib/hod
using this Jira.
HADOOP-7134. Major improvement reported by rvs and fixed by rvs (build)
configure files that are generated as part of the released tarball need to have executable bit set

Currently the configure files that are packaged in a tarball are -rw-rw-r--
HADOOP-7126. Major bug reported by pocheung and fixed by pocheung
TestDFSShell fails in trunk

TestDFSShell.testFilePermissions() fails on an assert in Windows. This originated from HDFS-1084 but the fix is in Common.

junit.framework.ComparisonFailure: null expected:<rwxr[w----]> but was:<rwxr[-xr-x]>
at junit.framework.Assert.assertEquals(Assert.java:81)
at junit.framework.Assert.assertEquals(Assert.java:87)
at org.apache.hadoop.hdfs.TestDFSShell.confirmPermissionChange(TestDFSShell.java:836)
at org.apache.hadoop.hdfs.TestDFSShell.testChmod(TestDFSShell.java:777)
at org.apache.hadoop.hdfs.TestDFSShell.testFilePermissions(TestDFSShell.java:856)
HADOOP-7122. Critical bug reported by tlipcon and fixed by tlipcon (util)
Timed out shell commands leak Timer threads

When a shell command times out, the TimerThread used to cause the timeout is leaked.
HADOOP-7121. Critical bug reported by tlipcon and fixed by tlipcon (ipc)
Exceptions while serializing IPC call response are not handled well

We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.
HADOOP-7118. Blocker bug reported by tlipcon and fixed by tlipcon
NPE in Configuration.writeXml

In HADOOP-7082 I stupidly introduced a regression whereby Configuration.writeXml will throw an NPE if it is called before any .get() call is made, since the properties member is not initialized. This is causing a failure in TestCapacitySchedulerWithJobTracker on my box, but doesn't appear to trigger any failures in the non-contrib tests since .get() is usually called first.

This JIRA is to fix the bug and add a unit test for writeXml in common (apparently it never had a unit test)
HADOOP-7110. Major improvement reported by tlipcon and fixed by tlipcon (io, native)
Implement chmod with JNI

MapReduce is currently using a race-prone workaround to approximate chmod() because forking chmod is too expensive. This race is causing build failures (and probably task failures too). We should implement chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races.
HADOOP-7106. Blocker improvement reported by nidaley and fixed by tlipcon (build)
Re-organize hadoop subversion layout

As discussed on general@ at http://tinyurl.com/4q6lhxm
HADOOP-7104. Major bug reported by kzhang and fixed by kzhang (ipc, security)
Remove unnecessary DNS reverse lookups from RPC layer

RPC connection authorization needs to verify client's Kerberos principal name matches what specified for the protocol. For service clients like DN's, their Kerberos principal names can be specified in the form of "datanode/_HOST@DOMAIN.COM". To get the expected
client principal name, the server needs to substitute "_HOST" with the client's fully qualified domain name, which requires a reverse DNS lookup from client IP address. However, for connections from clients whose principal name are either unspecified or specified not using the "_HOST" convention, the substitution is not required and the reverse DNS lookup should be avoided. Currently the reverse DNS lookup is done for all clients, which could slow services like NN down, when local named cache is not available.
HADOOP-7102. Major bug reported by shv and fixed by shv (conf)
Remove "fs.ramfs.impl" field from core-deafult.xml

"fs.ramfs.impl" used to be configuration parameter for InMemoryFileSystem, which was deprecated in 0.18 (HADOOP-3501) and removed in 0.21 (HADOOP-4648). Configuration should have been cleaned up at the time.
HADOOP-7101. Blocker bug reported by tlipcon and fixed by tlipcon (security)
UserGroupInformation.getCurrentUser() fails when called from non-Hadoop JAAS context

If a Hadoop client is run from inside a container like Tomcat, and the current AccessControlContext has a Subject associated with it that is not created by Hadoop, then UserGroupInformation.getCurrentUser() will throw NoSuchElementException, since it assumes that any Subject will have a hadoop User principal.
HADOOP-7100. Blocker bug reported by tlipcon and fixed by tlipcon (build, contrib/cloud)
Build broken by HADOOP-6811

The commit of HADOOP-6811 removed the ec2 contrib but didn't update build.xml, which references some of these files from the packaging targets. So, the hudson build is currently broken.
HADOOP-7097. Blocker bug reported by nwatkins and fixed by nwatkins (build, native)
java.library.path missing basedir

My Hadoop installation is having trouble loading the native code library. It appears from the log below that java.library.path is missing the basedir in its path. The libraries are built, and present in the directory shown below (relative to hadoop-common directory). Instead of seeing:

/build/native/Linux-amd64-64/lib

I would expect to see:

/path/to/hadoop-common/build/native/Linux-amd64-64/lib

I'm working in branch-0.22.

2011-01-10 17:09:27,695 DEBUG org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2011-01-10 17:09:27,695 DEBUG org.apache.hadoop.util.NativeCodeLoader: java.library.path=/build/native/Linux-amd64-64/lib
2011-01-10 17:09:27,695 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HADOOP-7094. Blocker bug reported by tlipcon and fixed by cos (build)
hadoop.css got lost during project split

hadoop.css no longer exists in common or HDFS, so the web UIs look pretty ugly. The HTML still refers to this file, it's just gone.
HADOOP-7093. Major bug reported by tlipcon and fixed by tlipcon (io)
Servlets should default to text/plain

In trunk the servlets like /stacks and /metrics are returning text/html content-type instead of text/plain. Security wise it's much safer to default to text/plain and require servlets to explicitly set the content-type to text/html when required.
HADOOP-7091. Major bug reported by kzhang and fixed by kzhang (security)
reloginFromKeytab() should happen even if TGT can't be found

HADOOP-6965 introduced a getTGT() method and prevents reloginFromKeytab() from happening when TGT is not found. This results in the RPC layer not being able to refresh TGT after TGT expires. The reason is RPC layer only does relogin when the expired TGT is used and an exception is thrown. However, when that happens, the expired TGT will be removed from Subject. Therefore, getTGT() will return null and relogin will not be performed. We observed, for example, JT will not be able to re-connect to NN after TGT expires.
HADOOP-7089. Minor bug reported by eli and fixed by eli (scripts)
Fix link resolution logic in hadoop-config.sh

The link resolution logic in bin/hadoop-config.sh fails when when executed via a symlink, from the root directory. We can replace this logic with cd -P and pwd -P, which should be portable across Linux, Solaris, BSD, and OSX.
HADOOP-7087. Blocker bug reported by tlipcon and fixed by tlipcon (io)
SequenceFile.createWriter ignores FileSystem parameter

The SequenceFile.createWriter methods that take a FileSystem ignore this parameter after HADOOP-6856. This is causing some MR tests to fail and is a breaking change when users pass unqualified paths to these calls.
HADOOP-7082. Critical bug reported by tlipcon and fixed by tlipcon (conf)
Configuration.writeXML should not hold lock while outputting

Common side of HDFS-1542
HADOOP-7070. Critical bug reported by tlipcon and fixed by tlipcon (security)
JAAS configuration should delegate unknown application names to pre-existing configuration

As reported here: https://issues.cloudera.org/browse/DISTRO-66 it is impossible to use secured Hadoop inside an application that relies on other JAAS configurations. This is because the static initializer of UserGroupInformation replaces the JAAS configuration, but we don't delegate unknown applications up to whatever Configuration was installed previously. The delegation technique seems to be used by JBoss's XMLLoginConfigImpl for example.
HADOOP-7068. Major bug reported by vicaya and fixed by vicaya
Ivy resolve force mode should be turned off by default

The problem is introduced by HADOOP-6486. Which have caused a lot of mysterious artifact issues (unable to downgrade or do parallel dev, without wiping out both m2 and ivy caches etc.) wasting countless hours of dev (many people's) time to track down the issue.
HADOOP-7057. Minor bug reported by cos and fixed by cos (util)
IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message

throw new IOException( "Premeture EOF from inputStream");
HADOOP-7054. Major improvement reported by sanjay.radia and fixed by sanjay.radia
Change NN LoadGenerator to use the new FileContext api
HADOOP-7046. Blocker bug reported by nidaley and fixed by pocheung (security)
1 Findbugs warning on trunk and branch-0.22

There is 1 findbugs warnings on trunk. See attached html file. This must be fixed or filtered out to get back to 0 warnings. The OK_FINDBUGS_WARNINGS property in src/test/test-patch.properties should also be set to 0 in the patch that fixes this issue.
HADOOP-7038. Minor bug reported by gnawux and fixed by gnawux (build)
saveVersion script includes an additional \r while running whoami under windows

I built common under windows occasionally, and found it failed because the 'user' in build/src/o/a/h/package-info.java is "myhostmyname^M".
It seems because the whoami of windows give a string with '\n\r' rather than '\n' only. thus I add an additional tr for it to eliminate the problem.
Since only windows would generate '\n\r' output, I think it won't harm to any other platform.
HADOOP-7034. Major test reported by eli and fixed by eli (fs)
Add TestPath tests to cover dot, dot dot, and slash normalization

Add tests for the current path normalization for dot, dot dot, and slash in TestPath (from HDFS-836).
HADOOP-7032. Major improvement reported by eli and fixed by eli (fs)
Assert type constraints in the FileStatus constructor

A FileStatus may represent a file, directory or symlink. This is indicated using the isdir and symlink members, let's add an assert that validates the contstraints on these members (eg a directory may not have the symlink member set). We could also verify this by having more than one constructor but we don't statically know the type of the file status when we create it.
HADOOP-7028. Minor bug reported by patrickangeles and fixed by patrickangeles (build)
ant eclipse does not include requisite ant.jar in the classpath

RccTask imports classes from ant.jar
Importing the project into Eclipse results in these classes not being found, because ant.jar is not in the classpath.
Note that this patch requires that ANT_HOME be set, but this is consistent with the documentation as per
http://wiki.apache.org/hadoop/EclipseEnvironment
HADOOP-7024. Major test reported by kzhang and fixed by kzhang (test)
Create a test method for adding file systems during tests.

It allows a (mocked) filesystem object to be added to cache for testing purposes. This is used by HDFS-1187.
HADOOP-7013. Major improvement reported by pkling and fixed by pkling
Add boolean field isCorrupt to BlockLocation

This is needed to allow DFSClient.getBlockLocations to notify the calling application when returning a BlockLocation that corresponds to a corrupt block. Currently, this happens when there are no uncorrupted replicas of a requested block.
HADOOP-7011. Major bug reported by atm and fixed by atm
KerberosName.main(...) throws NPE

The main method of KerberosName attempts to do short name translation before calling KerberosName.setConfiguration(...).
HADOOP-7010. Minor improvement reported by yaojingguo and fixed by yaojingguo (fs)
Typo in FileSystem.java

For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".
HADOOP-7009. Major improvement reported by hairong and fixed by hairong (io)
MD5Hash provides a public factory method that creates an instance of MessageDigest

MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.
HADOOP-7008. Major improvement reported by nidaley and fixed by gkesavan (test)
Enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

test-patch.sh should be able to accept a properties file containing an acceptable number of findbugs and javadoc warnings.
HADOOP-7007. Major improvement reported by gkesavan and fixed by gkesavan (build)
update the hudson-test-patch target to work with the latest test-patch script.

The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005
HADOOP-7006. Major bug reported by cnauroth and fixed by cnauroth (fs)
hadoop fs -getmerge does not work using codebase from trunk.

Running the codebase from trunk, the hadoop fs -getmerge command does not work. As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files. It would merge all files into a single file on the local file system. Running the same command using the codebase from trunk, it looks like nothing happens.
HADOOP-7005. Major improvement reported by nidaley and fixed by nidaley (test)
Update test-patch.sh to remove callback to Hudson master
HADOOP-6996. Major new feature reported by hairong and fixed by hairong (io)
Allow CodecFactory to return a codec object given a codec' class name

CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.

This is required by HDFS-1435, where
1. it allows an HDFS admin to configure which codec to use to save an image.
2. It stores the codec class name in its on-disk image instead of a file's suffix.

When saving and reading an image, I'd like to get an codec from CodecFactory by its class name.
HADOOP-6991. Minor bug reported by chris.douglas and fixed by chris.douglas
SequenceFile::Reader discards length for files, does not call openFile

While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.
HADOOP-6989. Major bug reported by jghoman and fixed by chris.douglas
TestSetFile is failing on trunk

Testsuite: org.apache.hadoop.io.TestSetFile
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
------------- Standard Output ---------------
2010-10-04 16:32:01,030 INFO io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
2010-10-04 16:32:01,249 INFO io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
2010-10-04 16:32:01,350 INFO io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
------------- ---------------- ---------------

Testcase: testSetFile took 0.964 sec
Caused an ERROR
key class or comparator option must be set
java.lang.IllegalArgumentException: key class or comparator option must be set
at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)
HADOOP-6987. Major improvement reported by jghoman and fixed by jghoman (test)
Use JUnit Rule to optionally fail test cases that run more than 10 seconds

Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run. This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.
HADOOP-6985. Minor improvement reported by rvadali and fixed by rvadali
Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template

For an administrator who wants to customize HADOOP_OPTS, it would be better to have

# if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi

instead of

# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
HADOOP-6984. Minor bug reported by chris.douglas and fixed by chris.douglas (io)
NPE from SequenceFile::Writer.CompressionCodecOption

The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}
HADOOP-6978. Blocker new feature reported by tlipcon and fixed by tlipcon (io, native, security)
Add JNI support for secure IO operations

In support of MAPREDUCE-2096, we need to add some JNI functionality. In particular, we need the ability to use fstat() on an open file stream, and to use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT.
HADOOP-6977. Major improvement reported by cos and fixed by cos (test)
Herriot daemon clients should vend statistics

The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.

The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.
HADOOP-6975. Major bug reported by pkling and fixed by pkling
integer overflow in S3InputStream for blocks > 2GB

S3InputStream has the same integer overflow issue as DFSInputStream (fixed in HDFS-96).
HADOOP-6970. Major bug reported by shv and fixed by boryas (build)
SecurityAuth.audit should be generated under /build

SecurityAuth.audit is generated under currently root project directory whenever I run anything, and is not being cleaned up by the clean target. It should be created under build directory instead.
HADOOP-6965. Major bug reported by jnp and fixed by jnp
Method in UGI to get Kerberos ticket.

The getTGT method in AutoRenewal thread is moved to the outer UGI class. It is still a private method but can be used by reloginFromKeyTab to check for TGT expiry. This jira covers common changes for HDFS-1364
HADOOP-6951. Major bug reported by atm and fixed by atm (security)
Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies

Because the protocol -> ACL mapping in ServiceAuthorizationManager is static, services which are run in the same JVM have the potential to clobber the other's service authorization ACLs whenever ServiceAuthorizationManager.refresh() is called. This causes authorization failures if one tries to launch a 2NN connected to a minicluster with hadoop.security.authorization enabled. Seems like each service should have its own instance of a ServiceAuthorizationManager, instead of using static methods.
HADOOP-6950. Trivial improvement reported by philip and fixed by philip (scripts)
Suggest that HADOOP_CLASSPATH should be preserved in hadoop-env.sh.template

HADOOP_CLASSPATH tends to be used to add to bin/hadoop's classpath. Because of the way the comment is written, administrator's who customize hadoop-env.sh often inadvertently disable user's abilities to use it, by not including the present value of the variable.

I propose we change the commented out suggestion code to include the present value.

# Extra Java CLASSPATH elements. Optional.
-# export HADOOP_CLASSPATH=
+# export HADOOP_CLASSPATH="<extra_entries>:$HADOOP_CLASSPATH"
HADOOP-6949. Major improvement reported by navis and fixed by mattf (io)
Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting

Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n

It would not be needed to specify each element types for primitive arrays.
HADOOP-6947. Major bug reported by tlipcon and fixed by tlipcon (security)
Kerberos relogin should set refreshKrb5Config to true

In working on securing a daemon that uses two different principals from different threads, I found that I wasn't able to login from a second keytab after I'd logged in from the first. This is because we don't set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won't switch over to the correct keytab file if it's different than the first.
HADOOP-6943. Major improvement reported by atm and fixed by atm (security)
The GroupMappingServiceProvider interface should be public

The GroupMappingServiceProvider interface is presently package-protected. It seems likely that many organizations will be implementing their own versions of this to suit their particular setup. It would be nice if this interface were made public, and annotated with "@InterfaceAudience.Private" and "@InterfaceStability.Evolving".
HADOOP-6940. Minor bug reported by tomwhite and fixed by tomwhite (fs)
RawLocalFileSystem's markSupported method misnamed markSupport

It should be named markSupported to override the method defined in InputStream. Since it doesn't change the default no harm is done.
HADOOP-6938. Major bug reported by kzhang and fixed by kzhang (ipc, security)
ConnectionId.getRemotePrincipal() should check if security is enabled

When security is not enabled, getRemotePrincipal() should return null, which means the Kerberos principal of the remote server is ignored. This bug was caught by TestCLI on Yahoo 20S branch.
HADOOP-6933. Minor bug reported by tlipcon and fixed by tlipcon (test)
TestListFiles is flaky

TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.
HADOOP-6932. Major bug reported by boryas and fixed by boryas
Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"

NameNode.initialize() calls login() method even when security set to simple
HADOOP-6930. Major bug reported by sharadag and fixed by sharadag (ipc)
AvroRpcEngine doesn't work with generated Avro code

AvroRpcEngine uses 'reflect' based java implementation. There should be a way to have it work with 'specific' (generated code from avro idl).
HADOOP-6926. Minor bug reported by tlipcon and fixed by tlipcon (io)
SocketInputStream incorrectly implements read()

SocketInputStream's read() implementation doesn't upcast to int correctly, so it can't read bytes > 0x80. This is the same bug as HADOOP-6925, but in a different spot.
HADOOP-6922. Major bug reported by ravidotg and fixed by ravidotg (documentation, security)
COMMON part of MAPREDUCE-1664

MAPREDUCE-1664 changes the behavior of queue acls and job acls. This needs documentation changes to cluster_setup.xml and a small change in AccessControlList.java
HADOOP-6913. Major bug reported by kzhang and fixed by kzhang (security)
Circular initialization between UserGroupInformation and KerberosName

If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.
HADOOP-6911. Minor improvement reported by boryas and fixed by
doc update for DelegationTokenFetcher (part of HDFS-1036)
HADOOP-6907. Major bug reported by kzhang and fixed by kzhang (ipc, security)
Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal

Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).
HADOOP-6906. Major bug reported by vinodkv and fixed by vinodkv (fs)
FileContext copy() utility doesn't work with recursive copying of directories.
HADOOP-6905. Major improvement reported by kzhang and fixed by kzhang (security)
Better logging messages when a delegation token is invalid

From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.
HADOOP-6903. Major improvement reported by sanjay.radia and fixed by sanjay.radia
Make AbstractFileSystem's methods public to allow filter-Fs like implementions in a differnt package than fs

Make AbstractFileSystem's methods public to allow filter-Fs like implementions in a differnt package than fs
HADOOP-6900. Major bug reported by sureshms and fixed by hairong
FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions

HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.
HADOOP-6899. Major bug reported by sanjay.radia and fixed by sanjay.radia (fs)
RawLocalFileSystem#setWorkingDir() does not work for relative names

RawLocalFileSystem#setWorkingDir() does not work for relative names
HADOOP-6898. Blocker bug reported by tlipcon and fixed by atm (fs, security)
FileSystem.copyToLocal creates files with 777 permissions

FileSystem.copyToLocal ends up calling through to FileUtil.copy, which calls create() on the target file system without passing any permission object. Therefore, the file ends up getting created locally with 777 permissions, which is dangerous -- even if the caller then fixes up permissions afterwards, it exposes a window in which an attacker can open the file.
HADOOP-6892. Major new feature reported by jghoman and fixed by jghoman (security)
Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)

HDFS-1150 will have changes to the start-up scripts and HttpServer. These are handled here.
HADOOP-6890. Major improvement reported by hairong and fixed by hairong (fs)
Improve listFiles API introduced by HADOOP-6870

This jira is mainly for addressing Suresh's review comments for HADOOP-6870:

1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
3. FileSystem.java
* listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
* listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
4. TestListFiles.java
* testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.
HADOOP-6889. Major new feature reported by hairong and fixed by johnvijoe (ipc)
Make RPC to have an option to timeout

Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.

But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.

I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.
HADOOP-6888. Major bug reported by kzhang and fixed by kzhang (fs, security)
Being able to close all cached FileSystem objects for a given UGI

This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.
HADOOP-6885. Major bug reported by eli and fixed by eli (security)
Fix java doc warnings in Groups and RefreshUserMappingsProtocol

There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.
HADOOP-6884. Major improvement reported by zasran and fixed by zasran
Add LOG.isDebugEnabled() guard for each LOG.debug("...")

Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).
HADOOP-6879. Major improvement reported by iyappans and fixed by cos (build, test)
Provide SSH based (Jsch) remote execution API for system tests

http://mvnrepository.com/
com.jcraft � jsch
0.1.42 version needs to be included in the build. This is needed to facilitate implementation of some system (Herriot) testcases .

Please include this in ivy.

jsch is originally located in http://www.jcraft.com/jsch/
HADOOP-6877. Major improvement reported by kzhang and fixed by kzhang (ipc)
Common part of HDFS-1178

This is the Common part of HDFS-1178.
HADOOP-6873. Major bug reported by boryas and fixed by boryas
using delegation token over hftp for long running clients (part of hdfs 1296)
HADOOP-6870. Major new feature reported by hairong and fixed by hairong (fs)
Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations

This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.
HADOOP-6862. Major improvement reported by amareshwari and fixed by amareshwari (security)
Add api to add user/group to AccessControlList

Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList
HADOOP-6861. Major improvement reported by jnp and fixed by jnp
Method in Credentials to read and write a token storage file.

The jira covers the changes in common corresponding to MAPREDUCE-1566.

This jira adds new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.
HADOOP-6859. Major improvement reported by sureshms and fixed by sureshms
Introduce additional statistics to FileSystem

Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.
HADOOP-6856. Major improvement reported by owen.omalley and fixed by owen.omalley (io)
SequenceFile and MapFile need cleanup to remove redundant constructors

Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.

All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile.

I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.
HADOOP-6853. Major bug reported by jghoman and fixed by jghoman
Common component of HDFS-1045

HDFS-1045 modified UGI, which is in Common on trunk. This JIRA is for that change.
HADOOP-6846. Major task reported by tomwhite and fixed by tomwhite (build)
Scripts for building Hadoop 0.22.0 release
HADOOP-6845. Major improvement reported by jnp and fixed by jnp
TokenStorage renamed to Credentials.

This jira tracks common changes for MAPREDUCE-1528.
HADOOP-6835. Major improvement reported by tomwhite and fixed by roelofs (io)
Support concatenated gzip files

When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)
HADOOP-6834. Major bug reported by ahadr and fixed by hong.tang (io)
TFile.append compares initial key against null lastKey

The following code in TFile.KeyReigster.close:

byte[] lastKey = lastKeyBufferOS.getBuffer();
int lastLen = lastKeyBufferOS.size();
if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
lastLen) < 0) {
throw new IOException("Keys are not added in sorted order");
}

compares the initial key (passed in via TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1 or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results.
HADOOP-6832. Major new feature reported by owen.omalley and fixed by owen.omalley (security)
Provide a web server plugin that uses a static user for the web UI

We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.
HADOOP-6825. Major improvement reported by rschmidt and fixed by rschmidt
FileStatus needs unit tests

We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.
HADOOP-6818. Major improvement reported by devaraj and fixed by devaraj (security)
Provide a JNI-based implementation of GroupMappingServiceProvider

The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.
HADOOP-6815. Major bug reported by boryas and fixed by boryas
refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
HADOOP-6814. Major improvement reported by jnp and fixed by jnp
Method in UGI to get the authentication method of the real user.

UGI should have a method to return the authentication method of the real user for a proxy-user scenario.
HADOOP-6812. Major bug reported by appodictic and fixed by chris.douglas (documentation)
fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.

http://hadoop.apache.org/common/docs/current/cluster_setup.html
fs.inmemory.size.mb does not appear in any xml file
HADOOP-6811. Blocker improvement reported by tomwhite and fixed by tomwhite
Remove EC2 bash scripts

The bash scripts are deprecated in 0.21 (HADOOP-6403) in favour of scripts in Whirr (http://incubator.apache.org/projects/whirr.html). They should be removed in 0.22.
HADOOP-6805. Major improvement reported by boryas and fixed by boryas
add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
HADOOP-6803. Major test reported by eli and fixed by eli (io)
Add native gzip read/write coverage to TestCodec

Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:

while (!compressor.finished()) {
compress();
}

I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.
HADOOP-6791. Major improvement reported by boryas and fixed by boryas
Refresh for proxy superuser config (common part for HDFS-1096)
HADOOP-6787. Major bug reported by vicaya and fixed by vicaya (fs)
Factor out glob pattern code from FileContext and Filesystem

Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)

Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).
HADOOP-6781. Major bug reported by boryas and fixed by boryas
security audit log shouldn't have exception in it.

security audit log in Server.java also prints the exception information. It shouldn't be there.
HADOOP-6778. Major bug reported by boryas and fixed by boryas
add isRunning() method to AbstractDelegationTokenSecretManager (for HDFS-1044)
HADOOP-6763. Major bug reported by owen.omalley and fixed by boryas
Remove verbose logging from the Groups class

2010-02-25 08:30:52,269 INFO security.Groups (Groups.java:<init>(60)) - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
...
2010-02-25 08:30:57,872 INFO security.Groups (Groups.java:getGroups(76)) - Returning cached groups for 'oom'

should both be demoted to debug level.
HADOOP-6761. Major improvement reported by dms and fixed by dms
Improve Trash Emptier

There are two inefficiencies in the Trash functionality right now that have caused some problems for us.
First if you configured your trash interval to be one day (24 hours) that means that you store 2 days worth of data eventually. The Current and the previous timestamp that will not be deleted until the end of the interval.
And another problem is accumulating a lot of data in Trash before the Emptier wakes up. If there are a couple of million files trashed and the Emptier does deletion on HDFS the NameNode will freeze until everything is removed. (this particular problem hopefully will be addressed with HDFS-1143).

My proposal is to have two configuration intervals. One for deleting the trashed data and another for checkpointing. This way for example for intervals of one day and one hour we will only store 25 hours of data instead of 48 right now and the deletions will be happening in smaller chunks every hour of the day instead of a huge deletion at the end of the day now.
HADOOP-6758. Major bug reported by azaroth and fixed by azaroth
MapFile.fix does not allow index interval definition

When using the static methond MapFile.fix() there is no way to override the default IndexInterval that is 128.

The IndexInterval should be taken from the configuration that is passed to the method.
int indexInterval = 128;
indexInterval = conf.getInt(INDEX_INTERVAL, indexInterval);
HADOOP-6756. Major bug reported by zasran and fixed by zasran (fs)
Clean up and add documentation for configuration keys in CommonConfigurationKeys.java

Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).
HADOOP-6747. Major bug reported by vicaya and fixed by tlipcon
TestNetUtils fails on Mac OS X

TestNetUtils fails consistently after HADOOP-6722 on Mac OS X Leopard 10.5.8:

------------- Standard Error -----------------
local address: /127.0.0.1
local port: 64991
------------- ---------------- ---------------

Testcase: testAvoidLoopbackTcpSockets took 0.421 sec
Caused an ERROR
Invalid argument
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:369)
at org.apache.hadoop.net.TestNetUtils.testAvoidLoopbackTcpSockets(TestNetUtils.java:46)

Although TCP spec seems to allow it, at least one implementation disallows this corner case.
HADOOP-6745. Minor improvement reported by boryas and fixed by boryas (ipc)
adding some java doc to Server.RpcMetrics, UGI
HADOOP-6730. Major bug reported by eli and fixed by raviphulari (fs, test)
Bug in FileContext#copy and provide base class for FileContext tests

Thanks to Eli, He noticed that there is no test for FileContext#Copy operation.
On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
*FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
HADOOP-6715. Major bug reported by ravidotg and fixed by ravidotg (security, util)
AccessControlList.toString() returns empty string when we set acl to "*"

AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".

I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".
HADOOP-6714. Major improvement reported by patrickangeles and fixed by patrickangeles
FsShell 'hadoop fs -text' does not support compression codecs

Currently, 'hadoop fs -text myfile' looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means 'fs -text' cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).

It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.
HADOOP-6706. Major bug reported by devaraj and fixed by devaraj (security)
Relogin behavior for RPC clients could be improved

Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.
HADOOP-6693. Major improvement reported by sureshms and fixed by sureshms
Add metrics to track kerberos login activity

Need metrics to track kerberos login activity such as login rate and the time taken for login.
HADOOP-6683. Minor sub-task reported by xiaokang and fixed by xiaokang (io)
the first optimization: ZlibCompressor does not fully utilize the buffer

Thanks for Hong Tang's advice.

Sub task created for the first optimization. HADOOP-6662 closed.
HADOOP-6682. Major bug reported by jghoman and fixed by jghoman (io)
NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly

public static String normalizeHostName(String name) {
if (Character.digit(name.charAt(0), 16) != -1) {
return name;

This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.
HADOOP-6674. Major improvement reported by jnp and fixed by jnp
Performance Improvement in Secure RPC

This jira introduces two performance improvements in Sasl RPC
1. Setting of Sasl 'quality of protection' to authentication only.
2. Addition of BufferedOutputStream underneath SaslOutputStream.
HADOOP-6670. Major bug reported by owen.omalley and fixed by owen.omalley (security)
UserGroupInformation doesn't support use in hash tables

The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.
HADOOP-6669. Minor bug reported by knoguchi and fixed by knoguchi (io)
zlib.compress.level ignored for DefaultCodec initialization

HADOOP-5879 added a compression level for codecs, but DefaultCodec seems to ignore this conf value when initialized.
This is only when codec is first created. reinit() probably sets it right.
HADOOP-6663. Major bug reported by xiaokang and fixed by xiaokang (io)
BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file

An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
HADOOP-6656. Major bug reported by owen.omalley and fixed by devaraj
Security framework needs to renew Kerberos tickets while the process is running

While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.
HADOOP-6652. Major bug reported by devaraj and fixed by devaraj
ShellBasedUnixGroupsMapping shouldn't have a cache

Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.
HADOOP-6648. Major bug reported by owen.omalley and fixed by devaraj (security)
Credentials should ignore null tokens

When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.
HADOOP-6642. Major bug reported by acmurthy and fixed by pocheung
Fix javac, javadoc, findbugs warnings

Unfortunately we've missed some javac, javadoc, findbugs warnings in the recent security work.
HADOOP-6632. Major improvement reported by kzhang and fixed by kzhang
Support for using different Kerberos keys for different instances of Hadoop services

We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.
HADOOP-6623. Minor improvement reported by tlipcon and fixed by tlipcon (util)
Add StringUtils.split for non-escaped single-character separator

This is for HDFS-1028 but useful generally. String.split("/") for example is way slower than an implementation that is specific to only single-character separators.
HADOOP-6620. Major bug reported by jnp and fixed by jnp
NPE if renewer is passed as null in getDelegationToken

If renewer is passed as null in getDelegationToken, an NPE is thrown. We should handle null renewer and the token must not be allowed to be renewed if the renewer is null or empty string.
HADOOP-6613. Major bug reported by kzhang and fixed by kzhang (ipc, security)
RPC server should check for version mismatch first

Currently AuthMethod is checked before Version.
HADOOP-6612. Major bug reported by boryas and fixed by boryas (security)
Protocols RefreshUserToGroupMappingsProtocol and RefreshAuthorizationPolicyProtocol will fail with security enabled
HADOOP-6605. Minor improvement reported by metcalfc and fixed by eli
Add JAVA_HOME detection to hadoop-config

The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.
HADOOP-6600. Major new feature reported by boryas and fixed by boryas
mechanism for authorization check for inter-server protocols

allow configuration of authorization for based on protocol (inter-server).
HADOOP-6599. Major improvement reported by sureshms and fixed by sureshms
Split RPC metrics into summary and detailed metrics

Currently RPC metrics tracks items that provides summary of RPC usage along with more detailed per method statistics that tracks number of method calls made, time spent on that call. Combining both summary and detailed together results in large metrics report which metrics collection systems may not handle. Proposal is to split the metrics into summary and detailed metrics.
HADOOP-6586. Major new feature reported by boryas and fixed by boryas (security)
Log authentication and authorization failures and successes

This jira will cover RPC authentication and SL authorizations logging.
HADOOP-6584. Major improvement reported by jghoman and fixed by jghoman
Provide Kerberized SSL encryption for webservices

Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.
HADOOP-6578. Minor improvement reported by tlipcon and fixed by pirroh (conf)
Configuration should trim whitespace around a lot of value types

I've seen multiple users make an error where they've listed some whitespace around a class name (eg for configuring a scheduler). This results in a ClassNotFoundException which is very hard to debug, as you don't notice the whitespace in the exception! We should simply trim the whitespace in Configuration.getClass and Configuration.getClasses to avoid this class of user error.

Similarly, we should trim in getInt, getLong, etc - anywhere that whitespace doesn't have semantic meaning we should be a little less strict on input.
HADOOP-6562. Minor improvement reported by eli and fixed by eli (test)
FileContextSymlinkBaseTest should use FileContextTestHelper

FileContextSymlinkBaseTest should use FileContextTestHelper, both for utility methods and defining the root test directory instead of using /tmp.
HADOOP-6536. Major bug reported by amareshwari and fixed by ravidotg (fs)
FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument

FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so.
Or it should be changed not to delete the contents of the sym-linked directory.
HADOOP-6496. Minor bug reported by lars_francke and fixed by tlipcon
HttpServer sends wrong content-type for CSS files (and others)

CSS files are send as text/html causing problems if the HTML page is rendered in standards mode. The HDFS interface for example still works because it is rendered in quirks mode, the HBase interface doesn't work because it is rendered in standards mode. See HBASE-2110 for more details.

I've had a quick look at HttpServer but I'm too unfamiliar with it to see the problem. I think this started happening with HADOOP-6441 which would lead me to believe that the filter is called for every request and not only *.jsp and *.html. I'd consider this a bug but I don't know enough about this to provide a fix.
HADOOP-6482. Minor bug reported by cwilkes and fixed by eli (util)
GenericOptionsParser constructor that takes Options and String[] ignores options

This constructor:

public GenericOptionsParser(Options opts, String[] args) {
this(new Configuration(), new Options(), args);
}

should do a this(new Configuration(), opts, args)
HADOOP-6472. Major new feature reported by boryas and fixed by boryas
add tokenCache option to GenericOptionsParser for passing file with secret keys to a map reduce job

for MAPRED-1338 - we need an option to pass a file with secreteKeys (tokens) to a mapreduce Job.
Name of the file is set into the config and will be picked up by JobSubmiter.
HADOOP-6436. Major improvement reported by eli and fixed by rvs
Remove auto-generated native build files

The repo currently includes the automake and autoconf generated files for the native build. Per discussion on HADOOP-6421 let's remove them and use the host's automake and autoconf. We should also do this for libhdfs and fuse-dfs.
HADOOP-6298. Major improvement reported by marz and fixed by owen.omalley
BytesWritable#getBytes is a bad name that leads to programming mistakes

Pretty much everyone at Rapleaf who has worked with Hadoop has misused BytesWritable#getBytes at some point, not expecting the byte array to be padded. I think we can completely alleviate these programming mistakes by deprecating and renaming this method (again) to be more descriptive. I propose "getPaddedBytes()" or "getPaddedValue()". It would also be helpful to have a helper method "getNonPaddedValue()" that makes a copy into a non-padded byte array.
HADOOP-6056. Major improvement reported by stevel@apache.org and fixed by pirroh (scripts)
Use java.net.preferIPv4Stack to force IPv4

This was mentioned on HADOOP-3427, there is a property, java.net.preferIPv4Stack, which you set to true for the java net process to switch to IPv4 everywhere.

As Hadoop doesn't work on IPv6, this should be set to true in the startup scripts. Hopefully this will ensure that Jetty will also pick it up.
HADOOP-4675. Major improvement reported by bockelman and fixed by bockelman (metrics)
Current Ganglia metrics implementation is incompatible with Ganglia 3.1

Ganglia changed its wire protocol in the 3.1.x series; the current implementation only works for 3.0.x.
HADOOP-4487. Major new feature reported by kzhang and fixed by kzhang (security)
Security features for Hadoop

This is a top-level tracking JIRA for security work we are doing in Hadoop. Please add reference to this when opening new security related JIRAs.
HDFS-2287. Trivial bug reported by tlipcon and fixed by tlipcon (test)
TestParallelRead has a small off-by-one bug

Noticed this bug when I was running TestParallelRead - a simple off-by-one error in some internal bounds checking.
HDFS-2573. Major bug reported by shv and fixed by cos (data-node, test)
TestFiDataXceiverServer is failing, not testing OOME

TestFiDataXceiverServer is failing occasionally. It turns out also that the test is not testing the desired condition, because OOME is not caught in DataXceiverServer.run()
HDFS-2514. Major bug reported by eli and fixed by eli (name-node)
Link resolution bug for intermediate symlinks with relative targets

There's a bug in the way the Namenode resolves intermediate symlinks (ie the symlink is not the final path component) in paths when the symlink's target is a relative path. Will post the full description in the first comment.
HDFS-2513. Major task reported by cos and fixed by cos (build)
Bump jetty to 6.1.26

HDFS part of Hadoop-7450
HDFS-2491. Major bug reported by umamaheswararao and fixed by umamaheswararao
TestBalancer can fail when datanode utilization and avgUtilization is exactly same.

Stack Trace:
junit.framework.AssertionFailedError: 127.0.0.1:60986is not an underUtilized node: utilization=22.0 avgUtilization=22.0 threshold=10.0
at org.apache.hadoop.hdfs.server.balancer.Balancer.initNodes(Balancer.java:1014)
at org.apache.hadoop.hdfs.server.balancer.Balancer.initNodes(Balancer.java:953)
at org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:1502)
at org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:247)
at org.apache.hadoop.hdfs.server.balancer.TestBalancer.test(TestBalancer.java:234)
at org.apache.hadoop.hdfs.server.balancer.TestBalancer.twoNodeTest(TestBalancer.java:312)
at org.apache.hadoop.hdfs.server.balancer.TestBalancer.__CLR2_4_39j3j5b10ou(TestBalancer.java:328)
at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0(TestBalancer.java:324)
HDFS-2452. Major bug reported by shv and fixed by umamaheswararao (data-node)
OutOfMemoryError in DataXceiverServer takes down the DataNode

OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread.
HDFS-2388. Major bug reported by shv and fixed by shv (build)
Remove dependency on different version of slf4j in avro

This is the HDFS part of HADOOP-7697.
HDFS-2383. Blocker bug reported by shv and fixed by shv (test)
TestDfsOverAvroRpc is failing on 0.22

{{TestDfsOverAvroRpc.testWorkingDirectory()}} is failing. Possible the result of Avro upgrade.
HDFS-2346. Blocker bug reported by umamaheswararao and fixed by lakshman (test)
TestHost2NodesMap & TestReplicasMap will fail depending upon execution order of test methods
HDFS-2343. Blocker bug reported by jrottinghuis and fixed by jrottinghuis (hdfs client)
Make hdfs use same version of avro as HBase

HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
When building HBase on top of hadoop, this should be consistent.
Moreover, this should be consistent between common, hdfs, and mapreduce.
HDFS-2341. Blocker bug reported by jrottinghuis and fixed by jrottinghuis (build)
Contribs not building

Contribs are not getting built.
Snippet from Jenkins:

compile:
[subant] No sub-builds to iterate on
HDFS-2315. Blocker bug reported by jrottinghuis and fixed by jrottinghuis (build)
Build fails with ant 1.7.0 but works with 1.8.0

Build failure:
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/80
build.xml calls build.xml in contrib, which calls fuse build, which in turn uses build-contrib.
The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0.
HDFS-2297. Blocker bug reported by jrottinghuis and fixed by jrottinghuis (build)
FindBugs OutOfMemoryError

When running the findbugs target from Jenkins, I get an OutOfMemory error.
The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.

We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).
HDFS-2290. Major bug reported by shv and fixed by benoyantony (name-node)
Block with corrupt replica is not getting replicated

A block has one replica marked as corrupt and two good ones. countNodes() correctly detects that there are only 2 live replicas, and fsck reports the block as under-replicated. But ReplicationMonitor never schedules replication of good replicas.
HDFS-2286. Trivial improvement reported by tlipcon and fixed by tlipcon (data-node)
DataXceiverServer logs AsynchronousCloseException at shutdown

During DN shutdown, the acceptor thread gets an AsynchronousCloseException, and logs it at WARN level. This exception is excepted, since another thread is closing the listener socket, so we should just swallow it.
HDFS-2285. Major bug reported by shv and fixed by shv (name-node)
BackupNode should reject requests trying to modify namespace

I am trying to remove file from BackupNode using
{code}hadoop fs -fs hdfs://backup.node.com:50100 -rm /README.txt{code}
which is supposed to fail. But it seems to be hanging forever.
Needs some investigation. It used to throw SafeModeException if I remember correctly.
HDFS-2281. Major bug reported by shv and fixed by umamaheswararao (name-node)
NPE in checkpoint during processIOError()

At the end of checkpoint BackupNode tries to convergeJournalSpool() and calls revertFileStreams(). The latter closes each file stream, and tries to rename the corresponding file to its permanent location current/edits. If for any reason the rename fails processIOError() is called for failed streams. processIOError() will try to close the stream again and will get NPE in EditLogFileOutputStream.close() because bufCurrent was set to null by the previous close.
HDFS-2280. Major bug reported by shv and fixed by shv (name-node)
BackupNode fails with MD5 checksum Exception during checkpoint if BN's image is outdated.

If BN starts after NN made changes to namespace it fails with MD5 checksum Exception during checkpoint when it reads new image upload from NN. This is happening because {{imageDigest}} is not reset to null, but keeps the value of the originally loaded BN image.
HDFS-2271. Major bug reported by umamaheswararao and fixed by umamaheswararao (name-node)
startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.

Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, BackUPNode will proceed with exceptions.
HDFS-2258. Major bug reported by shv and fixed by shv (name-node, test)
TestLeaseRecovery2 fails as lease hard limit is not reset to default

TestLeaseRecovery2.testSoftLeaseRecovery() fails as lease hard limit remains set to 1 sec from the previous test case. If initial file creation in testSoftLeaseRecovery() takes longer than 1 sec, NN correctly reassigns the lease to itself and starts recovery. The test fails as the client cannot hflush() and close the file.
HDFS-2232. Blocker bug reported by shv and fixed by zero45 (test)
TestHDFSCLI fails on 0.22 branch

Several HDFS CLI tests fail on 0.22 branch. I can see 2 reasons:
# Not generic enough regular expression for host names and paths. Similar to MAPREDUCE-2304.
# Some command outputs have new-line in the end.
# And some seem to produce [much] more output than expected.
HDFS-2214. Major bug reported by jrottinghuis and fixed by jrottinghuis (build)
Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

The generated poms inject the version of hdfs itsel, but hardcode the version of hadoop-common they depend on.
When trying to build downstream projects for example mapreduce, then they will require hadoop-common-0.22.0-SNAPSHOT.jar.

When trying to do an offline build this will fail to resolve as another hadoop-common has been installed in the local maven repo.
Even during online build, it should compile against the hadoop-common that hdfs compiled against.

When versions mismatch one cannot do a coherent build. That is particularly problematic when making simultaneous change in hadoop-common and hadoop-hdfs.
HDFS-2211. Blocker bug reported by jrottinghuis and fixed by jrottinghuis (build)
Build does not pass along properties to contrib builds

Subant call to compile contribs do not pass along parameters from parent build.
Properties such as hadoop-common.version, asfrepo, offline, etc. are not passed along.
Result is that build not connected to Internet fails, hdfs proxy refuses to build against own recently built common but rather downloads 0.22-SNAPSHOT from apache again.
HDFS-2189. Blocker bug reported by zero45 and fixed by jrottinghuis
guava-r09 dependency missing from "ivy/hadoop-hdfs-template.xml" in HDFS.

Corrected version of: https://issues.apache.org/jira/browse/MAPREDUCE-2627
HDFS-2071. Minor bug reported by kihwal and fixed by kihwal (data-node)
Use of isConnected() in DataXceiver is invalid

The use of Socket.isConnected() in DataXceiver.run() is not valid. It returns false until the connection is made and then always returns true after that. It will never return false after the initial connection is successfully made. Socket.isClosed() or SocketChannel.isOpen() should be used instead, assuming someone is handling SocketException and does Socket.close() or SocketChannel.close(). It seems the op handlers in DataXceiver are diligently using IOUtils.closeStream(), which will invoke SocketChannel.close().

{code}
- } while (s.isConnected() && socketKeepaliveTimeout > 0);
+ } while (!s.isClosed() && socketKeepaliveTimeout > 0);
{code}

The effect of this bug is very minor, as the socket is read again right after. If the connection was closed, the readOp() will throw an EOFException, which is caught and dealt with properly. The system still functions normally with probably only few microseconds of extra overhead in the premature connection closure cases.
HDFS-2054. Minor improvement reported by kihwal and fixed by kihwal (data-node)
BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()

The addition of ERROR was part of HDFS-1527. In environments where clients tear down FSInputStream/connection before reaching the end of stream, this error message often pops up. Since these are not really errors and especially not the fault of data node, the message should be toned down at least.
HDFS-2039. Major improvement reported by tlipcon and fixed by tlipcon
TestNameNodeMetrics uses a bad test root path

I found that running TestNameNodeMetrics within eclipse fails, since TEST_ROOT_DIR_PATH has a default which is a non-absolute path. Since this path is a DFS path, rather than a local FS path, it shouldn't use the test root dir system property anyhow - we can just hardcode it to a path on DFS.
HDFS-2012. Blocker bug reported by atm and fixed by umamaheswararao (balancer, test)
Recurring failure of TestBalancer due to incorrect treatment of nodes whose utilization equals avgUtilization.

This has been failing on Hudson for the last two builds and fails on my local box as well.
HDFS-2002. Major bug reported by shv and fixed by zero45 (name-node)
Incorrect computation of needed blocks in getTurnOffTip()

{{SafeModeInfo.getTurnOffTip()}} under-reports the number of blocks needed to reach the safemode threshold.
HDFS-2000. Major bug reported by atm and fixed by atm
Missing deprecation for io.bytes.per.checksum

Hadoop long ago deprecated the configuration "io.bytes.per.checksum" in favor of "dfs.bytes-per-checksum", but when the programmatic deprecation support was added, we didn't add an entry for this pair.

This is causing some tests to fail on branch-0.22 since the inclusion of HADOOP-7287, since some tests which were inadvertently using default config values are now having their settings actually picked up.
HDFS-1997. Major sub-task reported by tlipcon and fixed by tlipcon (name-node)
Image transfer process misreports client side exceptions

If the image transfer process receives a client-side error when transferring edit logs, it will throw an exception before it has completely read all of the input from the server-side servlet. Then, the finally clause will throw a new error, since the received length is less than the length given in the header. This masks the client-side exception and makes it look like a network error or a server-side problem.
HDFS-1981. Blocker bug reported by ram_krish and fixed by umamaheswararao (name-node)
When namenode goes down while checkpointing and if is started again subsequent Checkpointing is always failing

This scenario is applicable in NN and BNN case.

When the namenode goes down after creating the edits.new, on subsequent restart the divertFileStreams will not happen to edits.new as the edits.new file is already present and the size is zero.

so on trying to saveCheckPoint an exception occurs
2011-05-23 16:38:57,476 WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java.io.IOException: Namenode has an edit log with timestamp of 2011-05-23 16:38:56 but new checkpoint was created using editlog with timestamp 2011-05-23 16:37:30. Checkpoint Aborted.

This is a bug or is that the behaviour.
HDFS-1980. Major improvement reported by tlipcon and fixed by tlipcon (build)
Move build/webapps deeper in directory heirarchy to aid eclipse users

Currently in order to successfully run unit tests in Eclipse, you have to add the "build/" directory to the classpath, or else the tests won't be able to find the "webapps" dir. This is really annoying because Eclipse then often confuses the classes in build/classes with the source during operations like "go to definition of class".

If we move it into another directory, eg build/web/webapps then we can add build/web to the eclipse classpath and have tests work by default without also having two copies of every class.
HDFS-1978. Major bug reported by brocknoland and fixed by eli (libhdfs)
All but first option in LIBHDFS_OPTS is ignored

In getJNIEnv, we go though LIBHDFS_OPTS with strok and count the number of args. Then create an array of options based on that information. But when we actually setup the options we only the first arg.
HDFS-1969. Blocker bug reported by tlipcon and fixed by tlipcon (name-node)
Running rollback on new-version namenode destroys namespace

The following sequence leaves the namespace in an inconsistent/broken state:
- format NN using 0.20 (or any prior release, probably)
- run hdfs namenode -upgrade on 0.22. ^C the NN once it comes up.
- run hdfs namenode -rollback on 0.22 (this should fail but doesn't!)

This leaves the name directory in a state such that the version file claims it's an 0.20 namespace, but the fsimage is in 0.22 format. It then crashes when trying to start up.
HDFS-1965. Critical bug reported by tlipcon and fixed by tlipcon (security)
IPCs done using block token-based tickets can't reuse connections

This is the reason that TestFileConcurrentReaders has been failing a lot. Reproducing a comment from HDFS-1057:

The test has a thread which continually re-opens the file which is being written to. Since the file's in the middle of being written, it makes an RPC to the DataNode in order to determine the visible length of the file. This RPC is authenticated using the block token which came back in the LocatedBlocks object as the security ticket.

When this RPC hits the IPC layer, it looks at its existing connections and sees none that can be re-used, since the block token differs between the two requesters. Hence, it reconnects, and we end up with hundreds or thousands of IPC connections to the datanode.
HDFS-1964. Major bug reported by atm and fixed by atm
Incorrect HTML unescaping in DatanodeJspHelper.java

HDFS-1575 introduced some HTML unescaping of parameters so that viewing a file would work for paths containing HTML-escaped characters, but in two of the places did the unescaping either too early or too late.
HDFS-1957. Minor improvement reported by asrabkin and fixed by asrabkin (documentation)
Documentation for HFTP

There should be some documentation for HFTP.
HDFS-1954. Major improvement reported by philovivero and fixed by phunt (name-node)
Improve corrupt files warning message on NameNode web UI

On NameNode web interface, you may get this warning:

WARNING : There are about 32 missing blocks. Please check the log or run fsck.

If the cluster was started less than 14 days before, it would be great to add: "Is dfs.data.dir defined?"

If at the point of that error message, that parameter could be checked, and error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, troubleshooting undefined parameters is a difficult proposition.

I suspect this is an easy fix.
HDFS-1952. Major bug reported by mattf and fixed by azuriel
FSEditLog.open() appears to succeed even if all EDITS directories fail

FSEditLog.open() appears to "succeed" even if all of the individual directories failed to allow creation of an EditLogOutputStream. The problem and solution are essentially similar to that of HDFS-1505.
HDFS-1948. Major task reported by stack and fixed by stack
Forward port 'hdfs-1520 lightweight namenode operation to trigger lease reccovery'

This issue is about forward porting from branch-0.20-append the little namenode api that facilitates stealing of a file's lease. The forward port would be an adaption of hdfs-1520 and its companion patches, hdfs-1555 and hdfs-1554, to suit the TRUNK.

Intent is to get this fix into 0.22 time willing; i'll run a vote to get ok on getting it added to branch. HBase needs this facility.
HDFS-1947. Trivial improvement reported by eli and fixed by eli (hdfs client)
DFSClient should use mapreduce.task.attempt.id

DFSClient should use {{mapreduce.task.attempt.id}} instead of the deprecated {{mapred.task.id}}.
HDFS-1946. Major task reported by eli and fixed by eli
HDFS part of HADOOP-7291

The hudson-test-patch target needs to be updated to not pass python.home since this argument is no longer needed.
HDFS-1943. Blocker bug reported by weiyj and fixed by mattf (scripts)
fail to start datanode while start-dfs.sh is executed by root user

When start-dfs.sh is run by root user, we got the following error message:
# start-dfs.sh
Starting namenodes on [localhost ]
localhost: namenode running as process 2556. Stop it first.
localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out
localhost: Unrecognized option: -jvm
localhost: Could not create the Java virtual machine.

The -jvm options should be passed to jsvc when we starting a secure
datanode, but it still passed to java when start-dfs.sh is run by root
while secure datanode is disabled. This is a bug of bin/hdfs.
HDFS-1936. Blocker bug reported by sureshms and fixed by sureshms (name-node)
Updating the layout version from HDFS-1822 causes upgrade problems.

In HDFS-1822 and HDFS-1842, the layout versions for 203, 204, 22 and trunk were changed. Some of the namenode logic that depends on layout version is broken because of this. Read the comment for more description.
HDFS-1935. Minor improvement reported by tlipcon and fixed by jrottinghuis (build)
Build should not redownload ivy on every invocation

Currently we re-download ivy every time we build. If the jar already exists, we should skip this.
HDFS-1932. Critical bug reported by tlipcon and fixed by jolly
Ensure that HDFS configuration deprecations are set up in every spot that HDFS configurations are loaded.

Currently we have code blocks like the following in lots of places:

static{
Configuration.addDefaultResource("hdfs-default.xml");
Configuration.addDefaultResource("hdfs-site.xml");
}

This is dangerous since, if we don't remember to also classload HdfsConfiguration, the config key deprecations won't work. We should add a method like HdfsConfiguration.init() which would load the default resources as well as ensure that deprecation gets initialized properly.
HDFS-1925. Major bug reported by shv and fixed by fwiffo
SafeModeInfo should use DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_DEFAULT instead of 0.95

{{SafeMode()}} constructor has 0.95f default threshold hard-coded. This should be replaced by the constant {{DFS_NAMENODE_SAFEMODE_THRESHOLD_PCT_DEFAULT}}, which is correctly set to 0.999f.
HDFS-1921. Blocker bug reported by atm and fixed by mattf
Save namespace can cause NN to be unable to come up on restart

I discovered this in the course of trying to implement a fix for HDFS-1505.

Per the comment for {{FSImage.saveNamespace(...)}}, the algorithm for save namespace proceeds in the following order:

# rename current to lastcheckpoint.tmp for all of them,
# save image and recreate edits for all of them,
# rename lastcheckpoint.tmp to previous.checkpoint.

The problem is that step 3 occurs regardless of whether or not an error occurs for all storage directories in step 2. Upon restart, the NN will see non-existent or corrupt {{current}} directories, and no {{lastcheckpoint.tmp}} directories, and so will conclude that the storage directories are not formatted.

This issue appears to be present on both 0.22 and 0.23. This should arguably be a 0.22/0.23 blocker.
HDFS-1909. Major bug reported by tomwhite and fixed by tomwhite
TestHDFSCLI fails due to typo in expected output

"apended" is misspelled in testHDFSConf.xml and causes the test to fail on the 0.22 branch.
HDFS-1897. Minor bug reported by asrabkin and fixed by whangsf (documentation)
Documention refers to removed option dfs.network.script

The HDFS user guide tells users to use dfs.network.script for rack awareness. In fact, this option has been removed and using it will trigger a fatal error on DataNode startup. Documentation should describe the current rack awareness configuration system.
HDFS-1891. Major bug reported by sureshms and fixed by sureshms (test)
TestBackupNode fails intermittently

TestBackupNode fails due to unexpected ipv6 address format.
HDFS-1866. Minor improvement reported by eli and fixed by qwertymaniac (data-node, documentation)
Document dfs.datanode.max.transfer.threads in hdfs-default.xml

dfs.datanode.max.transfer.threads (formerly dfs.datanode.max.xcievers) should be covered in hdfs-default.xml.
HDFS-1861. Major improvement reported by eli and fixed by eli (data-node)
Rename dfs.datanode.max.xcievers and bump its default value

Reasonably sized jobs and HBase easily exhaust the current default for dfs.datanode.max.xcievers. 4096 works better in practice.

Let's also deprecate it in favor of a more intuitive name, eg dfs.datanode.max.receiver.threads.
HDFS-1855. Major test reported by mattf and fixed by mattf (test)
TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy() part 2 fails in two different ways

The second part of test case TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy(), "corrupt replica recovery for two corrupt replicas", always fails, half the time with a checksum error upon block replication, and half the time by timing out upon failure to delete the second corrupt replica.
HDFS-1850. Major bug reported by eli and fixed by eli (data-node, name-node)
DN should transmit absolute failed volume count rather than increments to the NN

The API added in HDFS-811 for the DN to report volume failures to the NN is "inc(DN)". However the given sequence of events will result in the NN forgetting about reported failed volumes:

# DN loses a volume and reports it
# NN restarts
# DN re-registers to the new NN

A more robust interface would be to have the DN report the total number of volume failures to the NN each heart beat (the same way other volume state is transmitted).
HDFS-1845. Major bug reported by johnvijoe and fixed by johnvijoe
symlink comes up as directory after namenode restart

When a symlink is first created, it get added to EditLogs. When namenode is restarted, it reads from this editlog and represents a symlink correctly and saves this information to its image. If the namenode is restarted again, it reads its from this FSImage, but thinks that a symlink is a directory. This is because it uses "Block[] blocks" to determine if an INode is a directory, a file, or symlink. Since both a directory and a symlink has blocks as null, it thinks that a symlink is a directory.
HDFS-1825. Major task reported by nidaley and fixed by nidaley
Remove thriftfs contrib

As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CEF44CFE2-692F-4956-8B33-D125D05E2B53@mac.com%3E) thriftfs can be removed:
svn remove hdfs/trunk/src/contrib/thriftfs
and wiki updated:
http://wiki.apache.org/hadoop/Attic
HDFS-1823. Blocker bug reported by tomwhite and fixed by tomwhite (scripts)
start-dfs.sh script fails if HADOOP_HOME is not set

HDFS portion of HADOOP-6953
HDFS-1822. Blocker bug reported by sureshms and fixed by sureshms (name-node)
Editlog opcodes overlap between 20 security and later releases

Same opcode are used for different operations between 0.20.security, 0.22 and 0.23. This results in failure to load editlogs on later release, especially during upgrades.
HDFS-1821. Major bug reported by johnvijoe and fixed by johnvijoe
FileContext.createSymlink with kerberos enabled sets wrong owner

TEST SETUP
Using attached sample hdfs java program that illustrates the issue.
Using cluster with Kerberos enabled on cluster

# Compile instructions
$ javac Symlink.java -cp `hadoop classpath`
$ jar -cfe Symlink.jar Symlink Symlink.class

# create test file for symlink to use
1. hadoop fs -touchz /user/username/filetest

# Create symlink using file context
2. hadoop jar Symlink.jar ln /user/username/filetest /user/username/testsymlink

# Verify owner of test file
3. hadoop jar Symlink.jar ls /user/username/
-rw-r--r-- username hdfs /user/jeagles/filetest
-rwxrwxrwx username@XX.XXXX.XXXXX.XXX hdfs /user/username/testsymlink

RESULTS
1. Owner shows 'username@XX.XXXX.XXXXX.XXX' for symlink, expecting 'username'.
2. Symlink is corrupted and can't removed, since it was created with an invalid user

------------------------
Sample program to create Symlink

FileContext fc = FileContext.getFileContext(getConf());
fc.createSymlink(target, symlink, false);

---------------------------------------
HDFS-1806. Major bug reported by mattf and fixed by mattf (data-node, name-node)
TestBlockReport.blockReport_08() and _09() are timing-dependent and likely to fail on fast servers

Method waitForTempReplica() polls every 100ms during block replication, attempting to "catch" a datanode in the state of having a TEMPORARY replica. But examination of a current Hudson test failure log shows that the replica goes from "start" to "TEMPORARY" to "FINALIZED" in only 50ms, so of course the poll usually misses it.
HDFS-1786. Minor bug reported by szetszwo and fixed by umamaheswararao (test)
Some cli test cases expect a "null" message

There are a few tests cases specified in {{TestHDFSCLI}} and {{TestDFSShell}} expecting "null" messages.
e.g. in {{testHDFSConf.xml}},
<expected-output>get: null</expected-output>
HDFS-1782. Major bug reported by johnvijoe and fixed by johnvijoe (name-node)
FSNamesystem.startFileInternal(..) throws NullPointerException

I'm observing when there is one balancer running trying to run another one results in
"Java.lang.NullPointerException" error. I was hoping to see message "Another balancer is running.
Exiting.... Exiting ...". This is a reproducible issue.
HDFS-1781. Major bug reported by johnvijoe and fixed by johnvijoe (scripts)
jsvc executable delivered into wrong package...

The jsvc executable is delivered in the 0.22 hdfs package, but the script that uses it (bin/hdfs) refers to
$HADOOP_HOME/bin/jsvc to find it.
HDFS-1762. Major test reported by tomwhite and fixed by cos (build, test)
Allow TestHDFSCLI to be run against a cluster

Currently TestHDFSCLI starts mini clusters to run tests against. It would be useful to be able to support running against arbitrary clusters for testing purposes.
HDFS-1736. Minor improvement reported by daryn and fixed by daryn (data-node)
Break dependency between DatanodeJspHelper and FsShell

DatanodeJspHelper has an artificial dependency on a date formatter field in FsShell. A pending bug is reorganizing the FsShell commands so this field will no longer be available. The dependency should be broken by having DataNodeJspHelper contain its own independent date formatter.
HDFS-1625. Minor bug reported by tlipcon and fixed by szetszwo (test)
TestDataNodeMXBean fails if disk space usage changes during test run

I've seen this on our internal hudson - we get failures like:

null expected:<...:{"freeSpace":857683[43552],"usedSpace":28672,"...> but was:<...:{"freeSpace":857683[59936],"usedSpace":28672,"...>

because some other build on the same build slave used up some disk space during the middle of the test.
HDFS-1621. Major bug reported by tlipcon and fixed by jolly
Fix references to hadoop-common-${version} in build.xml

Similar to MAPREDUCE-2315, we should fix any references to the hadoop common jar that use ${version} instead of ${hadoop-common.version}.
HDFS-1619. Major improvement reported by rvs and fixed by rvs (libhdfs)
Remove AC_TYPE* from the libhdfs

Remove AC_TYPE* from the libhdfs build since we get these via stdint.

Currently configure.ac uses AC_TYPE_INT16_T, AC_TYPE_INT32_T, AC_TYPE_INT64_T and AC_TYPE_UINT16_T and thus requires autoconf 2.61 or higher.
This prevents using it on such platforms as CentOS/RHEL 5.4 and 5.5. Given that those are pretty popular and also given that it is really difficult to find a platform
these days that doesn't natively define intXX_t types I'm curious as to whether we can simply remove those macros or perhaps fail ONLY if we happen to be on such
a platform.

Here's a link to GNU autoconf docs for your reference:
http://www.gnu.org/software/hello/manual/autoconf/Particular-Types.html
HDFS-1618. Major improvement reported by rvs and fixed by rvs (build)
configure files that are generated as part of the released tarball need to have executable bit set

Currently the configure files that are packaged in a tarball are -rw-rw-r--
HDFS-1615. Major bug reported by tlipcon and fixed by scott_carey
seek() on closed DFS input stream throws NPE

After closing an input stream on DFS, seeking slightly ahead of the last read will throw an NPE:

java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:749)
at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:42)
HDFS-1602. Major bug reported by cos and fixed by boryas (name-node)
NameNode storage failed replica restoration is broken

NameNode storage restore functionality doesn't work (as HDFS-903 demonstrated). This needs to be either disabled, or removed, or fixed. This feature also fails HDFS-1496
HDFS-1597. Blocker bug reported by tlipcon and fixed by tlipcon
Batched edit log syncs can reset synctxid throw assertions

The top of FSEditLog.logSync has the following assertion:
assert editStreams.size() > 0 : "no editlog streams";
which should actually come after checking to see if the sync was already batched in by another thread.

This is related to a second bug in which the same case causes synctxid to be reset to 0
HDFS-1591. Major bug reported by pocheung and fixed by pocheung
Fix javac, javadoc, findbugs warnings

Split from HADOOP-6642
HDFS-1582. Major improvement reported by rvs and fixed by rvs (libhdfs)
Remove auto-generated native build files

The repo currently includes the automake and autoconf generated files for the native build. Per discussion on HADOOP-6421 let's remove them and use the host's automake and autoconf. We should also do this for libhdfs and fuse-dfs.
HDFS-1575. Blocker bug reported by tlipcon and fixed by atm
viewing block from web UI broken

DatanodeJspHelper seems to expect the file path to be in the "path info" of the HttpRequest, rather than in a parameter. I see the following exception when visiting the URL {{http://localhost.localdomain:50075/browseBlock.jsp?blockId=5006108823351810567&blockSize=20&genstamp=1001&filename=%2Fuser%2Ftodd%2Fissue&datanodePort=50010&namenodeInfoPort=50070}}

java.io.FileNotFoundException: File does not exist: /
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:834)
...
at org.apache.hadoop.hdfs.server.datanode.DatanodeJspHelper.generateFileDetails(DatanodeJspHelper.java:258)
at org.apache.hadoop.hdfs.server.datanode.browseBlock_jsp._jspService(browseBlock_jsp.java:79)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
HDFS-1572. Blocker bug reported by liangly and fixed by jghoman
Checkpointer should trigger checkpoint with specified period.

{dfs.namenode.checkpoint.period} in configuration determines the period of checkpoint. However, with above code, the Checkpointer triggers a checkpoint every 5 minutes (periodMSec=5*60*1000).
HDFS-1562. Major test reported by eli and fixed by eli (name-node, test)
Add rack policy tests

The existing replication tests (TestBlocksWithNotEnoughRacks, TestPendingReplication, TestOverReplicatedBlocks, TestReplicationPolicy, TestUnderReplicatedBlocks, and TestReplication) are missing tests for rack policy violations. This jira adds the following tests which I created when generating a new patch for HDFS-15.

* Test that blocks that have a sufficient number of total replicas, but are not replicated cross rack, get replicated cross rack when a rack becomes available.
* Test that new blocks for an underreplicated file will get replicated cross rack.
* Mark a block as corrupt, test that when it is re-replicated that it is still replicated across racks.
* Reduce the replication factor of a file, making sure that the only block that is across racks is not removed when deleting replicas.
* Test that when a block is replicated because a replica is lost due to host failure the the rack policy is preserved.
* Test that when the execss replicas of a block are reduced due to a node re-joining the cluster the rack policy is not violated.
* Test that rack policy is still respected when blocks are replicated due to node decommissioning.
* Test that rack policy is still respected when blocks are replicated due to node decommissioning, even when the blocks are over-replicated.
HDFS-1561. Blocker bug reported by shv and fixed by shv (name-node)
BackupNode listens on default host

Currently BackupNode uses DNS to find its default host name, and then starts RPC server listening on that address ignoring the address specified in the configuration. Therefore, there is no way to start BackupNode on a particular ip or host address. BackupNode should use the address specified in the configuration instead.
HDFS-1560. Minor bug reported by tlipcon and fixed by tlipcon (data-node)
dfs.data.dir permissions should default to 700

Currently, dfs.data.dir defaults to 755 permissions, which isn't necessary for any reason, and is a security issue if not changed on a secured cluster. We should default to 700
HDFS-1550. Blocker bug reported by hairong and fixed by hairong (hdfs client)
NPE when listing a file with no location

Lines listed below will caused a NullPointerException in DFSUtil.locatedBlocks2Locations (line 208) because EMPTY_BLOCK_LOCS will return null when calling blocks.getLocatedBlocks()
/** a default LocatedBlocks object, its content should not be changed */
private final static LocatedBlocks EMPTY_BLOCK_LOCS = new LocatedBlocks();
HDFS-1544. Major bug reported by vicaya and fixed by vicaya
Ivy resolve force mode should be turned off by default

cf. HADOOP-7068
HDFS-1542. Critical bug reported by tlipcon and fixed by tlipcon (hdfs client)
Deadlock in Configuration.writeXml when serialized form is larger than one DFS block

Configuration.writeXml holds a lock on itself and then writes the XML to an output stream, during which DFSOutputStream will try to get a lock on ackQueue/dataQueue. Meanwihle the DataStreamer thread will call functions like conf.getInt() and deadlock against the other thread, since it could be the same conf object.

This causes a deterministic deadlock whenever the serialized form is larger than block size.
HDFS-1532. Major bug reported by tlipcon and fixed by tlipcon (test)
Exclude Findbugs warning in FSImageFormat$Saver

HDFS-1473 had a followup patch which changed a method name, but I forgot to modify the findbugs exclude file to match.
HDFS-1531. Trivial bug reported by tlipcon and fixed by tlipcon (data-node, name-node)
Clean up stack traces due to duplicate MXBean registration

In the minicluster unit tests, we try to register MXBeans for each DN, but since the JMX context is JVM-wide, we get a InstanceAlreadyExistsException for all but the first. This stack trace clutters test logs a lot.
HDFS-1529. Blocker bug reported by tlipcon and fixed by tlipcon (hdfs client)
Incorrect handling of interrupts in waitForAckedSeqno can cause deadlock

In HDFS-895 the handling of interrupts during hflush/close was changed to preserve interrupt status. This ends up creating an infinite loop in waitForAckedSeqno if the waiting thread gets interrupted, since Object.wait() has a strange semantic that it doesn't give up the lock even momentarily if the thread is already in interrupted state at the beginning of the call.

We should decide what the correct behavior is here - if a thread is interrupted while it's calling hflush() or close() should we (a) throw an exception, perhaps InterruptedIOException (b) ignore, or (c) wait for the flush to finish but preserve interrupt status on exit?
HDFS-1527. Major bug reported by pkling and fixed by pkling (data-node)
SocketOutputStream.transferToFully fails for blocks >= 2GB on 32 bit JVM

On 32 bit JVM, SocketOutputStream.transferToFully() fails if the block size is >= 2GB. We should fall back to a normal transfer in this case.
HDFS-1524. Blocker bug reported by hairong and fixed by hairong (name-node)
Image loader should make sure to read every byte in image file

When I work on HDFS-1070, I come across a very strange bug. Occasionally when loading a compressed image, NameNode throws an exception complaining that the image file is corrupt. However, the result returned by the md5sum utility matches the checksum value stored in the VERSION file.

It turns out the image loader leaves 4 bytes unread after loading all the real data of an image. Those unread bytes may be some compression-related meta-info. The image loader should make sure to read to the end of file in order for an correct checksum.
HDFS-1523. Major bug reported by cos and fixed by cos (test)
TestLargeBlock is failing on trunk

TestLargeBlock is failing for more than a week not on 0.22 and trunk with
java.io.IOException: Premeture EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:118)
at org.apache.hadoop.hdfs.BlockReader.readChunk(BlockReader.java:275)
HDFS-1513. Minor improvement reported by eli and fixed by eli
Fix a number of warnings

There are a bunch of warnings besides DeprecatedUTF8, HDFS-1512 and two warnings caused by a Java bug (http://bugs.sun.com/view_bug.do?bug_id=646014) that we can fix.
HDFS-1511. Blocker bug reported by nidaley and fixed by jghoman
98 Release Audit warnings on trunk and branch-0.22

There are 98 release audit warnings on trunk. See attached txt file. These must be fixed or filtered out to get back to a reasonably small number of warnings. The OK_RELEASEAUDIT_WARNINGS property in src/test/test-patch.properties should also be set appropriately in the patch that fixes this issue.
HDFS-1507. Minor bug reported by eli and fixed by eli (test)
TestAbandonBlock should abandon a block

TestAbandonBlock as written just tests that if one client tries to abandon another it will fail because the new client doesn't have a lease on the other client's file, which doesn't cover the block abandonement code path.
HDFS-1505. Blocker bug reported by tlipcon and fixed by atm
saveNamespace appears to succeed even if all directories fail to save

After HDFS-1071, saveNamespace now appears to "succeed" even if all of the individual directories failed to save.
HDFS-1504. Minor bug reported by tlipcon and fixed by tlipcon (name-node)
FSImageSaver should catch all exceptions, not just IOE

FSImageSaver currently just catches IOE. This means that if some other error like OOME or failed assert happens in saving one of the images, the coordinating thread won't know there was a problem.
HDFS-1503. Minor bug reported by eli and fixed by tlipcon (test)
TestSaveNamespace fails

Will attach the full log. Here's the relevant snippet:

Exception in thread "FSImageSaver for /home/eli/src/hdfs4/build/test/data/dfs/na
me1 of type IMAGE_AND_EDITS" java.lang.RuntimeException: Injected fault: saveFSI
mage second time
....
at java.lang.Thread.run(Thread.java:619)
Exception in thread "FSImageSaver for /home/eli/src/hdfs4/build/test/data/dfs/na
me2 of type IMAGE_AND_EDITS" java.lang.StackOverflowError
at java.util.Arrays.copyOf(Arrays.java:2882)
HDFS-1502. Minor bug reported by eli and fixed by hairong
TestBlockRecovery triggers NPE in assert

Testcase: testRBW_RWRReplicas took 10.333 sec
Caused an ERROR
null
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.datanode.DataNode.syncBlock(DataNode.java:1881)
at org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testSyncReplicas(TestBlockRecovery.java:144)
at org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testRBW_RWRReplicas(TestBlockRecovery.java:305)
HDFS-1500. Major bug reported by tlipcon and fixed by tlipcon (test, tools)
TestOfflineImageViewer failing on trunk

Testcase: testOIV took 22.679 sec
FAILED
Failed reading valid file: No image processor to read version -26 is available.
junit.framework.AssertionFailedError: Failed reading valid file: No image processor to read version -26 is available.
at org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.outputOfLSVisitor(TestOfflineImageViewer.java:171)
at org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.testOIV(TestOfflineImageViewer.java:86)
HDFS-1498. Minor bug reported by eli and fixed by eli (name-node)
FSDirectory#unprotectedConcat calls setModificationTime on a file

The HDFSConcat test fails when asserts are enabled because FSDirectory#unprotectedConcat calls INode#setModificationTime on a file, this method asserts that the argument is a directory. It should use setModificationTimeForce since we know the target is a file, it's mod time should be set to now unconditionally since we know we're modifying it. The behavior should be equivalent.
HDFS-1491. Major improvement reported by sanjay.radia and fixed by sanjay.radia
Update Hdfs to match the change of methods from protected to public in AbstractFileSystem (Hadoop-6903)
HDFS-1487. Major bug reported by zhong and fixed by zhong (name-node)
FSDirectory.removeBlock() should update diskspace count of the block owner node

This bug leads a quota computation error. When calling abandonBlock(), the cached diskspace count of INodeDirectoryWithQuota is larger than the actual consumed space value.
HDFS-1485. Minor improvement reported by yaojingguo and fixed by yaojingguo
Typo in BlockPlacementPolicy.java

In line 157, "thatis" should be "that is".
HDFS-1483. Major bug reported by pkling and fixed by pkling (hdfs client)
DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt

When there are no uncorrupted replicas of a block, FSNamesystem.getBlockLocations returns LocatedBlocks corresponding to corrupt blocks. When DFSClient converts these to BlockLocations, the information that the corresponding block is corrupt is lost. We should add a field to BlockLocation to indicate whether the corresponding block is corrupt in order to warn the client that reading this block will fail. This would be especially useful for tools such as RAID FSCK, which could then easily inspect whether data or parity blocks are corrupted without having to make direct RPC calls.
HDFS-1473. Major sub-task reported by tlipcon and fixed by tlipcon (name-node)
Refactor storage management into separate classes than fsimage file reading/writing

Currently the FSImage class is responsible both for storage management (eg moving around files, tracking file names, the VERSION file, etc) as well as for the actual serialization and deserialization of the "fsimage" file within the storage directory.

I'd like to refactor the loading and saving code into new classes. This will make testing easier and also make the major changes in HDFS-1073 easier to understand.
HDFS-1472. Major improvement reported by rvadali and fixed by rvadali (tools)
Refactor DFSck to allow programmatic access to output

DFSck prints the list of corrupt files to stdout. This jira proposes that it write to a PrintStream object that is passed to the constructor. This will allow components like RAID to programmatically get a list of corrupt files.
HDFS-1467. Blocker bug reported by tlipcon and fixed by tlipcon (data-node)
Append pipeline never succeeds with more than one replica

TestPipelines appears to be failing on trunk:

Should be RBW replica after sequence of calls append()/write()/hflush() expected:<RBW> but was:<FINALIZED>
junit.framework.AssertionFailedError: Should be RBW replica after sequence of calls append()/write()/hflush() expected:<RBW> but was:<FINALIZED>
at org.apache.hadoop.hdfs.TestPipelines.pipeline_01(TestPipelines.java:109)
HDFS-1466. Minor bug reported by tlipcon and fixed by eli (test)
TestFcHdfsSymlink relies on /tmp/test not existing

The test case testLinkAcrossFileSystems works with files in a directory /tmp/test. On our shared build systems this sometimes breaks because such a directory already exists but owned by some other user. The test case should use the build test dir instead.
HDFS-1462. Major sub-task reported by tlipcon and fixed by tlipcon (name-node)
Refactor edit log loading to a separate class from edit log writing

Before the major work in HDFS-1073, I'd like to do this refactor to clean up the monster FSEditLog class. We can separate all the functions that take of loading edits into an FSN from the functions that take care of writing edits, rolling, etc.
HDFS-1457. Major improvement reported by hairong and fixed by hairong (name-node)
Limit transmission rate when transfering image between primary and secondary NNs

If the fsimage is very big. The network is full in a short time when SeconaryNamenode do checkpoint, leading to Jobtracker access Namenode to get relevant file data to fail in job initialization phase. So we limit transmission speed and compress transmission to resolve the problem.
HDFS-1456. Major improvement reported by jghoman and fixed by jghoman (test)
Provide builder for constructing instances of MiniDFSCluster

Time to fix a broken window. Of the 293 occurences of "new MiniDFSCluster("... most look something like:
{noformat}cluster = new MiniDFSCluster(0, config, numDatanodes, true, false, true, null, null, null, null);{noformat}
The largest constructor takes 10 parameters, and even the overloaded constructors can be difficult to read as their mainaly nulls or booleans.

We should provide a Builder for constructing MiniDFSClusters to improve readability.
HDFS-1454. Major improvement reported by hammer and fixed by qwertymaniac (documentation, hdfs client)
Update the documentation to reflect true client caching strategy

As noted on the mailing list (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201010.mbox/%3CAANLkTi=2csK+aY05bTOuO-UZv=O4w6oX2Pq4nxGPDZBi@mail.gmail.com%3E), the "Staging" section of http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_design.html#Data+Organization is out of date.
HDFS-1440. Major bug reported by sureshms and fixed by sureshms (test)
TestComputeInvalidateWork fails intermittently

TestComputeInvalidateWork fails intermittently. This is due to incorrect synchronization introduced due to HDFS-1093. The test uses blocks synchronized on FSNamesystem monitor, however with HDFS-1093, the FSNamesystem uses explicit read/write lock for synchronization.
HDFS-1435. Major improvement reported by hairong and fixed by hairong (name-node)
Provide an option to store fsimage compressed

Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network bandwidth when secondary NN uploads a new fsimage to primary NN.

If we could store fsimage compressed, the problem could be greatly alleviated.

I plan to provide a new configuration hdfs.image.compressed with a default value of false. If it is set to be true, fsimage is stored as compressed.

The fsimage will have a new layout with a new field "compressed" in its header, indicating if the namespace is stored compressed or not.
HDFS-1434. Major improvement reported by sureshms and fixed by sureshms (data-node)
Refactor Datanode#startDataNode method

This method is very long, approximately 190 lines. Splitting this method into smaller methods. This also helps in federation where some of the initialization in this methods will be done once for a datanode and others once in every heartbeat thread per namenode in DN.
HDFS-1433. Major bug reported by sureshms and fixed by sureshms (test)
Fix test failures - TestPread and TestFileLimit
HDFS-1426. Major improvement reported by hairong and fixed by hairong (name-node)
Remove unused method BlockInfo#listCount

BlockInfo#listCount is not used anywhere so removing it.
HDFS-1417. Major improvement reported by sureshms and fixed by sureshms
Add @Override annotation to SimulatedFSDataset methods that implement FSDatasetInterface

@Override annotations are inconsistently added to methods implementing the interface.
HDFS-1409. Trivial bug reported by chingshen and fixed by chingshen (name-node)
The "register" method of the BackupNode class should be "UnsupportedActionException("register")"

The register method of the BackupNode class should be "UnsupportedActionException("register")" rather than "UnsupportedActionException("journal")".
HDFS-1408. Major improvement reported by almacro and fixed by cos (test)
Herriot NN and DN clients should vend statistics

The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
The Herriot interface to the namenode and datanode (as implemented in NNClient and DNClient, respectively) would benefit from the addition of some way to channel this information. In the case of DNClient this can be an injected method that returns a DatanodeDescriptor relevant to the underlying datanode.
There seems to be no analagous NamenodeDescriptor. It may be useful to add this as a facade to a visitor that aggregates values across the filesystem datanodes.
HDFS-1399. Major bug reported by atm and fixed by atm (security)
Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies

HDFS portion of HADOOP-6951.
HDFS-1395. Major improvement reported by sureshms and fixed by sureshms (data-node)
Add @Override annotation to FSDataset methods that implement FSDatasetInterface

@Override annotations are inconsistently added to methods implementing the interface.
HDFS-1387. Major improvement reported by tlipcon and fixed by tlipcon (documentation, security)
Update HDFS permissions guide for security

The HDFS permissions guide currently makes several statements that are no longer correct now that we provide security.
HDFS-1383. Major improvement reported by szetszwo and fixed by szetszwo
Better error messages on hftp

If the file is not accessible, HftpFileSystem returns only a HTTP response code.
2010-08-27 20:57:48,091 INFO org.apache.hadoop.tools.DistCp: FAIL README.txt : java.io.IOException:
Server returned HTTP response code: 400 for URL: http:/namenode:50070/data/user/tsz/README.txt?ugi=tsz,users
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1290)
at org.apache.hadoop.hdfs.HftpFileSystem.open(HftpFileSystem.java:143)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356)
...
HDFS-1369. Trivial improvement reported by eli and fixed by eli (data-node)
Invalid javadoc reference in FSDatasetMBean.java

FSDatasetMBean.java references non-existant class DataNodeStatisticsMBean, I believe it meant to reference DataNodeActivityMBean.
After this patch the javadoc and javadoc-dev targets run cleanly:

[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
HDFS-1368. Major improvement reported by hairong and fixed by hairong (name-node)
Add a block counter to DatanodeDescriptor

Many of time, we need to count the total number of blocks in a datanode by calling DatanodeDescriptor#numBlocks(). It has to traverse the block list to get the counter, which may not be trivial if there are millions of blocks. Adding a counter for each datanode does not cost much space but help save CPU processing time.
HDFS-1364. Major bug reported by kzhang and fixed by jnp (security)
HFTP client should support relogin from keytab

If a user starts a long-running HFTP client using a keytab, we should do relogin automatically whenever TGT expires. Currently, HFTP client uses TGT to fetch a delegation token and cache that delegation token for HFTP operations. The delegation token is automatically renewed/refetched using TGT. However, when TGT expires, delegation token renewal/refetch will fail and no further HFTP operation is possible. This is unsatisfactory since the user has given us her keytab. We should be able to relogin from keytab and continue.
HDFS-1361. Major bug reported by shv and fixed by shv
Add -fileStatus operation to NNThroughputBenchmark

getFileStatus() is a frequently used operation in HDFS. It important to benchmark the name-node throughput on it.
HDFS-1357. Major bug reported by kzhang and fixed by kzhang (data-node, security)
HFTP traffic served by DataNode shouldn't use service port on NameNode

HDFS-599 introduced a new service port on NameNode to separate system traffic (e.g., heartbeats/blockreports) from client file access requests so that they can be prioritized. All Datanode traffic now goes to the service port. However, datanode also serves as a proxy for HFTP requests from client (served by StreamFile servlet). These HFTP traffic should continue to use the client port on NameNode. Moreover, using the service port for HFTP is incompatible with the existing way of selecting delegation tokens.
HDFS-1356. Major improvement reported by boryas and fixed by boryas
Provide information as to whether or not security is enabled on web interface for NameNode (part of HADOOP-6822)
HDFS-1355. Major bug reported by vicaya and fixed by vicaya (build)
ant veryclean (clean-cache) doesn't clean enough

Looks like since HDFS-1159, ant veryclean no longer work as expected for the case when hadoop common jars are changed. The proposed patch does a more through cleaning on hadoop jars.
HDFS-1353. Major improvement reported by jghoman and fixed by jghoman (name-node)
Remove most of getBlockLocation optimization
HDFS-1352. Major bug reported by eli and fixed by eli (build)
Fix jsvc.location

The jsvc specified in build.xml 404s, causing the build to fail, because version 1.0.2 has been archived. Let's update the url, not sure we want to move to 1.0.3 or play the game where the build breaks with every jsvc dot release.
HDFS-1349. Major bug reported by tomwhite and fixed by eli
Remove empty java files

Doug identified some empty java files that should be deleted:

src/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeStatisticsMBean.java
src/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeStatistics.java
src/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeStatistics.java
src/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeStatisticsMBean.java
HDFS-1347. Major bug reported by boryas and fixed by boryas (test)
TestDelegationToken uses mortbay.log for logging

needs to be changed to commons.log
HDFS-1340. Major bug reported by jnp and fixed by jnp
A null delegation token is appended to the url if security is disabled when browsing filesystem.

When filesystem is being browsed and if security is disabled a null delegation token is added to the url. Also if a user changes the url and adds any random string for delegation token, it is retained in the links on returned html page. If security is disabled no delegation token parameter is required on the url.
HDFS-1334. Major bug reported by jnp and fixed by jnp
open in HftpFileSystem does not add delegation tokens to the url.

open method in HftpFileSystem uses ByteRangeInputStream for url connection. But it does not add the delegation tokens, even if security is enabled, to the url before passing it to the ByteRangeInputStream. Therefore request fails if security is enabled.
HDFS-1330. Major new feature reported by hairong and fixed by johnvijoe (data-node)
Make RPCs to DataNodes timeout

This jira aims to make client/datanode or datanode/datanode RPC to have a timeout of DataNode#socketTimeout.
HDFS-1320. Major improvement reported by zasran and fixed by zasran
Add LOG.isDebugEnabled() guard for each LOG.debug("...")

Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason wheather it's neccessary or not).
HDFS-1319. Major sub-task reported by jghoman and fixed by jghoman (name-node)
Fix location of re-login for secondary namenode from HDFS-999

A bugfix to the original patch (HDFS-999) hasn't been applied to trunk yet, causing the 2nn to not be logged in at the correct time when it makes an RPC call. This JIRA is to forward port the bugfix.
HDFS-1318. Major new feature reported by sureshms and fixed by tanping
HDFS Namenode and Datanode WebUI information needs to be accessible programmatically for scripts

Currently Namenode and Datanode web page needs to be scraped by scripts to get this information. Having an interface where this structured information is provided, will help building scripts around it.
HDFS-1317. Major bug reported by rohini and fixed by rohini (contrib/hdfsproxy)
HDFSProxy needs additional changes to work after changes to streamFile servlet in HDFS-1109

Before HDFS-1109, streamFile had filename passed as a parameter in the request. With HDFS-1109, the filename is part of the request path similar to listPaths and data servlets. The AuthorizationFilter in HdfsProxy needs updating to pick up the path from the request path instead of looking for filename parameter.
HDFS-1315. Major improvement reported by sureshms and fixed by sureshms (name-node, tools)
Add fsck event to audit log and remove other audit log events corresponding to FSCK listStatus and open calls

In Yahoo production cluster audit logs more than 50% of entries comes from FSCK. FSCK makes same {{Namenode}} method calls as clients. Because of this, the operations are logged in audit logs similar to operations invoked by the client. I want to make a change to namenode not to log operations invoked by FSCK in audit logs.
HDFS-1310. Major test reported by sureshms and fixed by rash37 (hdfs client)
TestFileConcurrentReader fails

For details of test failure see http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/testReport/
HDFS-1308. Major bug reported by boryas and fixed by boryas
job conf key for the services name of DelegationToken for HFTP url is constructed incorrectly in HFTPFileSystem (part of MR-1718)

change HFTP init code that checks for existing delegation tokens
HDFS-1307. Major improvement reported by sureshms and fixed by sureshms (name-node, tools)
Add start time, end time and total time taken for FSCK to FSCK report

FSCK is a long running operation and makes namenode very busy when it runs. Adding information such as start time, end time and time taken helps in determining when the FSCK was run and the impact of that on Namenode.
HDFS-1304. Major improvement reported by szetszwo and fixed by szetszwo (test)
There is no unit test for HftpFileSystem.open(..)

HftpFileSystem.open(..) first opens an URL connection to namenode's FileDataServlet and then is redirected to datanode's StreamFile servlet. Such redirection does not work in the unit test environment because the redirect URL uses real hostname instead of localhost.

One way to get around it is to use fault-injection in order to replace the real hostname with localhost.
HDFS-1302. Major improvement reported by jnp and fixed by jnp
Use writeTokenStorageToStream method in Credentials to store credentials to a file.

This jira covers hdfs changes correspoding to HADOOP-6861 and MAPREDUCE-1566.
HDFS-1301. Major bug reported by boryas and fixed by boryas
TestHDFSProxy need to use server side conf for ProxyUser stuff.

currently TestHdfsProxy sets hadoop.proxyuser.USER.groups in local copy of configuration.
But ProxyUsers only looks at the server side config.
For test we can uses static method in ProxyUsers to load the config.
HDFS-1298. Major improvement reported by sureshms and fixed by sureshms
Add support in HDFS to update statistics that tracks number of file system operations in FileSystem

See HADOOP-6859 for the new statistics.
HDFS-1297. Trivial improvement reported by midojeff and fixed by midojeff (documentation)
Fix some comments
HDFS-1296. Major bug reported by boryas and fixed by boryas
using delegation token over hftp for long running clients (spawn from hdfs-1007).

this patch incorporates patches
https://issues.apache.org/jira/secure/attachment/12446280/hdfs-1007-long-running-hftp-client.patch
and
https://issues.apache.org/jira/secure/attachment/12446362/hdfs-1007-securityutil-fix.patch

patches are attached.
HDFS-1289. Major bug reported by kzhang and fixed by kzhang (data-node)
Datanode secure mode is broken

HDFS-520 introduced a new DataNode constructor, which tries to set up an RPC connection to the NN before a Kerberos login is done. This causes datanode to fail.
HDFS-1284. Major bug reported by shv and fixed by kzhang (test)
TestBlockToken fails

Hudson runs fail several tests. {{TestBlockToken.testBlockTokenRpc}} is one of them.
[See here|http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/423/testReport/junit/org.apache.hadoop.hdfs.security.token.block/TestBlockToken/testBlockTokenRpc/]
HDFS-1283. Major bug reported by jghoman and fixed by jghoman (build)
ant eclipse-files has drifted again

Library versions specified by ant eclipse-files have drifted from those used by ivy again. Until HDFS-1035 meanders into the code, we need to make sure we keep these updated, otherwise bedlam breaks loose for Eclipse developers.
HDFS-1272. Major improvement reported by jnp and fixed by jnp
HDFS changes corresponding to rename of TokenStorage to Credentials

TokenStorage is renamed to Credentials as part of MAPREDUCE-1528 and HADOOP-6845. This jira tracks hdfs changes corresponding to that.
HDFS-1250. Major bug reported by sureshms and fixed by sureshms (name-node)
Namenode accepts block report from dead datanodes

When a datanode heartbeat times out namenode marks it dead. The subsequent heartbeat from the datanode is rejected with a command to datanode to re-register. However namenode accepts block report from the datanode although it is marked dead.
HDFS-1205. Major improvement reported by tlipcon and fixed by tlipcon (data-node)
FSDatasetAsyncDiskService should name its threads

FSDatasetAsyncService creates threads but doesn't name them. The ThreadFactory should name them with the volume they work on as well as a thread index.
HDFS-1203. Major improvement reported by tlipcon and fixed by tlipcon (data-node)
DataNode should sleep before reentering service loop after an exception

When the DN gets an exception in response to a heartbeat, it logs it and continues, but there is no sleep. I've occasionally seen bugs produce a case where heartbeats continuously produce exceptions, and thus the DN floods the NN with bad heartbeats. Adding a 1 second sleep at least throttles the error messages for easier debugging and error isolation.
HDFS-1202. Major bug reported by tlipcon and fixed by tlipcon (data-node)
DataBlockScanner throws NPE when updated before initialized

Missing an isInitialized() check in updateScanStatusInternal
HDFS-1201. Major improvement reported by jnp and fixed by kzhang
Support for using different Kerberos keys for Namenode and datanode.

This jira covers the hdfs changes to support different Kerberos keys for Namenode and datanode. This corresponds to changes in HADOOP-6632
HDFS-1198. Major bug reported by jnp and fixed by jnp (name-node)
Resolving cross-realm principals

This jira covers hdfs changes corresponding to HADOOP-6603. Kerberos has bug in resolving cross realm principals. This jira provides a work-around for that.
HDFS-1192. Major bug reported by boryas and fixed by boryas
refreshSuperUserGroupsConfiguration should use server side configuration for the refresh (for HADOOP-6815)

Currently refreshSuperUserGroupsConfiguration is using client side Configuration.
One of the issues with this is that if the cluster is restarted it will loose the "refreshed' values (unless they are copied to the NameNode/JobTracker machine).
HDFS-1190. Minor improvement reported by midojeff and fixed by midojeff (data-node)
Remove unused getNamenode() method from DataNode.

This patch removes the getNamenode() method from DataNode, which appears to be unused.
HDFS-1187. Major improvement reported by owen.omalley and fixed by owen.omalley (security)
Modify fetchdt to allow renewing and canceling token

I would like to extend fetchdt to allow renewing and canceling tokens.
HDFS-1185. Minor improvement reported by midojeff and fixed by midojeff (data-node, name-node)
Remove duplicate now() functions in DataNode, FSNamesystem

An identical now() function is defined in Util, DataNode, and FSNamesystem. This patch removes the latter two and converts all calls to use the Util.now function.
HDFS-1184. Minor improvement reported by midojeff and fixed by midojeff
Replace tabs in code with spaces

Replace some code indentation that uses tabs with spaces, to match the coding convention
HDFS-1183. Minor improvement reported by midojeff and fixed by midojeff (name-node)
Remove some duplicate code in NamenodeJspHelper.java

This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java.
HDFS-1178. Major improvement reported by owen.omalley and fixed by owen.omalley (name-node)
The NameNode servlets should not use RPC to connect to the NameNode

Currently some of the NameNode servlets use RPC to connect from the NameNode to itself. They should do it more directly with the NameNode object.
HDFS-1167. Major task reported by vinaythota and fixed by vinaythota (test)
New property for local conf directory in system-test-hdfs.xml file.

Adding new property in system-test.xml file for local configuration directory.
HDFS-1164. Major bug reported by eli and fixed by tlipcon (contrib/hdfsproxy)
TestHdfsProxy is failing

TestHdfsProxy is failing on trunk, seen in HDFS-1132 and HDFS-1143. It doesn't look like hudson posts test results for contrib and it's hard to see what's going on from the raw console output. Can someone with access to hudson upload the individual test output for TestHdfsProxy so we can see what the issue is?
HDFS-1163. Major bug reported by boryas and fixed by boryas
normalize property names for JT/NN kerberos principal names in configuration (from HADOOP 6633)
HDFS-1160. Major improvement reported by eli and fixed by eli (data-node)
Improve some FSDataset warnings and comments

The warning in FSDataset that prints "removing block" is more clear as "Removing replica info for block" since the DN is not actually removing the block. The attached patch also cleans up and fixes typos in some related comments and code.
HDFS-1157. Major bug reported by cos and fixed by cos (test)
Modifications introduced by HDFS-1150 are breaking aspect's bindings

Modifications introduced by HDFS-1150 to DataNode class brakes the binding of some of Herriot (test framework) bindings. This JIRA is to track the fix.
HDFS-1150. Major new feature reported by jghoman and fixed by jghoman (data-node)
Verify datanodes' identities to clients in secure clusters

Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves.
HDFS-1146. Major bug reported by jnp and fixed by jnp
Javadoc for getDelegationTokenSecretManager in FSNamesystem

Javadoc is missing for public method getDelegationTokenSecretManager in FSNamesystem.
HDFS-1145. Major bug reported by dhruba and fixed by dhruba (name-node)
When NameNode is shutdown it tries to exit safemode

Suppose the NameNode is in safemode. Then we try to shuut it down by invoking NameNode.stop(). The stop() method interrupts all waiting threads, which in turn, causes the SafeMode monitor to exit and thus triggering replicating/deleting of blocks.
HDFS-1141. Blocker bug reported by tlipcon and fixed by tlipcon (name-node)
completeFile does not check lease ownership

completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139.
HDFS-1140. Minor improvement reported by dms and fixed by dms (name-node)
Speedup INode.getPathComponents

When the namenode is loading the image there is a significant amount of time being spent in the DFSUtil.string2Bytes. We have a very specific workload here. The path that namenode does getPathComponents for shares N - 1 component with the previous path this method was called for (assuming current path has N components).
Hence we can improve the image load time by caching the result of previous conversion.
We thought of using some simple LRU cache for components, but the reality is, String.getBytes gets optimized during runtime and LRU cache doesn't perform as well, however using just the latest path components and their translation to bytes in two arrays gives quite a performance boost.
I could get another 20% off of the time to load the image on our cluster (30 seconds vs 24) and I wrote a simple benchmark that tests performance with and without caching.
HDFS-1138. Major bug reported by dms and fixed by dms
Modification times are being overwritten when FSImage loads

A very easy way to spot the bug is to do a second restart in TestRestartDFS and check that the modification time on root is the same as it was before the second restart.

The problem is modifying time of the parent if the modification time of the child is greater than parent's in addToParent.

So if you have /DIR/File then on creation of a file modification time of the DIR will be set, but on cluster restart, or when secondary is checkpointing and reading the image it will add DIR to "/" and write the new modification time for "/" which is the modification time of DIR.

This is clearly a bug. I will attach a patch with one more parameter being passed from the loadFSImage that says to not propagate the time.
HDFS-1132. Minor test reported by eli and fixed by eli (test)
Refactor TestFileStatus

When writing HADOOP-6585 I noticed TestFileStatus is an old test with just one monolithic test function. We should rewrite it as a junit4 style test and break up the functions into setup, teardown, and five individual test cases. Same tests being run, just easier to follow, modify and add news tests. Patch coming.
HDFS-1130. Major bug reported by amareshwari and fixed by devaraj (security)
Pass Administrator acl to HTTPServer for common servlet access.

Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer is constructed.
HDFS-1119. Major sub-task reported by szetszwo and fixed by szetszwo (name-node)
Refactor BlocksMap with GettableSet

The data structure required in BlocksMap is a GettableSet. See also [this comment|https://issues.apache.org/jira/browse/HDFS-1114?focusedCommentId=12862118&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12862118].
HDFS-1118. Major bug reported by zshao and fixed by zshao
DFSOutputStream socket leak when cannot connect to DataNode

The offending code is in {{DFSOutputStream.nextBlockOutputStream}}

This function retries several times to call {{createBlockOutputStream}}. Each time when it fails, it leaves a {{Socket}} object in {{DFSOutputStream.s}}.
That object is never closed, but overwritten the next time {{createBlockOutputStream}} is called.
HDFS-1114. Major improvement reported by szetszwo and fixed by szetszwo (name-node)
Reducing NameNode memory usage by an alternate hash table

NameNode uses a java.util.HashMap to store BlockInfo objects. When there are many blocks in HDFS, this map uses a lot of memory in the NameNode. We may optimize the memory usage by a light weight hash table implementation.
HDFS-1112. Major bug reported by hairong and fixed by hairong (name-node)
Edit log buffer should not grow unboundedly

Currently HDFS does not impose an upper limit on the edit log buffer. In case there are a large number of open operations coming in with access time update on, since open does not call sync automatically, there is a possibility that the buffer grow to a large size, therefore causes memory leak and full GC in extreme cases as described in HDFS-1104.

The edit log buffer should be automatically flushed when the buffer becomes full.
HDFS-1111. Major new feature reported by rschmidt and fixed by sriramsrao
getCorruptFiles() should give some hint that the list is not complete

If the list of corruptfiles returned by the namenode doesn't say anything if the number of corrupted files is larger than the call output limit (which means the list is not complete). There should be a way to hint incompleteness to clients.

A simple hack would be to add an extra entry to the array returned with the value null. Clients could interpret this as a sign that there are other corrupt files in the system.

We should also do some rephrasing of the fsck output to make it more confident when the list is not complete and less confident when the list is known to be incomplete.
HDFS-1110. Major improvement reported by sureshms and fixed by sureshms
Namenode heap optimization - reuse objects for commonly used file names

There are a lot of common file names used in HDFS, mainly created by mapreduce, such as file names starting with "part". Reusing byte[] corresponding to these recurring file names will save significant heap space used for storing the file names in millions of INodeFile objects.
HDFS-1109. Major bug reported by dms and fixed by dms (contrib/hdfsproxy, data-node)
HFTP and URL Encoding

We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.

The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
HDFS-1096. Major new feature reported by boryas and fixed by boryas (security)
allow dfsadmin/mradmin refresh of superuser proxy group mappings
HDFS-1093. Major improvement reported by dhruba and fixed by dhruba (name-node)
Improve namenode scalability by splitting the FSNamesystem synchronized section in a read/write lock

Most critical data structures in the NameNode (NN) are protected by a syncronized methods in the FSNamesystem class. This essentially makes critical code paths in the NN single-threaded. However, a large percentage of the NN calls are listStatus, getBlockLocations, etc which do not change internal data structures at all, these are read-only calls. If we change the FSNamesystem lock to a read/write lock, many of the above operations can occur in parallel, thus improving the scalability of the NN.
HDFS-1085. Major bug reported by knoguchi and fixed by szetszwo (data-node)
hftp read failing silently

When performing a massive distcp through hftp, we saw many tasks fail with

2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:159)

This means that read itself didn't fail but the resulted file was somehow smaller.
HDFS-1081. Major improvement reported by jghoman and fixed by jghoman (security)
Performance regression in DistributedFileSystem::getFileBlockLocations in secure systems

We've seen a significant decrease in the performance of DistributedFileSystem::getFileBlockLocations() with security turned on Y20. This JIRA is for correcting and tracking it both on Y20 and trunk.
HDFS-1080. Major bug reported by jghoman and fixed by jghoman (name-node)
SecondaryNameNode image transfer should use the defined http address rather than local ip address

Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr.
HDFS-1079. Major new feature reported by sureshms and fixed by sureshms (name-node)
HDFS implementation should throw exceptions defined in AbstractFileSystem

HDFS implementation Hdfs.java should throw exceptions as defined in AbstractFileSystem. To facilitate this, ClientProtocol should be changed to throw specific exceptions, as defined in AbstractFileSystem.
HDFS-1071. Major sub-task reported by dhruba and fixed by dms (name-node)
savenamespace should write the fsimage to all configured fs.name.dir in parallel

If you have a large number of files in HDFS, the fsimage file is very big. When the namenode restarts, it writes a copy of the fsimage to all directories configured in fs.name.dir. This takes a long time, especially if there are many directories in fs.name.dir. Make the NN write the fsimage to all these directories in parallel.
HDFS-1061. Minor improvement reported by bharathm and fixed by bharathm (name-node)
Memory footprint optimization for INodeFile object.

I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal.
HDFS-1055. Major improvement reported by tlipcon and fixed by tlipcon (data-node)
Improve thread naming for DataXceivers

The DataXceiver threads are named using the default Daemon naming, which is Runnable.toString(). Currently this isn't implemented, so threads have names like org.apache.hadoop.hdfs.server.datanode.DataXceiver@579c9a6b. It would be very handy for debugging (and even ops maybe) to have a better name like "DataXceiver for client 1.2.3.4 [reading block_234254242]"
HDFS-1045. Major bug reported by jghoman and fixed by jghoman (security)
In secure clusters, re-login is necessary for https clients before opening connections

Ticket credentials expire and therefore clients opening https connections (only the NN and SNN doing image/edits exchange) should re-login before opening those connections.
HDFS-1044. Major bug reported by boryas and fixed by boryas
Cannot submit mapreduce job from secure client to unsecure sever

Looks like it tries to get DelegationToken and fails because SecureManger on Server doesn't start in non-secure environment.
HDFS-1039. Major bug reported by jnp and fixed by jnp
Service should be set in the token in JspHelper.getUGI

The delegation token added to the UGI in getUGI method in the JspHelper does not have service set. Therefore, this token cannot be used to connect to the namenode.
HDFS-1038. Major bug reported by jnp and fixed by jnp
In nn_browsedfscontent.jsp fetch delegation token only if security is enabled.

nn_browsedfscontent.jsp calls getDelegationToken even if security is disabled, which causes NPE.
HDFS-1036. Major bug reported by boryas and fixed by boryas
in DelegationTokenFetch dfs.getURI returns no port

dfs.getUri().getPort() returns -1.
HDFS-1035. Major improvement reported by tomwhite and fixed by nidaley (build)
Generate Eclipse's .classpath file from Ivy config

HDFS companion issue for HADOOP-6407.
HDFS-1033. Major improvement reported by jghoman and fixed by jghoman (name-node, security)
In secure clusters, NN and SNN should verify that the remote principal during image and edits transfer

Currently anyone can connect and download image/edits from Namenode. In a secure cluster we can verify the identity of the principal making the request; we should disallow requests from anyone except the NN and SNN principals (and their hosts due to the lousy KerbSSL limitation).
HDFS-1028. Minor improvement reported by tlipcon and fixed by dms (name-node)
INode.getPathNames could split more efficiently

INode.getPathnames uses String.split(String) which actually uses the full Java regex implementation. Since we're always splitting on a single char, we could implement a faster one like StringUtils.split() (except without the escape character). This takes a significant amount of CPU during FSImage loading so should be a worthwhile speedup.
HDFS-1027. Trivial bug reported by raviphulari and fixed by raviphulari
Update year to 2010.

Copyright year needs to be updated from 2009 to 2010.
HDFS-1023. Major improvement reported by jghoman and fixed by jghoman (name-node)
Allow http server to start as regular principal if https principal not defined.

Currently limitations in Sun's KerbSSL implementation require the https server to be run as "host/[machine]@realm." and another Sun KerbSSL limitation appears to require you to store all principals in the same keytab, meaning fully functional, secured Namenodes require combined keytabs. However, it may be that one wishes to run a namenode without a secondary namenode or other utilities that require https. In this case, we should allow the http server to start and log a warning that it will not be able to accept https connections.
HDFS-1021. Major bug reported by boryas and fixed by boryas (security)
specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in DFSAdmin (for HADOOP-6612)
HDFS-1019. Major bug reported by jnp and fixed by jnp
Incorrect default values for delegation tokens in hdfs-default.xml

The default values for delegation token parameters in hdfs-default.xml are incorrect.
HDFS-1017. Major bug reported by jghoman and fixed by jghoman (security)
browsedfs jsp should call JspHelper.getUGI rather than using createRemoteUser()

Currently the JSP for browsing the file system calls getRemoteUser(), which doesn't correctly auth the user on the web, causing failures when trying to browse the web. It should call the utility method JspHelper.getUGI instead.
HDFS-1007. Major bug reported by devaraj and fixed by devaraj (security)
HFTP needs to be updated to use delegation tokens

HFTPFileSystem should be updated to use the delegation tokens so that it can talk to the secure namenodes.
HDFS-1006. Major bug reported by boryas and fixed by boryas
getImage/putImage http requests should be https for the case of security enabled.

should use https:// and port 50475
HDFS-1005. Major new feature reported by jnp and fixed by boryas
Fsck security

This jira tracks implementation of security for Fsck. Fsck should make an authenticated connection to the namenode.
HDFS-1004. Major new feature reported by jghoman and fixed by jghoman (name-node)
Update NN to support Kerberized SSL from HADOOP-6584

Namenode needs to be tweaked to use the new kerberized-back ssl connector.
HDFS-1003. Major new feature reported by boryas and fixed by boryas
authorization checks for inter-server protocol (based on HADOOP-6600)

authorization checks for inter-server protocol (based on HADOOP-6600)
HDFS-1001. Minor bug reported by bcwalrus and fixed by bcwalrus (data-node)
DataXceiver and BlockReader disagree on when to send/recv CHECKSUM_OK

Running the TestPread with additional debug statements reveals that the BlockReader sends CHECKSUM_OK when the DataXceiver doesn't expect it. Currently it doesn't matter since DataXceiver closes the connection after each op, and CHECKSUM_OK is the last thing on the wire. But if we want to cache connections, they need to agree on the exchange of CHECKSUM_OK.
HDFS-992. Major new feature reported by kzhang and fixed by kzhang (security)
Re-factor block access token implementation to conform to the generic Token interface in Common

This makes it possible to use block access token as shared key for client-to-datanode authentication over RPC. However, access authorization is still based on block access token semantics.
HDFS-988. Blocker bug reported by dhruba and fixed by eli (name-node)
saveNamespace race can corrupt the edits log

The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20.

https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853
HDFS-982. Blocker test reported by eli and fixed by pocheung (contrib/hdfsproxy, security)
TestDelegationToken#testDelegationTokenWithRealUser is failing

Hudson is reporting that TestDelegationToken#testDelegationTokenWithRealUser is failing on trunk.
HDFS-981. Blocker test reported by eli and fixed by cos (contrib/hdfsproxy)
test-contrib fails due to test-cactus failure

Relevant output from a recent run http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/232/console

[exec] BUILD FAILED
[exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:568: The following error occurred while executing this line:
[exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line:
[exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:292: org.codehaus.cargo.container.ContainerException: Failed to download [http://apache.osuosl.org/tomcat/tomcat-6/v6.0.18/bin/apache-tomcat-6.0.18.zip]
HDFS-977. Trivial bug reported by stevel@apache.org and fixed by qwertymaniac (data-node)
DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level

My IDE tells me that this code snippet in {{DataNode.createInterDataNodeProtocolProxy()}} is guarding the info log entry at debug level, and it should be reviewed
if (InterDatanodeProtocol.LOG.isDebugEnabled()) {
InterDatanodeProtocol.LOG.info("InterDatanodeProtocol addr=" + addr);
}
HDFS-970. Critical bug reported by tlipcon and fixed by tlipcon (name-node)
FSImage writing should always fsync before close

Without an fsync, it's common that filesystems will delay the writing of metadata to the journal until all of the data blocks have been flushed. If the system crashes while the dirty pages haven't been flushed, the file is left in an indeterminate state. In some FSs (eg ext4) this will result in a 0-length file. In others (eg XFS) it will result in the correct length but any number of data blocks getting zeroed. Calling FileChannel.force before closing the FSImage prevents this issue.
HDFS-947. Major improvement reported by dhruba and fixed by dms
The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas

A client that uses the Hftp protocol to read a file is redirected by the namenode to a random datanode. It would be nice if the client gets redirected to a datanode that has the maximum number of local replicas of the blocks of the file.
HDFS-941. Major improvement reported by tlipcon and fixed by bcwalrus (data-node, hdfs client, performance)
Datanode xceiver protocol should allow reuse of a connection

Right now each connection into the datanode xceiver only processes one operation.

In the case that an operation leaves the stream in a well-defined state (eg a client reads to the end of a block successfully) the same connection could be reused for a second operation. This should improve random read performance significantly.
HDFS-912. Minor bug reported by aw and fixed by aw
sed in build.xml fails

This is the HDFS version of HADOOP-6505.
HDFS-903. Critical bug reported by eli and fixed by hairong (name-node)
NN should verify images and edit logs on startup

I was playing around with corrupting fsimage and edits logs when there are multiple dfs.name.dirs specified. I noticed that:
* As long as your corruption does not make the image invalid, eg changes an opcode so it's an invalid opcode HDFS doesn't notice and happily uses a corrupt image or applies the corrupt edit.
* If the first image in dfs.name.dir is "valid" it replaces the other copies in the other name.dirs, even if they are different, with this first image, ie if the first image is actually invalid/old/corrupt metadata than you've lost your valid metadata, which can result in data loss if the namenode garbage collects blocks that it thinks are no longer used.

How about we maintain a checksum as part of the image and edit log and check those on startup and refuse to startup if they are different. Or at least provide a configuration option to do so if people are worried about the overhead of maintaining checksums of these files. Even if we assume dfs.name.dir is reliable storage this guards against operator errors.
HDFS-900. Blocker bug reported by tlipcon and fixed by shv
Corrupt replicas are not tracked correctly through block report from DN

This one is tough to describe, but essentially the following order of events is seen to occur:

# A client marks one replica of a block to be corrupt by telling the NN about it
# Replication is then scheduled to make a new replica of this node
# The replication completes, such that there are now 3 good replicas and 1 corrupt replica
# The DN holding the corrupt replica sends a block report. Rather than telling this DN to delete the node, the NN instead marks this as a new *good* replica of the block, and schedules deletion on one of the good replicas.

I don't know if this is a dataloss bug in the case of 1 corrupt replica with dfs.replication=2, but it seems feasible. I will attach a debug log with some commentary marked by '============>', plus a unit test patch which I can get to reproduce this behavior reliably. (it's not a proper unit test, just some edits to an existing one to show it)
HDFS-895. Major improvement reported by dhruba and fixed by tlipcon (hdfs client)
Allow hflush/sync to occur in parallel with new writes to the file

In the current trunk, the HDFS client methods writeChunk() and hflush./sync are syncronized. This means that if a hflush/sync is in progress, an applicationn cannot write data to the HDFS client buffer. This reduces the write throughput of the transaction log in HBase.

The hflush/sync should allow new writes to happen to the HDFS client even when a hflush/sync is in progress. It can record the seqno of the message for which it should receice the ack, indicate to the DataStream thread to star flushing those messages, exit the synchronized section and just wai for that ack to arrive.
HDFS-884. Minor improvement reported by stevel@apache.org and fixed by stevel@apache.org (data-node)
DataNode makeInstance should report the directory list when failing to start up

When {{Datanode.makeInstance()}} cannot work with one of the directories in dfs.data.dir, it logs this at warn level (while losing the stack trace).

It should include the nested exception for better troubleshooting. Then, when all dirs in the list fail, an exception is thrown, but this exception does not include the list of directories. It should list the absolute path of every missing/failing directory, so that whoever sees the exception can see where to start looking for problems: either the filesystem or the configuration.
HDFS-882. Minor improvement reported by stevel@apache.org and fixed by stevel@apache.org (data-node)
Datanode could log what hostname and ports its listening on on startup

The Datanode logs the port it is listening too, but if you have having problems with multi-NIC systems or IPv6 grief, knowing the hostname is handy too. Also, it doesn't log the SSL port info.

Proposed: provide more details at the debug level
HDFS-881. Major improvement reported by tlipcon and fixed by tlipcon
Refactor DataNode Packet header into DataTransferProtocol

The Packet Header format is used ad-hoc in various places. This JIRA is to refactor it into a class inside DataTransferProtocol (like was done with PipelineAck)
HDFS-874. Major bug reported by tlipcon and fixed by tlipcon (hdfs client, test)
TestHDFSFileContextMainOperations fails on weirdly configured DNS hosts

On an internal build machine I see exceptions like this:
java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:47262/data/1/scratch/patchqueue/patch-worker-20518/patch_21/svnrepo/build/test/data/test/test/testRenameWithQuota/srcdir, expected: hdfs://localhost.localdomain:47262

"hostname" and "hostname -f" both show the machine's FQDN (not localhost). /etc/hosts is stock after CentOS 5 install. "host 127.0.0.1" reverses to "localhost"
HDFS-853. Major improvement reported by dhruba and fixed by dms (name-node)
The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage

It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
HDFS-829. Major bug reported by aw and fixed by aw
hdfsJniHelper.c: #include <error.h> is not portable

hdfsJniHelper.c includes <error.h> but this appears to be unnecessary, since even under Linux none of the routines that are prototyped are used. Worse yet, error.h doesn't appear to be a standard header file so this breaks on Mac OS X and Solaris and prevents libhdfs from being built.
HDFS-811. Minor new feature reported by raviphulari and fixed by eli (test)
Add metrics, failure reporting and additional tests for HDFS-457

HDFS-457 introduced a improvement which allows datanode to continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed.

I suggest following additional tests for this improvement.

#1 Test successive volume failures ( Minimum 4 volumes )
#2 Test if each volume failure reports reduction in available DFS space and remaining space.
#3 Test if failure of all volumes on a data nodes leads to the data node failure.
#4 Test if correcting failed storage disk brings updates and increments available DFS space.
HDFS-752. Major new feature reported by sureshms and fixed by sureshms
Add interface classification stable & scope to HDFS

This jira addresses adding interface classification for the classes in hadoop hdfs, based on the mechanism described in Hadoop-5073.
HDFS-727. Blocker bug reported by eli and fixed by eli (libhdfs)
bug setting block size hdfsOpenFile

In hdfsOpenFile in libhdfs invokeMethod needs to cast the block size argument to a jlong so a full 8 bytes are passed (rather than 4 plus some garbage which causes writes to fail due to a bogus block size).
HDFS-718. Minor improvement reported by andrewr and fixed by andrewr (name-node)
configuration parameter to prevent accidental formatting of HDFS filesystem

Currently, any time the NameNode is not running, an HDFS filesystem will accept the 'format' command, and will duly format itself. There are those of us who have multi-PB HDFS filesystems who are really quite uncomfortable with this behavior. There is "Y/N" confirmation in the format command, but if the formatter genuinely believes themselves to be doing the right thing, the filesystem will be formatted.

This patch adds a configuration parameter to the namenode, dfs.namenode.support.allowformat, which defaults to "true," the current behavior: always allow formatting if the NameNode is down or some other process is not holding the namenode lock. But if dfs.namenode.support.allowformat is set to "false," the NameNode will not allow itself to be formatted until this config parameter is changed to "true".

The general idea is that for production HDFS filesystems, the user would format the HDFS once, then set dfs.namenode.support.allowformat to "false" for all time.

The attached patch was generated against trunk and +1's on my test machine. We have a 0.20 version that we are using in our cluster as well.
HDFS-712. Major task reported by eli and fixed by eli (libhdfs)
Move libhdfs from mr to hdfs

Here's an hdfs jira for MAPREDUCE-665. During the project split libhdfs was put in the mapreduce repo instead of hdfs, lets move it to hdfs.
HDFS-708. Major new feature reported by shv and fixed by harlowja (test, tools)
A stress-test tool for HDFS.

It would be good to have a tool for automatic stress testing HDFS, which would provide IO-intensive load on HDFS cluster.
The idea is to start the tool, let it run overnight, and then be able to analyze possible failures.
HDFS-697. Major test reported by eli and fixed by eli
Enable asserts for tests by default

See HADOOP-6309. Let's make the tests run with java asserts by default.
HDFS-696. Major test reported by eli and fixed by eli (test)
Java assertion failures triggered by tests

Re-purposing as catch-all ticket for assertion failures when running tests with java asserts enabled. Running with the attached patch on trunk@823732 the following tests all trigger assertion failures:

TestAccessTokenWithDFS
TestInterDatanodeProtocol
TestBackupNode
TestBlockUnderConstruction
TestCheckpoint
TestNameEditsConfigs
TestStartup
TestStorageRestore
HDFS-671. Blocker bug reported by jnp and fixed by tomwhite
Documentation change for updated configuration keys.

HDFS-531, HADOOP-6233 and HDFS-631 have resulted in changes in several config keys. The hadoop documentation needs to be updated to reflect those changes.
HDFS-613. Major bug reported by shv and fixed by tlipcon (test)
TestBalancer and TestBlockTokenWithDFS fail Balancer assert

Running TestBalancer with asserts on. The asserts in {{Balancer.chooseNode()}} is triggered and the test fails. We do not see it in the builds because asserts are off there. So either the assert is irrelevant or there is another bug in the Balancer code.
HDFS-599. Major improvement reported by dhruba and fixed by dms (name-node)
Improve Namenode robustness by prioritizing datanode heartbeats over client requests

The namenode processes RPC requests from clients that are reading/writing to files as well as heartbeats/block reports from datanodes.

Sometime, because of various reasons (Java GC runs, inconsistent performance of NFS filer that stores HDFS transacttion logs, etc), the namenode encounters transient slowness. For example, if the device that stores the HDFS transaction logs becomes sluggish, the Namenode's ability to process RPCs slows down to a certain extent. During this time, the RPCs from clients as well as the RPCs from datanodes suffer in similar fashion. If the underlying problem becomes worse, the NN's ability to process a heartbeat from a DN is severly impacted, thus causing the NN to declare that the DN is dead. Then the NN starts replicating blocks that used to reside on the now-declared-dead datanode. This adds extra load to the NN. Then the now-declared-datanode finally re-establishes contact with the NN, and sends a block report. The block report processing on the NN is another heavyweight activity, thus casing more load to the already overloaded namenode.

My proposal is tha the NN should try its best to continue processing RPCs from datanodes and give lesser priority to serving client requests. The Datanode RPCs are integral to the consistency and performance of the Hadoop file system, and it is better to protect it at all costs. This will ensure that NN recovers from the hiccup much faster than what it does now.
HDFS-556. Major improvement reported by jghoman and fixed by eli (name-node)
Provide info on failed volumes in the web ui

HDFS-457 provided better handling of failed volumes but did not provide a corresponding view of this functionality on the web ui, such as a view of which datanodes have failed volumes. This would be a good feature to have.
HDFS-528. Major new feature reported by tlipcon and fixed by tlipcon (scripts)
Add ability for safemode to wait for a minimum number of live datanodes

When starting up a fresh cluster programatically, users often want to wait until DFS is "writable" before continuing in a script. "dfsadmin -safemode wait" doesn't quite work for this on a completely fresh cluster, since when there are 0 blocks on the system, 100% of them are accounted for before any DNs have reported.

This JIRA is to add a command which waits until a certain number of DNs have reported as alive to the NN.
HDFS-455. Minor improvement reported by pirroh and fixed by pirroh (data-node, name-node)
Make NN and DN handle in a intuitive way comma-separated configuration strings

The following configuration causes problems:
<property>
<name>dfs.data.dir</name>
<value>/mnt/hstore2/hdfs, /home/foo/dfs</value>
</property>

The problem is that the space after the comma causes the second directory for storage to be " /home/foo/dfs" which is in a directory named <SPACE> which contains a sub-dir named "home" in the hadoop datanodes default directory. This will typically cause the user's home partition to fill, but will be very hard for the user to understand since a directory with a whitespace name is hard to understand.

(ripped from HADOOP-2366)
HDFS-330. Trivial improvement reported by aw and fixed by aw (data-node)
Datanode Web UIs should provide robots.txt

There is a potential issue that someone might have an internal corporate crawler that goes through HDFS browser accidentally. It might be a good idea to provide a default robots file that disables crawling. [No, this didn't happen to us. :) ]
HDFS-259. Major sub-task reported by tlipcon and fixed by tlipcon
Remove intentionally corrupt 0.13 directory layout creation

Given that 0.13 is incredibly old at this point, I think the likelihood of anyone trying to start an 0.20+ namenode on an 0.13 directly layout is essentially nil. The intentionally-corrupt "${dfs.name.dir}/image/" directory just serves to confuse new users at this point, so I propose removing it.

If no one objects, I'll submit a patch. If there are objections, I propose at least adding a README explaining its purpose (same text as in the corrupt fsimage file)
HDFS-202. Major new feature reported by acmurthy and fixed by hairong (hdfs client, name-node)
Add a bulk FIleSystem.getFileBlockLocations

Currently map-reduce applications (specifically file-based input-formats) use FileSystem.getFileBlockLocations to compute splits. However they are forced to call it once per file.
The downsides are multiple:
# Even with a few thousand files to process the number of RPCs quickly starts getting noticeable
# The current implementation of getFileBlockLocations is too slow since each call results in 'search' in the namesystem. Assuming a few thousand input files it results in that many RPCs and 'searches'.

It would be nice to have a FileSystem.getFileBlockLocations which can take in a directory, and return the block-locations for all files in that directory. We could eliminate both the per-file RPC and also the 'search' by a 'scan'.

When I tested this for terasort, a moderate job with 8000 input files the runtime halved from the current 8s to 4s. Clearly this is much more important for latency-sensitive applications...
HDFS-96. Major bug reported by dhruba and fixed by pkling
HDFS does not support blocks greater than 2GB

HDFS currently does not support blocks greater than 2GB in size.
MAPREDUCE-3438. Major bug reported by shv and fixed by rvadali (contrib/raid)
TestRaidNode fails because of "Too many open files"

TestRaidNode fails because it opens many connections.
MAPREDUCE-2328. Major bug reported by tlipcon and fixed by qwertymaniac
memory-related configurations missing from mapred-default.xml

HADOOP-5881 added new configuration parameters for memory-based scheduling, but they weren't added to mapred-default.xml
MAPREDUCE-3429. Major bug reported by cos and fixed by cos (contrib/capacity-sched, contrib/gridmix)
Few contrib tests are failing because of the missing commons-lang dependency

As the result of MAPREDUCE-3311 fix a transient {{commons-lang}} isn't available anymore to contrib tests. This causing silent failure with timeout. The problem is only seeing if tests are ran with {{-Dtest.output=yes}}
MAPREDUCE-3311. Major task reported by cos and fixed by cos (build)
Bump jetty to 6.1.26

MapReduce part of HADOOP-7450
MAPREDUCE-3156. Major test reported by cos and fixed by cos (test)
Allow TestMRCLI to be run against a cluster

Mapreduce part of HDFS-1762
MAPREDUCE-3151. Major bug reported by jrottinghuis and fixed by jrottinghuis (contrib/vertica)
Contrib tests failing

Jenkins builds fail:
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-22-branch/80/console
MAPREDUCE-3139. Major bug reported by shv and fixed by jghoman (test)
SlivePartitioner generates negative partitions

{{SlivePartitioner.getPartition()}} returns negative partition numbers on some occasions, which is illegal.
MAPREDUCE-3138. Blocker bug reported by acmurthy and fixed by owen.omalley (client, mrv2)
Allow for applications to deal with MAPREDUCE-954

MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We need a bridge for them to move to 0.23.
MAPREDUCE-3088. Major bug reported by shv and fixed by shv (build)
Clover 2.4.3 breaks build for 0.22 branch

Due to known bug in Clover 2.4.3 build for 0.22 branch is broken.
MAPREDUCE-3039. Major bug reported by jrottinghuis and fixed by jrottinghuis (contrib/capacity-sched, contrib/fair-share, contrib/gridmix, contrib/mrunit, contrib/mumak, contrib/raid, contrib/streaming, jobhistoryserver)
Make mapreduce use same version of avro as HBase

HBase depends on avro 1.5.3 whereas hadoop-common depends on 1.3.2.
When building HBase on top of hadoop, this should be consistent.
Moreover, this should be consistent between common, hdfs, and mapreduce.

Contribs seem to have declared a dependency on avro but are not in fact depending on it.
MAPREDUCE-3026. Major bug reported by mayank_bansal and fixed by mayank_bansal (jobtracker)
When user adds hierarchical queues to the cluster, mapred queue -list returns NULL Pointer Exception

When User adds the hierarchical queues, and try to see them from the command line using
mapred queue -list
It returns Null Pointer Exception.
MAPREDUCE-3025. Blocker bug reported by jrottinghuis and fixed by jrottinghuis (build)
Contribs not building

Contribs are not getting built.
Snippet from Jenkins:

compile:
[subant] No sub-builds to iterate on
MAPREDUCE-2991. Major bug reported by priyomustafi and fixed by priyomustafi (scheduler)
queueinfo.jsp fails to show queue status if any Capacity scheduler queue name has dash/hiphen in it.

If any queue name has a dash/hiphen in it, the queueinfo.jsp doesn't show any queue information. This is happening because the queue name is used to create javascript variables and javascript doesn't allow dash in variable names.
MAPREDUCE-2948. Major bug reported by milindb and fixed by mahadev (contrib/streaming)
Hadoop streaming test failure, post MR-2767

After removing LinuxTaskController in MAPREDUCE-2767, one of the tests in contrib/streaming: TestStreamingAsDifferentUser.java is failing since it imports import org.apache.hadoop.mapred.ClusterWithLinuxTaskController. Patch forthcoming.
MAPREDUCE-2940. Major bug reported by jrottinghuis and fixed by jrottinghuis (build)
Build fails with ant 1.7.0 but works with 1.8.0

contrib builds fail when using Ant 1.7.

build.xml calls build.xml in contrib, which calls block-forensics build, which in turn uses build-contrib.
The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0.
MAPREDUCE-2939. Major task reported by jrottinghuis and fixed by jrottinghuis (build)
Ant setup on hadoop7 jenkins host

From the build error it looks like
a) ant is not set up on the machine
b) $ANT_HOME point to the wrong spot
MAPREDUCE-2779. Major bug reported by mingma and fixed by mingma (job submission)
JobSplitWriter.java can't handle large job.split file

We use cascading MultiInputFormat. MultiInputFormat sometimes generates big job.split used internally by hadoop, sometimes it can go beyond 2GB.
In JobSplitWriter.java, the function that generates such file uses 32bit signed integer to compute offset into job.split.
MAPREDUCE-2767. Blocker bug reported by milindb and fixed by milindb (security)
Remove Linux task-controller from 0.22 branch

There's a potential security hole in the task-controller as it stands. Based on the discussion on general@, removing task-controller from the 0.22 branch will pave way for 0.22.0 release. (This was done for the 0.21.0 release as well: see MAPREDUCE-2014.) We can roll a 0.22.1 release with the task-controller when it is fixed.
MAPREDUCE-2753. Major bug reported by jrottinghuis and fixed by jrottinghuis (build)
Generated POMs hardcode dependency on hadoop-common version 0.22.0-SNAPSHOT

The generated poms inject the version of mapred itself, but hardcode the version of hadoop-common they depend on.
When trying to build downstream projects (HBase), then they will require hadoop-common-0.22.0-SNAPSHOT.jar instead of the version they compiled against.

When trying to do an offline build this will fail to resolve as another hadoop-common has been installed in the local maven repo.
Even during online build, it should compile against the hadoop-common that hdfs compiled against.

When versions mismatch one cannot do a coherent build. That is particularly problematic when making simultaneous change in hadoop-common and hadoop-mapreduce and you want to try this locally before committing each.
MAPREDUCE-2752. Minor bug reported by jrottinghuis and fixed by jrottinghuis (build)
Build does not pass along properties to contrib builds

Subant call to compile contribs do not pass along parameters from parent build.
Properties such as hadoop-common.version, asfrepo, offline, etc. are not passed along.
Result is that build not connected to Internet fails, hdfs proxy refuses to build against own recently built common but rather downloads 0.22-SNAPSHOT from apache again.
MAPREDUCE-2571. Blocker bug reported by sinofool and fixed by sinofool
CombineFileInputFormat.getSplits throws a java.lang.ArrayStoreException

The getSplits methods of
org.apache.hadoop.mapred.lib.CombineFileInputFormat
not work.

...mapred.lib.CombineFileInputFormat(0.20-style) is a proxy for ...mapreduce.lib.input.CombineFileInputFormat(0.21-style)

The 0.21-style getSplits returns ArrayList<...mapreduce.lib.input.CombineFileSplit>
and the 0.20-style delegation calls toArray(...mapred.InputSplit[])

The ...mapreduce.lib.input.CombineFileSplit is based on ...mapreduce.InputSplit
and ...mapred.InputSplit is a interface, not a super-class of ...mapreduce.InputSplit
MAPREDUCE-2531. Blocker bug reported by revans2 and fixed by revans2 (client)
org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast exception

When using a combination of the mapred and mapreduce APIs (PIG) it is possible to have the following exception

Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.JobID cannot be cast to
org.apache.hadoop.mapred.JobID
at org.apache.hadoop.mapred.jobcontrol.Job.getAssignedJobID(Job.java:71)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:239)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1325)
... 29 more

This is because the JobID is just downcast. It should be calling JobID.downgrade
MAPREDUCE-2516. Minor bug reported by asrabkin and fixed by asrabkin
option to control sensitive web actions

as per HADOOP-7302, webinterface.private.actions should not be in trunk. But it should be here, and should have a clearer name.
MAPREDUCE-2515. Major bug reported by asrabkin and fixed by asrabkin (jobtracker)
MapReduce references obsolete options

Option topology.node.switch.mapping.impl has been renamed to net.topology.node.switch.mapping.impl; JT still uses old name. Likewise, JT uses old names for several other since-renamed options.
MAPREDUCE-2505. Minor improvement reported by matei and fixed by matei (contrib/fair-share, documentation)
Explain how to use ACLs in the fair scheduler

The fair scheduler already works with the ACL system introduced through the mapred.queue.* parameters, but the documentation doesn't explain how to use this. We should add a paragraph or two about it.
MAPREDUCE-2502. Trivial improvement reported by eli and fixed by eli (job submission)
JobSubmitter should use mapreduce.job.maps

JobSubmitter should use {{mapreduce.job.maps}} instead of the deprecated {{mapred.map.tasks}}.
MAPREDUCE-2487. Minor bug reported by forvines and fixed by devaraj.k
ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE

On line 293 of o.a.h.mapred.lib.Chain in setReducer(...):

reducerConf.setBoolean(MAPPER_BY_VALUE, byValue);

this should be REDUCER_BY_VALUE.

http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293
MAPREDUCE-2486. Blocker bug reported by dvryaboy and fixed by tlipcon
0.22 - snapshot incorrect dependency published in .pom files

The pom at https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.22.0-SNAPSHOT/ publishes a dependency on hadoop-common version "0.22.0-dev-SNAPSHOT" while hadoop-common only publishes "0.22.0-SNAPSHOT" (no -dev).
MAPREDUCE-2472. Major bug reported by tlipcon and fixed by atm (task-controller)
Extra whitespace in mapred.child.java.opts breaks JVM initialization

When creating taskjvm.sh, we split mapred.child.java.opts on " " and then create a quoted argument for each of those results. So, if you have an extra space anywhere in this configuration, you get an argument '' in the child command line, which the JVM interprets as an empty class name. This results in a ClassNotFoundException and the task cannot run.
MAPREDUCE-2457. Critical bug reported by tucu00 and fixed by tucu00 (jobtracker)
job submission should inject group.name (on the JT side)

Until Hadoop 0.20, the JobClient was injecting the property 'group.name' on the JobConf submitted to the JobTracker.
Since Hadoop 0.21, due to security related changes, this is not done anymore.
This breaks backwards compatibility for jobs/components that expect the 'group.name' to be automatically set at submission time.
An example of a component being affected by this change is the FairScheduler where it is common to use the group.name as pool name. Different from other properties, a special characteristic of the group.name is that its value cannot be tampered by a user.
For security reasons this should not be done (as it was done before) in the JobClient side. Instead, it should be done in the JobTracker when the JobConf is received.
MAPREDUCE-2448. Minor bug reported by szetszwo and fixed by eli (contrib/raid, test)
NoSuchMethodError: org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(..)

java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(Ljava/lang/String;I)Z
at org.apache.hadoop.raid.TestBlockFixer.corruptBlock(TestBlockFixer.java:643)
at org.apache.hadoop.raid.TestBlockFixer.implBlockFix(TestBlockFixer.java:189)
at org.apache.hadoop.raid.TestBlockFixer.testBlockFixLocal(TestBlockFixer.java:139)
MAPREDUCE-2445. Major bug reported by tlipcon and fixed by tlipcon (security, test)
TestMiniMRWithDFSWithDistinctUsers is very broken

This test has a number of issues:
- it side steps the normal job submission API for no apparent reason, manually writing splits file and uploading submission files. (but forgets to upload the job jar, so the jobs all fail)
- it doesn't call waitForCompletion, or check job status (so it doesn't notice that the jobs all fail)
- it doesn't verify in any way that the job output is owned by the user who supposedly ran the job
- it shuts down DFS before MR

These all conspire to make it pass, but it isn't actually testing anything.
MAPREDUCE-2437. Blocker bug reported by shv and fixed by shv (test)
SLive should process only part* files while generating the report.

SliveTest when producing the final report scans all files in the reduce output directory. The directory now may contain {{_SUCCESS}} and {{_logs}} entries. SliveTest should process only files starting with {{part*}}.
MAPREDUCE-2428. Blocker bug reported by tomwhite and fixed by tomwhite
start-mapred.sh script fails if HADOOP_HOME is not set

MapReduce portion of HADOOP-6953
MAPREDUCE-2410. Minor improvement reported by dieter_be and fixed by qwertymaniac (contrib/streaming, documentation)
document multiple keys per reducer oddity in hadoop streaming FAQ

Hi,
for a newcomer to hadoop streaming, it comes as a surprise that the reducer receives arbitrary keys, unlike the "real" hadoop where a reducer works on a single key.
An explanation for this is @ http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201103.mbox/browser

I suggest to add this to the FAQ of hadoop streaming
MAPREDUCE-2394. Blocker bug reported by tlipcon and fixed by tlipcon
JUnit output format doesn't propagate into some contrib builds

Some of the contribs seem to have an issue where the test.junit.output.format property isn't propagating down into their builds. So, Hudson is unable to parse the test output, and we see failed builds with no actual parsed test results showing what failed.

This is at least true for {{contrib/raid}} but maybe others as well.
MAPREDUCE-2392. Major bug reported by tomwhite and fixed by tomwhite
TaskTracker shutdown in the tests sometimes take 60s

There are a lot of the following in the test logs:

2011-03-16 13:47:02,267 INFO mapred.TaskTracker (TaskTracker.java:shutdown(1275)) - Shutting down StatusHttpServer
2011-03-16 13:48:02,349 ERROR mapred.TaskTracker (TaskTracker.java:offerService(1609)) - Caught exception: java.io.IOException: Call to localhost/127.0.0.1:57512 failed on local exception: java.nio.channels.ClosedByInterruptException

Note there is over one minute between the first line and the second.
MAPREDUCE-2383. Major task reported by tlipcon and fixed by qwertymaniac (distributed-cache, documentation)
Improve documentation of DistributedCache methods

Users find the various methods in DistributedCache confusing - it's not clearly documented what the difference is between addArchiveToClassPath vs addFileToClassPath. We should improve the docs to clarify this and perhaps add an example that uses the DistributedCache.
MAPREDUCE-2372. Major improvement reported by tlipcon and fixed by tlipcon (task)
TaskLogAppender mechanism shouldn't be set in log4j.properties

The TaskLogAppender log4j appender relies on using log4j.properties to pass in some Java system properties into properties of the logger. This is problematic since we've often found that users have customized log4j.properties and don't upgrade it when they upgrade the version of Hadoop.

Since this is really an internal mechanism of how the task runner passes task info to the TLA, we shouldn't rely on these settings in log4j.properties at all. Rather, we should just get the system properties directly from System.getProperty.
MAPREDUCE-2337. Major improvement reported by tomwhite and fixed by tomwhite
Remove dependence of public MapReduce API on classes in server package

Cluster#getJobTrackerState() returns a org.apache.hadoop.mapreduce.server.jobtracker.State enum, which makes the API in o.a.h.mapreduce have a dependency on the server package. It would be better to make the public API self-contained by using an equivalent enum in the Cluster class.
MAPREDUCE-2336. Major bug reported by tomwhite and fixed by tomwhite (documentation)
Tool-related packages should be in the Tool javadoc group

Some of the tool packages are mistakenly in the general group.
MAPREDUCE-2327. Blocker bug reported by tlipcon and fixed by tlipcon
MapTask doesn't need to put username information in SpillRecord

This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.

This bug causes task failures in the following case:
- Cluster is not set up with LinuxTaskController (ie not secured cluster)
- Job submitter is not the same as the user running the TT
- Map output is more than one spill's worth

The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.
MAPREDUCE-2315. Blocker bug reported by tlipcon and fixed by tlipcon
javadoc is failing in nightly

Last nightly build failed to publish javadoc because the javadoc build failed:

javadoc:
[mkdir] Created dir: /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-22-branch/trunk/build/docs/api
[javadoc] Generating Javadoc
[javadoc] Javadoc execution
[javadoc] 1 error
[javadoc] javadoc: error - Cannot find doclet class org.apache.hadoop.classification.tools.ExcludePrivateAnnotationsStandardDoclet
MAPREDUCE-2314. Major improvement reported by rvs and fixed by rvs (build)
configure files that are generated as part of the released tarball need to have executable bit set

Currently the configure files that are packaged in a tarball are -rw-rw-r--
MAPREDUCE-2311. Blocker bug reported by tlipcon and fixed by schen (contrib/fair-share)
TestFairScheduler failing on trunk

Most of the test cases in this test are failing on trunk, unclear how long since the contrib tests weren't running while the core tests were failed.
MAPREDUCE-2304. Minor bug reported by priyomustafi and fixed by priyomustafi (test)
TestMRCLI fails when hostname has a hyphen (-)

TestMRCLI fails with below

Comparator: [RegexpComparator]
Comparision result: [fail]
Expected output: [mv: Wrong FS: har:/dest/dir0.har/dir0/file0, expected: hdfs://\w+[.a-z]*:[0-9]+]
Actual output: [mv: Wrong FS: har:/dest/dir0.har/dir0/file0, expected: hdfs://lab-something.host.com:34039
MAPREDUCE-2285. Blocker bug reported by rvadali and fixed by tlipcon (test)
MiniMRCluster does not start after ant test-patch

Any test using MiniMRCluster hangs in the MiniMRCluster constructor after running ant test-patch. Steps to reproduce:
1. ant -Dpatch.file=<dummy patch to CHANGES.txt> -Dforrest.home=<path to forrest> -Dfindbugs.home=<path to findbugs> -Dscratch.dir=/tmp/testpatch -Djava5.home=<path to java5> test-patch
2. Run any test that creates MiniMRCluster, say ant test -Dtestcase=TestFileArgs (contrib/streaming)

Expected result: Test should succeed
Actual result: Test hangs in MiniMRCluster.<init>. This does not happen if we run ant clean after ant test-patch
MAPREDUCE-2284. Critical bug reported by tlipcon and fixed by tlipcon (test)
TestLocalRunner.testMultiMaps times out

This test has timed out in a number of Hudson builds.
MAPREDUCE-2283. Blocker bug reported by nidaley and fixed by rvadali (contrib/raid)
TestBlockFixer hangs initializing MiniMRCluster

TestBlockFixer (a raid contrib test) is hanging the precommit testing on Hudson
MAPREDUCE-2282. Blocker bug reported by tomwhite and fixed by shv (test)
MapReduce tests don't compile following HDFS-1561

TestMRServerPorts depends on TestHDFSServerPorts which was changed by HDFS-1561, resulting in a compilation failure.
MAPREDUCE-2281. Major bug reported by pocheung and fixed by pocheung
Fix javac, javadoc, findbugs warnings

Split from HADOOP-6642
MAPREDUCE-2277. Minor bug reported by tlipcon and fixed by tlipcon (contrib/capacity-sched)
TestCapacitySchedulerWithJobTracker fails sometimes

Sometimes the testJobTrackerIntegration test fails on my Hudson. It seems the issue is that it doesn't ever wait for the first job to complete before checking its success status. Since the two jobs are in different queues, the first job may complete after the second job.
MAPREDUCE-2272. Trivial bug reported by tlipcon and fixed by qwertymaniac (tasktracker)
Job ACL file should not be executable

For some reason the job ACL file is localized with permissions 700. This doesn't make sense, since it's not executable. It should be 600.
MAPREDUCE-2260. Major improvement reported by rvs and fixed by rvs (build)
Remove auto-generated native build files

The repo currently includes the automake and autoconf generated files for the native build. Per discussion on HADOOP-6421 let's remove them and use the host's automake and autoconf. We should also do this for libhdfs and fuse-dfs.
MAPREDUCE-2256. Major bug reported by priyomustafi and fixed by priyomustafi (contrib/fair-share)
FairScheduler fairshare preemption from multiple pools may preempt all tasks from one pool causing that pool to go below fairshare.

Scenarios:
You have a cluster with 600 map slots and 3 pools. Fairshare for each pool is 200 to start with. Fairsharepreemption timeout is 5 mins.
1) Pool1 schedules 300 map tasks first
2) Pool2 then schedules another 300 map tasks
3) Pool3 demands 300 map tasks but doesn't get any slot as all slots are taken.
4) After 5 mins pool3 should preempt 200 map-slots. Instead of peempting 100 slots each from pool1 and pool2, the bug would cause it to preempt all 200 slots from pool2 (last started) causing it to go below fairshare. This is happening because the preemptTask method is not reducing the tasks left from a pool while preempting the tasks.

The above scenario could be an extreme case but some amount of excess preemption would happen because of this bug.

The patch I created was for 0.22.0 but the code fix should work on 0.21 as well as looks like it has the same bug.
MAPREDUCE-2253. Critical bug reported by tlipcon and fixed by tlipcon
Servlets should specify content type

HADOOP-7093 will change the default content-type to text/plain. So TaskLogServlet, which outputs HTML, needs to change to specify this content type. I believe the other HTML servlets already correctly specify a content type. The MapOutputServlet appears to specify no content type and work fine without one, but to be "correct" we may as well specify application/octet-stream
MAPREDUCE-2251. Major bug reported by tlipcon and fixed by qwertymaniac
Remove mapreduce.job.userhistorylocation config

Best I can tell, this config parameter is no longer used as of MAPREDUCE-157 but still exists in the code and in mapred-default.xml. We should remove it to avoid user confusion.
MAPREDUCE-2241. Trivial test reported by tlipcon and fixed by tlipcon (task-controller, test)
ClusterWithLinuxTaskController should accept relative path on the command line

Currently if you pass a relative path for the -Dtaskcontroller-path option when running these tests, it fails in a fairly unintuitive way. We should absolutize it inside the tests to make it easier for people to run them.
MAPREDUCE-2238. Critical bug reported by eli and fixed by tlipcon (build, test)
Undeletable build directories

The MR hudson job is failing, looks like it's due to a test chmod'ing a build directory so the checkout can't clean the build dir.
MAPREDUCE-2234. Major sub-task reported by tlipcon and fixed by tlipcon (tasktracker)
If Localizer can't create task log directory, it should fail on the spot

Currently, it simply emits a warning. Then, when the taskjvm.sh tries to pipe its output into this directory, it fails with a strange error code like "exit code: 1" which is not intuitive to ops. Instead it should simply throw an exception at initialization time rather than attempting to run the task.
MAPREDUCE-2224. Critical bug reported by tlipcon and fixed by tlipcon (tasktracker)
Synchronization bugs in JvmManager

JvmManager.JvmManagerForType has several HashMap members that are inconsistently synchronized. I've seen sporadic NPEs in the 0.20 version of this code which has similar bugs.
MAPREDUCE-2222. Major bug reported by vicaya and fixed by vicaya
Ivy resolve force mode should be turned off by default

cf. HADOOP-7068
MAPREDUCE-2219. Major bug reported by tlipcon and fixed by tlipcon (jobtracker)
JT should not try to remove mapred.system.dir during startup

During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the *contents* of mapred.system.dir instead of the directory itself.
MAPREDUCE-2200. Major bug reported by cos and fixed by cos (test)
TestUmbilicalProtocolWithJobToken is failing without Krb evironment: needs to be conditional

TestUmbilicalProtocolWithJobToken requires Krb environment to be set. For testing some 'pseudo' environment is needed (similar to HDFS-1284).
MAPREDUCE-2195. Major bug reported by cos and fixed by cos (test)
New property for local conf directory in system-test-mapreduce.xml file.

As its counter-part HDFS-1167: new parameter needs to be added to the system-test configuration file to serve 'cluster restart with new configuration' feature
MAPREDUCE-2188. Major bug reported by owen.omalley and fixed by owen.omalley
The new API MultithreadedMapper doesn't call the initialize method of the RecordReader

The wrapping RecordReader in the Multithreaded Mapper is never initialized. With HADOOP-6685, this becomes a problem because the ReflectionUtils.copy requires a non-null configuration.
MAPREDUCE-2184. Major improvement reported by rvadali and fixed by rvadali (contrib/raid)
Port DistRaid.java to new mapreduce API

DistRaid.java was implemented with the older mapred API, this task is for porting it to the new API
MAPREDUCE-2180. Minor test reported by tlipcon and fixed by tlipcon (contrib/fair-share)
Add coverage of fair scheduler servlet to system test

MAPREDUCE-2051 added a system test for the fair scheduler which starts a minicluster and runs a couple jobs with preemption on. I recently found a deadlock in a previous version of the scheduler that was due to lock inversion between the scheduler servlet and some JT internals. I'd like to modify the existing system test to also hit the /scheduler servlet, allowing jcarder to detect such lock inversions in the future.
MAPREDUCE-2179. Blocker bug reported by gkesavan and fixed by rvadali (contrib/raid)
RaidBlockSender.java compilation fails

Mapreduce trunk compilation is broken
MAPREDUCE-2173. Major bug reported by pkling and fixed by pkling
Race condition in TestBlockFixer causes intermittent failure

TestBlockFixer sometimes fails in reportCorruptBlocks because a corrupt block is deleted before in.readFully is called. This causes a BlockMissingException instead of the expected ChecksumException.
MAPREDUCE-2169. Major task reported by rvadali and fixed by rvadali (contrib/raid)
Integrated Reed-Solomon code with RaidNode

Scott Chen recently checked in an implementation of the Reed Solomon code. This task will track the integration of the code with RaidNode.
MAPREDUCE-2167. Major improvement reported by rvadali and fixed by rvadali (contrib/raid)
Faster directory traversal for raid node

The RaidNode currently iterates over the directory structure to figure out which files to RAID. With millions of files, this can take a long time - especially if some files are already RAIDed and the RaidNode needs to look at parity files / parity file HARs to determine if the file needs to be RAIDed.

The directory traversal is encapsulated inside the class DirectoryTraversal, which examines one file at a time, using the caller's thread.

My proposal is to make this multi-threaded as follows:
* use a pool of threads inside DirectoryTraversal
* The caller's thread is used to retrieve directories, and each new directory is assigned to a thread in the pool. The worker thread examines all the files the directory.
* If there sub-directories, those are added back as workitems to the pool.

Comments?
MAPREDUCE-2150. Major improvement reported by rvadali and fixed by rvadali (contrib/raid)
RaidNode should periodically fix corrupt blocks

(Recreating HDFS-1171 since RAID is in mapreduce)

RaidNode currently does not fix missing blocks. The missing blocks have to be fixed using RaidShell.
This task proposes that recovery be more automated:
1. RaidNode periodically fetches a list of corrupt files from the NameNode
2. If the corrupt files can be fixed using RAID, it should generate the block.
3. Choose a datanode and send the block contents along with checksum to the datanode.
MAPREDUCE-2147. Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (jobtracker)
JobInProgress has some redundant lines in its ctor

In the ctor of JobInProgress class that's used by the JT, lines that create the various lists of TIPs are repeated for no purpose. Might've been due to an overlook I think.

Attaching a patch that removes these unnecessary repeats of re-init.
MAPREDUCE-2146. Minor bug reported by rvadali and fixed by rvadali (contrib/raid)
Raid should not affect access time of a source file

After a file is read for creating a raid parity file, the access time should be set back to the value before the read. The read by RAID code is not an application read and should not affect the access time.
MAPREDUCE-2145. Major improvement reported by gkesavan and fixed by gkesavan
upgrade clover on the builds servers to v 3.0.2

upgrade clover on the builds servers to v 3.0.2, as the mr builds requires the latest version of clover for better coverage reporting...
MAPREDUCE-2143. Major bug reported by rvadali and fixed by rvadali (harchive)
HarFileSystem is not able to handle spaces in its path

If the Path to the HAR contains spaces, Path.getFileSystem() fails. The problem is in HarFileSystem.initialize(), which uses URI.toString() to get a string for getting to the .har suffix. URI.toString() returns a percent-encoded string when the path contains spaces. When this string is subsequently used to get the _index file, we get a FileNotFoundException. The fix is to use URI.getPath().
MAPREDUCE-2142. Major task reported by pkling and fixed by pkling
Refactor RaidNode to remove dependence on map reduce

I am refactoring the RaidNode code as follows: The base class RaidNode will contain the common functionality needed for raiding files. The derived class LocalRaidNode contains an implementation of RaidNode that performs raiding locally. The derived class DistRaidNode performs raiding using map reduce jobs. This way, only DistRaidNode has a dependency on map reduce code and RaidNode and LocalRaidNode can be moved to HDFS.
MAPREDUCE-2141. Minor improvement reported by matei and fixed by matei
Add an "extra data" field to Task for use by Mesos

In order to support running Hadoop on the Mesos cluster manager (http://mesos.berkeley.edu/), I'd like to add an extra String field to the Task class to allow extra data (a Mesos task ID) to be associated with each task. This should have no impact on normal operation other than making the serialized form of Task a few bytes longer. In the Mesos support patch for Hadoop, this field is set by a pluggable Hadoop scheduler implementation to allow code on the TaskTracker side to see which Mesos task corresponds to each Hadoop task.
MAPREDUCE-2132. Major improvement reported by rvadali and fixed by rvadali (contrib/raid)
Need a command line option in RaidShell to fix blocks using raid

RaidShell currently has an option to recover a file and return the path to the recovered file. The administrator can then rename the recovered file to the damaged file.

The problem with this is that the file metadata is altered, specifically the modification time. Instead we need a way to just repair the damaged blocks and send the fixed blocks to a data node.

Once this is done, we can put automation around it.
MAPREDUCE-2127. Major bug reported by gkesavan and fixed by bmahe (build, pipes)
mapreduce trunk builds are failing on hudson

https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/507/console

[exec] checking for pthread.h... yes
[exec] checking for pthread_create in -lpthread... yes
[exec] checking for HMAC_Init in -lssl... no
[exec] configure: error: Cannot find libssl.so
[exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required
[exec] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/src/c++/pipes/configure: line 4250: exit: please: numeric argument required

BUILD FAILED
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build.xml:1647: exec returned: 255
MAPREDUCE-2126. Minor improvement reported by yaojingguo and fixed by yaojingguo (jobtracker)
JobQueueJobInProgressListener's javadoc is inconsistent with source code

JobQueueJobInProgressListener.java has {@link #JobQueueJobInProgressListener(Collection)} in Javadoc. But it does not have the corresponding constructor. It has constructor JobQueueJobInProgressListener(Map<JobSchedulingInfo, JobInProgress> jobQueue). So {@link JobQueueJobInProgressListener(Collection)} should be {@link #JobQueueJobInProgressListener(Map)}.
MAPREDUCE-2103. Trivial improvement reported by tlipcon and fixed by tlipcon (task-controller)
task-controller shouldn't require o-r permissions

The task-controller currently checks that "other" users don't have read permissions. This is unnecessary - we just need to make it's not executable. The debian policy manual explains it well:
Setuid and setgid executables should be mode 4755 or 2755 respectively, and owned by the appropriate user or group. They should not be made unreadable (modes like 4711 or 2711 or even 4111); doing so achieves no extra security, because anyone can find the binary in the freely available Debian package; it is merely inconvenient. For the same reason you should not restrict read or execute permissions on non-set-id executables.
Some setuid programs need to be restricted to particular sets of users, using file permissions. In this case they should be owned by the uid to which they are set-id, and by the group which should be allowed to execute them. They should have mode 4754; again there is no point in making them unreadable to those users who must not be allowed to execute them.
MAPREDUCE-2099. Major bug reported by rvadali and fixed by rvadali (contrib/raid)
RaidNode should recreate outdated parity HARs

After parity files are archived into a parity HAR, a change in the source file does not cause the HAR to be recreated. Instead, individual parity files are created for the modified files but the HAR is not touched. This causes increased disk usage for parity data.

The parity HAR could be recreated if a certain percentage of files in the HAR are determined to be outdated.
MAPREDUCE-2096. Blocker bug reported by tlipcon and fixed by tlipcon (jobtracker, security, tasktracker)
Secure local filesystem IO from symlink vulnerabilities

This JIRA is to contribute a patch developed on the private security@ mailing list.

The vulnerability is that MR daemons occasionally open files that are located in a path where the user has write access. A malicious user may place a symlink in place of the expected file in order to cause the daemon to instead read another file on the system -- one which the attacker may not naturally be able to access. This includes delegation tokens belong to other users, log files, keytabs, etc.
MAPREDUCE-2095. Major bug reported by vinaythota and fixed by ranjit (contrib/gridmix)
Gridmix unable to run for compressed traces(.gz format).

I was trying to run gridmix with compressed trace file.However, it throws a JsonParseException and exit.
MAPREDUCE-2093. Major improvement reported by cos and fixed by cos (test)
Herriot JT and TT clients should vend statistics

Mapreduce counterpart of HDFS-1408
MAPREDUCE-2084. Blocker bug reported by tomwhite and fixed by tomwhite (documentation)
Deprecated org.apache.hadoop.util package in MapReduce produces deprecations in Common classes in Eclipse

As reported in [this thread|http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201009.mbox/%3C4C9A0A08.3030901@web.de%3E] the classes in org.apache.hadoop.util from the Common JAR, like Tool, are marked as deprecated by Eclipse, even though they are not deprecated. The fix is to mark the individual classes in the MapReduce org.apache.hadoop.util class as deprecated, not the whole package.
MAPREDUCE-2082. Major bug reported by jnp and fixed by jnp
Race condition in writing the jobtoken password file when launching pipes jobs

In Application.java, when jobtoken password file is written, there is a race condition because the file is written in job's work directory. The file should rather be written in the task's working directory.
MAPREDUCE-2078. Major bug reported by vinaythota and fixed by amar_kamat (tools/rumen)
TraceBuilder unable to generate the traces while giving the job history path by globing.

I was trying to generate the traces for MR job histories by using TraceBuilder. However, it's unable to generate the traces while giving the job history path by globing. It throws a file not found exception even though the job history path is exists.
MAPREDUCE-2077. Major bug reported by vicaya and fixed by vicaya
Name clash in the deprecated o.a.h.util.MemoryCalculatorPlugin

Name clash compile error in the deprecated org.apache.hadoop.util.MemoryCalculatorPlugin due to JLS3 8.4.8.3 (cf. http://bugs.sun.com/view_bug.do?bug_id=6182950)

The bug doesn't manifest in jdk 1.6 up to 20, but shows up in NetBeans 6.9+ due to its bundled (conforming) compiler. Fix is trivial: just remove the offending method in the deprecated subclass as its equivalent erasure is inherited from the parent class anyway.
MAPREDUCE-2073. Trivial test reported by tlipcon and fixed by tlipcon (distributed-cache, test)
TestTrackerDistributedCacheManager should be up-front about requirements on build environment

TestTrackerDistributedCacheManager will fail on a system where the build directory is in any path where an ancestor doesn't have a+x permissions. On one of our hudson boxes, for example, hudson's workspace had 700 permissions and caused this test to fail reliably, but not in an obvious manner. It would be helpful if the test failed with a more obvious error message during setUp() when the build environment is misconfigured.
MAPREDUCE-2067. Major bug reported by atm and fixed by atm (security)
Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies

MR portion of HADOOP-6951.
MAPREDUCE-2059. Major bug reported by dadkins and fixed by subrotosanyal (jobtracker)
RecoveryManager attempts to add jobtracker.info

The jobtracker is treating the file 'jobtracker.info' in the system data directory as a job to be recovered
MAPREDUCE-2054. Major bug reported by sandholm and fixed by sandholm (contrib/dynamic-scheduler)
Hierarchical queue implementation broke dynamic queue addition in Dynamic Scheduler

Queue names were returned from the queue manager as an immutable set after the hierarchical queuname feature which breaks the dynamic priority scheduler
MAPREDUCE-2051. Major test reported by tlipcon and fixed by tlipcon (contrib/fair-share)
Contribute a fair scheduler preemption system test

We've seen a number of bugs in the fair share scheduler not caught by its unit tests, which are heavily mock-based. This JIRA is to add an integration/system/stress test for the fairshare scheduler. This test can help identify races and deadlocks, and when run within the jcarder framework has identified several potential deadlocks for us that aren't turned up running small scale testing.
MAPREDUCE-2046. Major improvement reported by namit and fixed by dhruba
A CombineFileInputSplit cannot be less than a dfs block

I ran into this while testing some hive features.

Whether we use hiveinputformat or combinehiveinputformat, a split cannot be less than a dfs block size.
This is a problem if we want to increase the block size for older data to reduce memory consumption for the
name node.

It would be useful if the input split was independent of the dfs block size.
MAPREDUCE-2034. Trivial test reported by tlipcon and fixed by tlipcon (test)
TestSubmitJob triggers NPE instead of permissions error

TestSubmitJob.testSecureJobExecution catches _any_ IOException and assumes a permissions error has been caught. In fact, it was passing an invalid path name to the NameNode and triggering an NPE, not a Permission denied error, in one case, but the test was not specific enough to detect this.
MAPREDUCE-2031. Major bug reported by amareshwari and fixed by ravidotg (tasktracker, test)
TestTaskLauncher and TestTaskTrackerLocalization fail with NPE in trunk.

TestTaskLauncher and TestTaskTrackerLocalization fail in trunk after the commit of MAPREDUCE-1881 with NPE:
NPE happens because taskTracker.myInstrumentation is not initialized.
MAPREDUCE-2029. Major bug reported by pauly and fixed by rvadali (contrib/raid)
DistributedRaidFileSystem not removed from cache on close()

When DistributedRaidFileSystem.close() is called, it does not remove itself from the FileSystem cache, but it does close the underlying filesystem, e.g. DFS.

Because the DRFS with the closed DFS is still in the cache, calling FileSystem.get() returns a stale DRFS that throws 'filesystem closed' exceptions.
MAPREDUCE-2026. Major improvement reported by schen and fixed by jsensarma
JobTracker.getJobCounters() should not hold JobTracker lock while calling JobInProgress.getCounters()

JobTracker.getJobCounter() will lock JobTracker and call JobInProgress.getCounters().
JobInProgress.getCounters() can be very expensive because it aggregates all the task counters.
We found that from the JobTracker jstacks that this method is one of the bottleneck of the JobTracker performance.

JobInProgress.getCounters() should be able to be called out side the JobTracker lock because it already has JobInProgress lock.
For example, it is used by jobdetails.jsp without a JobTracker lock.
MAPREDUCE-2023. Major bug reported by hong.tang and fixed by hong.tang (benchmarks)
TestDFSIO read test may not read specified bytes.

TestDFSIO's read test may read less bytes than specified when reading large files.
MAPREDUCE-2022. Blocker bug reported by amareshwari and fixed by amareshwari (test)
trunk build broken

Trunk compilation fails with following errors:
[javac] /home/amarsri/workspace/trunk/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:267: getListing(java.lang.String,byte[],boolean) in org.apache.hadoop.hdfs.protocol.ClientProtocol cannot be applied to (java.lang.String,byte[])
[javac] client.getListing(
[javac] ^
[javac] /home/amarsri/workspace/trunk/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:281: getListing(java.lang.String,byte[],boolean) in org.apache.hadoop.hdfs.protocol.ClientProtocol cannot be applied to (java.lang.String,byte[])
[javac] client.getListing(
[javac] ^

This is due to commit of HDFS-202
MAPREDUCE-2021. Major bug reported by amareshwari and fixed by amareshwari (client)
CombineFileInputFormat returns duplicate hostnames in split locations

CombineFileInputFormat.getSplits creates splits with duplicate locations. It adds locations of the files in the split to an ArrayList; if all the files are on same location, the location is added again and again. Instead, it should add it to a Set instead of List to avoid duplicates.
MAPREDUCE-2000. Major bug reported by hong.tang and fixed by hong.tang (tools/rumen)
Rumen is not able to extract counters for Job history logs from Hadoop 0.20

Rumen tries to match the end of a value string through indexOf("\""). It does not take into account the case when an escaped '"' in the value string. This leads to the incorrect parsing the remaining key=value properties in the same line.
MAPREDUCE-1999. Major bug reported by jnp and fixed by jnp
ClientProtocol incorrectly uses hdfs DelegationTokenSelector

ClientProtocol in MR incorrectly uses the DelegationTokenSelector in hdfs due to a wrong import. It should use the DelegationTokenSelector in mapreduce.
MAPREDUCE-1996. Trivial bug reported by glynnbach and fixed by qwertymaniac (documentation)
API: Reducer.reduce() method detail misstatement

method detail for Reducer.reduce() method has paragraph starting:

"Applications can use the Reporter provided to report progress or just indicate that they are alive. In scenarios where the application takes an insignificant amount of time to process individual key/value pairs, this is crucial since the framework might assume that the task has timed-out and kill that task. "

s/an insignificant amount of time/a significant amount of time/
MAPREDUCE-1993. Major bug reported by devaraj and fixed by devaraj
TestTrackerDistributedCacheManagerWithLinuxTaskController fails on trunk

TestTrackerDistributedCacheManagerWithLinuxTaskController fails when run with the DefaultTaskController.
MAPREDUCE-1992. Major bug reported by ravidotg and fixed by kzhang (jobtracker)
NPE in JobTracker's constructor

On my local machine, JobTracker is *not* coming up with current trunk. Logs show the following NPE:

2010-08-03 14:01:41,449 FATAL org.apache.hadoop.mapred.JobTracker: java.lang.NullPointerException
at org.apache.hadoop.security.UserGroupInformation.isLoginKeytabBased(UserGroupInformation.java:703)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1383)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:275)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:267)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:262)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4236)

2010-08-03 14:01:41,449 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
MAPREDUCE-1989. Major bug reported by ravidotg and fixed by ravidotg (contrib/gridmix)
Gridmix3 doesn't emit out proper mesage when user resolver is set and no user list is given

Currently, Gridmix3 emits out the following mesage when user resolver is set and no user list is given.
"Resource null ignored".
This is not clear/meaningful to user.
MAPREDUCE-1982. Major bug reported by amar_kamat and fixed by ravidotg (tools/rumen)
[Rumen] TraceBuilder's output shows jobname as NULL for jobhistory files with valid jobnames

{{TraceBuilder}} fails to extract configuration properties (like job-name) from the job-conf if the job-conf has the properties stored using the deprecated keys.
MAPREDUCE-1979. Major bug reported by ravidotg and fixed by ravidotg (contrib/gridmix)
"Output directory already exists" error in gridmix when gridmix.output.directory is not defined

"Output directory already exists" error is seen in gridmix when gridmix.output.directory is not defined. When gridmix.output.directory is not defined, then gridmix uses inputDir/gridmix/ as output path for gridmix run. Because gridmix is creating outputPath(in this case, inputDir/gridmix/) at the begining, the output path to generate-data-mapreduce-job(i.e. inputDir) already exists and becomes error from mapreduce.

There is need for creation of this outputPath in any case(whether user specifies the path using gridmix.output.directory OR gridmix itself considering inputDir/gridmix/ ) even though the paths are automatically created for output paths of mapreduce jobs(like mkdir -p), because gridmix needs to set 777 permissions for this outputPath sothat different users can create different output directories of different mapreduce jobs within this gridmix run.

The other case in which this problem is seen is when gridmix.output.directory is defined as a relative path. This is because in this case also, gridmix tries to create relative path under ioPath/ and thus the same issue.
MAPREDUCE-1975. Major bug reported by ravidotg and fixed by ravidotg (contrib/gridmix)
gridmix shows unnecessary InterruptedException

The following InterruptedException is seen when gridmix is run and it ran successfully:

10/06/24 20:43:03 INFO gridmix.ReplayJobFactory: START REPLAY @ 11331037109
10/06/24 20:43:03 ERROR gridmix.Statistics: Statistics interrupt while waiting for polling null
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObjec\
t.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObjec\
t.await(AbstractQueuedSynchronizer.java:2066)
at org.apache.hadoop.mapred.gridmix.Statistics$StatCollector.run(Statis\
tics.java:190)
10/06/24 20:43:03 INFO gridmix.Gridmix: Exiting...
MAPREDUCE-1974. Major bug reported by schen and fixed by schen (contrib/fair-share)
FairScheduler can preempt the same task many times

In FairScheduler.preemptTasks(), tasks are collected from JobInProgress.runningMapCache.
But tasks repeat multiple times in JobInProgress.runningMapCache (on rack, node and cluster).
This makes FairScheduler preempt the same task many times.
MAPREDUCE-1970. Major sub-task reported by schen and fixed by schen (contrib/raid)
Reed-Solomon code implementation to be used in raid

A Reed-Solomon erasure code implementation.
MAPREDUCE-1961. Major bug reported by hong.tang and fixed by hong.tang (contrib/gridmix)
[gridmix3] ConcurrentModificationException when shutting down Gridmix

We observed the following exception occasionally at the end of the Gridmix run:

{code}
Exception in thread "StatsCollectorThread" java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.hadoop.mapred.gridmix.Statistics$StatCollector.updateAndNotifyClusterStatsListeners(Statistics.java:220)
at org.apache.hadoop.mapred.gridmix.Statistics$StatCollector.run(Statistics.java:205)
{code}
MAPREDUCE-1958. Major bug reported by boryas and fixed by boryas
using delegation token over hftp for long running clients (part of hdfs 1296)
MAPREDUCE-1945. Major improvement reported by kzhang and fixed by kzhang
Support for using different Kerberos keys for Jobtracker and TaskTrackers

This is the MapRed part of HADOOP-6632.
MAPREDUCE-1936. Major improvement reported by hong.tang and fixed by hong.tang (contrib/gridmix)
[gridmix3] Make Gridmix3 more customizable.

I'd like to make gridmix3 more customizable. Specifically, the proposed customizations include:
- add (random) location information for each sleep map task.
- make the parameters used in stress submission load throttling configurable.
MAPREDUCE-1935. Major improvement reported by boryas and fixed by boryas
HFTP needs to be updated to use delegation tokens (from HDFS-1007)
MAPREDUCE-1931. Major improvement reported by rksingh and fixed by ranjit (contrib/gridmix)
Gridmix forrest documentation

Gridmix forrest documentation
MAPREDUCE-1925. Major bug reported by amareshwari and fixed by ravidotg (tools/rumen)
TestRumenJobTraces fails in trunk

TestRumenJobTraces failed with following error:
Error Message

the gold file contains more text at line 1 expected:<56> but was:<0>

Stacktrace

at org.apache.hadoop.tools.rumen.TestRumenJobTraces.testHadoop20JHParser(TestRumenJobTraces.java:294)

Full log of the failure is available at http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/292/testReport/org.apache.hadoop.tools.rumen/TestRumenJobTraces/testHadoop20JHParser/
MAPREDUCE-1918. Major improvement reported by amar_kamat and fixed by amar_kamat (tools/rumen)
Add documentation to Rumen

Add forrest documentation to Rumen tool.
MAPREDUCE-1915. Trivial bug reported by rvernica and fixed by priyomustafi (tasktracker)
IndexCache - getIndexInformation - check reduce index Out Of Bounds

When checking if the "reduce" index is out of bounds the check should be

info.mapSpillRecord.size() <= reduce

instead of:

info.mapSpillRecord.size() < reduce

Not a big bug since an Out Of Bounds is thrown downstream anyway.
MAPREDUCE-1911. Major bug reported by amareshwari and fixed by amareshwari (contrib/streaming)
Fix errors in -info option in streaming

Here are some of the findings by Karam while verifying -info option in streaming:
# We need to add "Optional" for -mapper, -reducer,-combiner and -file options.
# For -inputformat and -outputformat options, we should put "Optional" in the prefix for the sake on uniformity.
# We need to remove -cluster decription.
# -help option is not displayed in usage message.
# when displaying message for -info or -help options, we should not display "Streaming Job Failed!"; also exit code should be 0 in case of -help/-info option.
MAPREDUCE-1908. Major bug reported by rvadali and fixed by rvadali (contrib/raid)
DistributedRaidFileSystem does not handle ChecksumException correctly

ChecksumException reports the offset of corruption within a block,
whereas DistributedRaidFileSystem.setAlternateLocations was expecting it
to report the offset of corruption within the file.

The best way of dealing with a missing block/corrupt block is to just
use the current seek offset in the file as the position of corruption.
MAPREDUCE-1900. Major bug reported by devaraj and fixed by kzhang (jobtracker, tasktracker)
MapReduce daemons should close FileSystems that are not needed anymore

Related to HADOOP-6843, this jira is to make MapReduce behave better with respect to closing FileSystems when they are not needed anymore.
MAPREDUCE-1894. Major bug reported by rvadali and fixed by rvadali (contrib/raid)
DistributedRaidFileSystem.readFully() does not return

DistributedRaidFileSystem.readFully() has a while(true) loop with no return. The read(*) functions do not have this problem.
MAPREDUCE-1893. Major improvement reported by shv and fixed by shv (benchmarks, test)
Multiple reducers for Slive

Slive currently uses single reducer. It could use multiple ones.
MAPREDUCE-1892. Major improvement reported by rvadali and fixed by rvadali (contrib/raid)
RaidNode can allow layered policies more efficiently

The RaidNode policy file can have layered policies that can cover a file more than once. To avoid processing a file multiple times (for RAIDing), RaidNode maintains a list of processed files that is used to avoid duplicate processing attempts.

This is problematic in that a large number of processed files could cause the RaidNode to run out of memory.

This task proposes a better method of detecting processed files. The method is based on the observation that a more selective policy will have a better match with a file name than a less selective one. Specifically, the more selective policy will have a longer common prefix with the file name.

So to detect if a file has already been processed, the RaidNode only needs to maintain a list of processed policies and compare the lengths of the common prefixes. If the file has a longer common prefix with one of the processed policies than with the current policy, it can be assumed to be processed already.
MAPREDUCE-1888. Major bug reported by amareshwari and fixed by ravidotg (contrib/streaming)
Streaming overrides user given output key and value types.

The following code in StreamJob.java overrides user given output key and value types.
idResolver.resolve(conf.get(StreamJobConfig.MAP_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.MAP_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setMapOutputKeyClass(idResolver.getOutputKeyClass());
job.setMapOutputValueClass(idResolver.getOutputValueClass());

idResolver.resolve(conf.get(StreamJobConfig.REDUCE_OUTPUT,
IdentifierResolver.TEXT_ID));
conf.setClass(StreamJobConfig.REDUCE_OUTPUT_READER_CLASS,
idResolver.getOutputReaderClass(), OutputReader.class);
job.setOutputKeyClass(idResolver.getOutputKeyClass());
job.setOutputValueClass(idResolver.getOutputValueClass());
MAPREDUCE-1887. Major bug reported by kimballa and fixed by kimballa
MRAsyncDiskService does not properly absolutize volume root paths

In MRAsyncDiskService, volume names are sometimes specified as relative paths, which are not converted to absolute paths. This can cause errors of the form "cannot delete </full/path/to/foo> since it is outside of <relative/volume/root>" even though the actual path is inside the root.
MAPREDUCE-1878. Major improvement reported by kimballa and fixed by kimballa (contrib/mrunit)
Add MRUnit documentation

A short user guide for MRUnit, written in asciidoc.
MAPREDUCE-1868. Major improvement reported by ramach and fixed by ramach (client)
Add read timeout on userlog pull

Add read and connection timeout to prevent job client hangs

jobclient can block indefinitely during task log pull if read or connect fails

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1049)
- locked <0xeed0f0f0> (a sun.net.www.protocol.http.HttpURLConnection)
at org.apache.hadoop.mapred.JobClient.getTaskLogs(JobClient.java:1396)
MAPREDUCE-1867. Minor bug reported by amareshwari and fixed by amareshwari (contrib/streaming)
Remove unused methods in org.apache.hadoop.streaming.StreamUtil

There are many unused methods in org.apache.hadoop.streaming.StreamUtil. They should be removed from the class for maintainability.
MAPREDUCE-1866. Minor bug reported by amareshwari and fixed by amareshwari (contrib/streaming)
Remove deprecated class org.apache.hadoop.streaming.UTF8ByteArrayUtils

The class org.apache.hadoop.streaming.UTF8ByteArrayUtils is deprecated in favor of org.apache.hadoop.util.UTF8ByteArrayUtils in branch 0.19.
The same should be removed.
MAPREDUCE-1865. Major bug reported by amar_kamat and fixed by amar_kamat (tools/rumen)
[Rumen] Rumen should also support jobhistory files generated using trunk

Rumen code in trunk parses and process only jobhistory files from pre-21 hadoop mapreduce clusters. It should also support jobhistory files generated using trunk.
MAPREDUCE-1864. Major bug reported by amareshwari and fixed by amareshwari (contrib/streaming)
PipeMapRed.java has uninitialized members log_ and LOGNAME

PipeMapRed.java has members log_ and LOGNAME, which are never initialized and they are used in code for logging in several places.
They should be removed and PipeMapRed should use commons LogFactory and Log for logging. This would improve code maintainability.

Also, as per [comment | https://issues.apache.org/jira/browse/MAPREDUCE-1851?focusedCommentId=12878530&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12878530], stream.joblog_ configuration property can be removed.
MAPREDUCE-1863. Major bug reported by amar_kamat and fixed by amar_kamat (tools/rumen)
[Rumen] Null failedMapAttemptCDFs in job traces generated by Rumen

All the traces generated by Rumen for jobs having failed task attempts has null value for failedMapAttemptCDFs.
MAPREDUCE-1857. Trivial bug reported by amareshwari and fixed by amareshwari (contrib/streaming)
Remove unused streaming configuration from src

The configuration stream.numinputspecs is just set and not read anywhere. It can be removed.
MAPREDUCE-1851. Major improvement reported by amareshwari and fixed by amareshwari (contrib/streaming, documentation)
Document configuration parameters in streaming

There are several streaming options such as stream.map.output.field.separator, stream.num.map.output.key.fields, stream.map.input.field.separator, stream.reduce.input.field.separator, stream.map.input.ignoreKey, stream.non.zero.exit.is.failure etc which are spread everywhere. These should be documented at single place with description and default-value.
MAPREDUCE-1850. Major improvement reported by ramach and fixed by ramach
Include job submit host information (name and ip) in jobconf and jobdetails display

Enhancement to identify the source (submit host and ip) of a job request.
MAPREDUCE-1848. Major improvement reported by schen and fixed by schen (jobtracker)
Put number of speculative, data local, rack local tasks in JobTracker metrics

It will be nice that we can collect these information in JobTracker metrics
MAPREDUCE-1840. Major improvement reported by amar_kamat and fixed by amar_kamat (contrib/gridmix)
[Gridmix] Exploit/Add security features in GridMix

Use security information while replaying jobs in Gridmix. This includes
- Support for multiple users
- Submitting jobs as different users
- Allowing usage of secure cluster (hdfs + mapreduce)
- Support for multiple queues

Other features include :
- Support for sleep job
- Support for load job

+ testcases for verifying all of the above changes
MAPREDUCE-1838. Minor improvement reported by rvadali and fixed by rvadali (contrib/raid)
DistRaid map tasks have large variance in running times

HDFS RAID uses map-reduce jobs to generate parity files for a set of source files. Each map task gets a subset of files to operate on. The current code assigns files by walking through the list of files given in the constructor of DistRaid

The problem is that the list of files given to the constructor has the order of (pretty much) the directory listing. When a large number of files is added, files in that order tend to have the same size. Thus a map task can end up with large files where as another can end up with small files, increasing the variance in run times.

We could do smarter assignment by using the file sizes.
MAPREDUCE-1836. Major bug reported by boryas and fixed by boryas
Refresh for proxy superuser config (mr part for HDFS-1096)
MAPREDUCE-1834. Major bug reported by amareshwari and fixed by hong.tang (contrib/mumak)
TestSimulatorDeterministicReplay timesout on trunk

TestSimulatorDeterministicReplay timesout on trunk.
See hudson patch build http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/216/testReport/org.apache.hadoop.mapred/TestSimulatorDeterministicReplay/testMain/
MAPREDUCE-1829. Major improvement reported by schen and fixed by schen (jobtracker)
JobInProgress.findSpeculativeTask should use min() to find the candidate instead of sort()

findSpeculativeTask needs only one candidate to speculate so it does not need to sort the whole list. It may looks OK but someone can still submit big jobs with small slow task thresholds. In this case, this sorting becomes expensive.
MAPREDUCE-1825. Major bug reported by amareshwari and fixed by schen (jobtracker)
jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps and finishedReduces when job is not initialized

JobInProgress.finishedMaps() and finishedReduces() are synchronized. They are called from jobqueue_details.jsp and FairSchedulerServelet which iterates through all jobs. If any job is in initialization, these pages don't come up until the initialization finishes.

See [comment|https://issues.apache.org/jira/browse/MAPREDUCE-1354?focusedCommentId=12834139&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12834139] for more details
MAPREDUCE-1820. Major bug reported by alexvk and fixed by alexvk
InputSampler does not create a deep copy of the key object when creating a sample, which causes problems with some formats like SequenceFile<Text,Text>

I tried to use the InputSampler on a SequenceFile<Text,Text> and found that it comes up with duplicate keys in the sample. The problem was tracked down to the fact that the Text object returned from the reader is essentially a wrapper pointing to a byte array, which changes as the sequence file reader progresses. There was also a bug in that the reader should be initialized before the use. The am attaching a patch that fixes both of the issues. --Alex K
MAPREDUCE-1819. Major improvement reported by rvadali and fixed by rvadali (contrib/raid)
RaidNode should be smarter in submitting Raid jobs

The RaidNode currently computes parity files as follows:
1. Using RaidNode.selectFiles() to figure out what files to raid for a policy
2. Using #1 repeatedly for each configured policy to accumulate a list of files.
3. Submitting a mapreduce job with the list of files from #2 using DistRaid.doDistRaid()

This task addresses the fact that #2 and #3 happen sequentially. The proposal is to submit a separate mapreduce job for the list of files for each policy and use another thread to track the progress of the submitted jobs. This will help reduce the time taken for files to be raided.
MAPREDUCE-1818. Minor improvement reported by rvadali and fixed by rvadali (contrib/raid)
RaidNode should specify a pool name incase the cluster is using FairScheduler

contrib/fairscheduler (FairScheduler) supports scheduling based on pools. The RaidNode should specify a pool name based on configuration to make use of pools.
MAPREDUCE-1816. Minor improvement reported by rvadali and fixed by rvadali (contrib/raid)
HAR files used for RAID parity need to have configurable partfile size

RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number of files present in a HAR archive depends on the size of HAR part files - higher the size, lower the number of files.
The size of HAR part files is configurable through the setting har.partfile.size, but that is a global setting. This task introduces a new setting specific to raid.har.partfile.size, that is used in-turn to set har.partfile.size
MAPREDUCE-1813. Major bug reported by amareshwari and fixed by ravidotg (contrib/streaming)
NPE in PipeMapred.MRErrorThread

Some reduce tasks fail with following NPE
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540)
at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
at org.apache.hadoop.mapred.Child.main(Child.java:159)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517)
at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449)
MAPREDUCE-1804. Major new feature reported by shv and fixed by shv (benchmarks, test)
Stress-test tool for HDFS introduced in HDFS-708

This issue is to commit the SLive test developed in HDFS-708 to MR trunk.
MAPREDUCE-1798. Major improvement reported by boryas and fixed by boryas
normalize property names for JT kerberos principal names in configuration (from HADOOP 6633)
MAPREDUCE-1785. Minor improvement reported by eli and fixed by eli (contrib/streaming)
Add streaming config option for not emitting the key

PipeMapper currently does not emit the key when using TextInputFormat. If you switch to input formats (eg LzoTextInputFormat) the key will be emitted. We should add an option so users can explicitly make streaming not emit the key so they can change input formats without breaking or having to modify their existing programs.
MAPREDUCE-1784. Minor bug reported by eli and fixed by eli
IFile should check for null compressor

IFile assumes that when it has a codec it can always get a compressor. This fails when mapred.compress.map.output is true but the native libraries are not installed, resulting in an NPE:
Let's make IFile handle this case by logging and using non-compressed streams.l
MAPREDUCE-1783. Major improvement reported by rvadali and fixed by rvadali (contrib/fair-share)
Task Initialization should be delayed till when a job can be run

The FairScheduler task scheduler uses PoolManager to impose limits on the number of jobs that can be running at a given time. However, jobs that are submitted are initiaiized immediately by EagerTaskInitializationListener by calling JobInProgress.initTasks. This causes the job split file to be read into memory. The split information is not needed until the number of running jobs is less than the maximum specified. If the amount of split information is large, this leads to unnecessary memory pressure on the Job Tracker.
To ease memory pressure, FairScheduler can use another implementation of JobInProgressListener that is aware of PoolManager limits and can delay task initialization until the number of running jobs is below the maximum.
MAPREDUCE-1780. Major bug reported by ravidotg and fixed by ravidotg (jobtracker)
AccessControlList.toString() is used for serialization of ACL in JobStatus.java

HADOOP-6715 is created to fix AccessControlList.toString() for the case of WILDCARD. JobStatus.write() and readFields() assume that toString() returns the serialized String of AccessControlList object, which is not true. Once HADOOP-6715 gets fixed in COMMON, JobStatus.write() and JobStatus.readFields() should be fixed depending on the fix of HADOOP-6715.
MAPREDUCE-1778. Major improvement reported by amar_kamat and fixed by ramach (jobtracker)
CompletedJobStatusStore initialization should fail if {mapred.job.tracker.persist.jobstatus.dir} is unwritable

If {mapred.job.tracker.persist.jobstatus.dir} points to an unwritable location or mkdir of {mapred.job.tracker.persist.jobstatus.dir} fails, then CompletedJobStatusStore silently ignores the failure and disables CompletedJobStatusStore. Ideally the JobTracker should bail out early indicating a misconfiguration.
MAPREDUCE-1773. Major bug reported by aloknsingh and fixed by amareshwari (contrib/streaming)
streaming doesn't support jobclient.output.filter

the streaming Jobclient implementation i.e contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java is significantly different than the core hadoop mapred/org/apache/hadoop/mapred/JobClient.java.
for example unlike StreamJob.java, JobClient.java it gets tasks log when jobclient.output.filter=ALL is specified .
With hod-logs going away in hadoop 0.20 (due to new scheduler) user has no good way of programmitically getting logs
We should have intermediate adaptor class to implement Tools for the purpose of submitting jobs via m/r, streaming, pipes so that we don't miss some core functionality.
GenericJobClient implements Tools and then StreamJob extends GenericJobClient, JobClient extends GenericJobClient
MAPREDUCE-1772. Minor bug reported by menicosia and fixed by amareshwari (contrib/streaming, documentation)
Hadoop streaming doc should not use IdentityMapper as an example

From the URL http://hadoop.apache.org/core/docs/current/streaming.html

This example doesn't work:
You can supply a Java class as the mapper and/or the reducer. The above example is equivalent to:

$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
-input myInputDirs \
-output myOutputDir \
-mapper org.apache.hadoop.mapred.lib.IdentityMapper \
-reducer /bin/wc

This will produce the following exception:
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
MAPREDUCE-1762. Major improvement reported by schen and fixed by schen
Add a setValue() method in Counter

Counters are very useful because of the logging and transmitting are already there.
It is very convenient to transmit and store numbers. But currently Counter only has an increment() method.
It will be nice if there can be a setValue() method in this class that will allow us to transmit wider variety of information through it.

What do you think?
MAPREDUCE-1761. Major improvement reported by schen and fixed by schen
FairScheduler should allow separate configuration of node and rack locality wait time

It would be nice that we can separately assign rack locality wait time.
In our use case, we would set node locality wait to zero and wait only rack locality.

I propose that we add two parameters
mapred.fairscheduler.locality.delay.nodetorack
mapred.fairscheduler.locality.delay.racktoany
This allows specifying the wait time on each stage.

And we can use
mapred.fairscheduler.locality.delay
as the default value of the above fields so that this is backward compatible.

Thoughts?
MAPREDUCE-1754. Major bug reported by amareshwari and fixed by amareshwari (jobtracker)
Replace mapred.persmissions.supergroup with an acl : mapreduce.cluster.administrators

mapred.permissions.supergroup should be replaced with an acl so that it does not restrict the admins to a single group.
See more details on MAPREDUCE-1542.
MAPREDUCE-1733. Major new feature reported by jnp and fixed by jnp
Authentication between pipes processes and java counterparts.

The connection between a pipe process and its parent java process should be authenticated.
MAPREDUCE-1718. Major bug reported by boryas and fixed by boryas
job conf key for the services name of DelegationToken for HFTP url is constructed incorrectly in HFTPFileSystem

the key (build in TokenCache) is hdfs.service.host_HOSTNAME.PORT, but
in HftpFileSystem it is sometimes built as hdfs.service.host_IP.PORT.

Fix. change it to always be IP.
MAPREDUCE-1711. Major improvement reported by hong.tang and fixed by rksingh (contrib/gridmix)
Gridmix should provide an option to submit jobs to the same queues as specified in the trace.

Gridmix should provide an option to submit jobs to the same queues as specified in the trace.
MAPREDUCE-1707. Major bug reported by amareshwari and fixed by vinodkv (tasktracker)
TaskRunner can get NPE in getting ugi from TaskTracker

The following code in TaskRunner can get NPE in the scenario described below.
UserGroupInformation ugi =
tracker.getRunningJob(t.getJobID()).getUGI();

The scenario:
Tracker got a LaunchTaskAction; Task is localized and TaskRunner is started.
Then Tracker got a KillJobAction; This would issue a kill for the task. But, kill will be a no-op because the task did not actually start; The job is removed from runningJobs.
Then if TaskRunner calls tracker.getRunningJob(t.getJobID()), it will be null.

Instead of TaskRunner doing a back call to tasktracker to get the ugi, tracker.getRunningJob(t.getJobID()).getUGI(), ugi should be passed a parameter in the constructor of TaskRunner.
MAPREDUCE-1701. Major bug reported by devaraj and fixed by boryas (jobtracker)
AccessControlException while renewing a delegation token in not correctly handled in the JobTracker

The timer task for renewing delegation token gets scheduled even when an AccessControlException is obtained.
MAPREDUCE-1686. Minor bug reported by pburkhardt and fixed by pburkhardt (contrib/streaming, test)
ClassNotFoundException for custom format classes provided in libjars

The StreamUtil::goodClassOrNull method assumes user-provided classes have package names and if not, they are part of the Hadoop Streaming package. For example, using custom InputFormat or OutputFormat classes without package names will fail with a ClassNotFound exception which is not indicative given the classes are provided in the libjars option. Admittedly, most Java packages should have a package name so this should rarely come up.
MAPREDUCE-1683. Major bug reported by chris.douglas and fixed by vicaya (jobtracker)
Remove JNI calls from ClusterStatus cstr

The {{ClusterStatus}} constructor makes two JNI calls to the {{Runtime}} to fetch memory information. {{ClusterStatus}} instances are often created inside the {{JobTracker}} to obtain other, unrelated metrics (sometimes from schedulers' inner loops). Given that this information is related to the {{JobTracker}} process and not the cluster, the metrics are also available via {{JvmMetrics}}, and the jsps can gather this information for themselves: these fields can be removed from {{ClusterStatus}}
MAPREDUCE-1680. Major new feature reported by hong.tang and fixed by dking (jobtracker)
Add a metrics to track the number of heartbeats processed

It would be nice to add a metrics that tracks the number of heartbeats processed by JT.
MAPREDUCE-1670. Major bug reported by rschmidt and fixed by rvadali (contrib/raid)
RAID should avoid policies that scan their own destination path

Raid currently allows policies that include the destination directory into the source directory and vice-versa.
Both situations can create cycles and should be avoided.
MAPREDUCE-1668. Major bug reported by rschmidt and fixed by rvadali (contrib/raid)
RaidNode should only Har a directory if all its parity files have been created

In the current code, it can happen that a directory will be Archived (Har'ed) before all its parity files have been generated since parity file generation is not atomic. We should verify if all the parity files are present before Archiving a directory.
MAPREDUCE-1664. Major bug reported by ravidotg and fixed by ravidotg (security)
Job Acls affect Queue Acls

MAPREDUCE-1307 introduced job ACLs for securing job level operations. So in current trunk, queue ACLs and job ACLs are checked(with AND for both acls) for allowing job level operations. So for doing operations like killJob, killTask and setJobPriority user should be part of both mapred.queue.{queuename}.acl-administer-jobs and in mapreduce.job.acl-modify-job. This needs to change so that users who are part of mapred.queue.{queuename}.acl-administer-jobs will be able to do killJob,killTask,setJobPriority and users part of mapreduce.job.acl-modify-job will be able to do killJob,killTask,setJobPriority.
MAPREDUCE-1662. Major bug reported by amareshwari and fixed by amareshwari (tasktracker)
TaskRunner.prepare() and close() can be removed

TaskRunner.prepare() and close() methods call only mapOutputFile.removeAll(). The removeAll() call is a always a no-op in prepare(), because the directory is always empty during start up of the task. The removeAll() call in close() is useless, because it is followed by a attempt directory cleanup. Since the map output files are in attempt directory, the call to close() is useless.
After MAPREDUCE-842, these calls are under TaskTracker space, passing the wrong conf. Now, the calls do not make sense at all.
I think we can remove the methods.
MAPREDUCE-1626. Major improvement reported by tomwhite and fixed by jolly (documentation)
Publish Javadoc for all contrib packages with user-facing APIs

Some packages don't appear in the Javadoc. E.g. MRUnit, Vertica.
MAPREDUCE-1621. Major bug reported by tlipcon and fixed by amareshwari (contrib/streaming)
Streaming's TextOutputReader.getLastOutput throws NPE if it has never read any output

If TextOutputReader.readKeyValue() has never successfully read a line, then its bytes member will be left null. Thus when logging a task failure, PipeMapRed.getContext() can trigger an NPE when it calls outReader_.getLastOutput().
MAPREDUCE-1617. Major bug reported by amareshwari and fixed by vicaya (test)
TestBadRecords failed once in our test runs

org.apache.hadoop.mapred.TestBadRecords.testBadMapRed failed with the following
exception:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
at org.apache.hadoop.mapred.TestBadRecords.runMapReduce(TestBadRecords.java:94)
at org.apache.hadoop.mapred.TestBadRecords.testBadMapRed(TestBadRecords.java:211)
MAPREDUCE-1599. Major bug reported by jnp and fixed by jnp
MRBench reuses jobConf and credentials there in.

MRBench reuses the jobconf and therefore credentials are re-used, but JobTracker cancels the delegation tokens therefore the test fails sometimes.
The fix is to pass the mapreduce.job.complete.cancel.delegation.tokens=false in the jobconf so that JobTracker does not cancel the tokens.
MAPREDUCE-1597. Major bug reported by namit and fixed by amareshwari
combinefileinputformat does not work with non-splittable files

CombineFileInputFormat.getSplits() does not take into account whether a file is splittable.
This can lead to a problem for compressed text files - for example, getSplits() may return more
than 1 split depending on the size of the compressed file, all the splits recordreader will read the
complete file.

I ran into this problem while using Hive on hadoop 20.
MAPREDUCE-1594. Major new feature reported by rksingh and fixed by rksingh (contrib/gridmix)
Support for Sleep Jobs in gridmix

Support for Sleep jobs in gridmix
MAPREDUCE-1592. Major improvement reported by tomwhite and fixed by tomwhite (build)
Generate Eclipse's .classpath file from Ivy config

MapReduce companion issue for HADOOP-6407.
MAPREDUCE-1566. Major bug reported by owen.omalley and fixed by jnp (security)
Need to add a mechanism to import tokens and secrets into a submitted job.

We need to include tokens and secrets into a submitted job. I propose adding a configuration attribute that when pointed at a token storage file will include the tokens and secrets from that token storage file.
MAPREDUCE-1559. Major bug reported by devaraj and fixed by devaraj (jobtracker)
The DelegationTokenRenewal timer task should use the jobtracker's credentials to create the filesystem

The submitJob RPC finally creates a timer task for renewing the delegation tokens of the submitting user. This timer task inherits the context of the RPC handler that runs in the context of the job submitting user, and when it tries to create a filesystem, the RPC client tries to use the user's credentials. This should instead use the JobTracker's credentials.
MAPREDUCE-1558. Major bug reported by boryas and fixed by boryas (security)
specify correct server principal for RefreshAuthorizationPolicyProtocol and RefreshUserToGroupMappingsProtocol protocols in MRAdmin (for HADOOP-6612)
MAPREDUCE-1548. Major improvement reported by rschmidt and fixed by rschmidt (harchive)
Hadoop archives should be able to preserve times and other properties from original files

Files inside hadoop archives don't keep their original:

- modification time
- access time
- permission
- owner
- group

all such properties are currently taken from the file storing the archive index, and not the stored files. This doesn't look very correct.

There should be possible to preserve the original properties of the stored files.
MAPREDUCE-1546. Minor improvement reported by schen and fixed by schen
Jobtracker JSP pages should automatically redirect to the corresponding history page if not in memory

MAPREDUCE-1185 redirects jobdetails.jsp to it's corresponding history page.

For convenience, we should also redirect the following JSP pages to the corresponding history pages:
jobconf.jsp
jobtasks.jsp
taskdetails.jsp
taskstats.jsp
MAPREDUCE-1545. Major improvement reported by acmurthy and fixed by vicaya (jobtracker)
Add 'first-task-launched' to job-summary

It would be useful to track 'first-task-launched' time to job-summary for better reporting.
MAPREDUCE-1543. Major bug reported by vinodkv and fixed by vicaya (security)
Log messages of JobACLsManager should use security logging of HADOOP-6586

{{JobACLsManager}} added in MAPREDUCE-1307 logs the successes and failures w.r.t job-level authorization in the corresponding Daemons' logs. The log messages should instead use security logging of HADOOP-6586.
MAPREDUCE-1533. Major bug reported by rajesh.balamohan and fixed by dking (jobtracker)
Reduce or remove usage of String.format() usage in CapacityTaskScheduler.updateQSIObjects and Counters.makeEscapedString()

When short jobs are executed in hadoop with OutOfBandHeardBeat=true, JT executes heartBeat() method heavily. This internally makes a call to CapacityTaskScheduler.updateQSIObjects().

CapacityTaskScheduler.updateQSIObjects(), internally calls String.format() for setting the job scheduling information. Based on the datastructure size of "jobQueuesManager" and "queueInfoMap", the number of times String.format() gets executed becomes very high. String.format() internally does pattern matching which turns to be out very heavy (This was revealed while profiling JT. Almost 57% of time was spent in CapacityScheduler.assignTasks(), out of which String.format() took 46%.

Would it be possible to do String.format() only at the time of invoking JobInProgress.getSchedulingInfo?. This might reduce the pressure on JT while processing heartbeats.
MAPREDUCE-1532. Major bug reported by devaraj and fixed by devaraj (job submission, security)
Delegation token is obtained as the superuser

When the UserGroupInformation.doAs is invoked for proxy users, the delegation token is incorrectly obtained as the real user.
MAPREDUCE-1528. Major bug reported by owen.omalley and fixed by jnp
TokenStorage should not be static

Currently, TokenStorage is a singleton. This doesn't work for some use cases, such as Oozie. I think that each Job should have a TokenStorage that is associated it.
MAPREDUCE-1526. Major improvement reported by rksingh and fixed by rksingh (contrib/gridmix)
Cache the job related information while submitting the job , this would avoid many RPC calls to JobTracker.
MAPREDUCE-1517. Major new feature reported by sinofool and fixed by sinofool (contrib/streaming)
streaming should support running on background

StreamJob submit the job and use a while loop monitor the progress.
I prefer it running on background.

Just add "&" at the end of command is a alternative solution, but it keeps a java process on client machine.
When submit hundreds jobs at the same time, the client machine is overloaded.

Adding a -background option to StreamJob, tell it only submit and don't monitor the progress.
MAPREDUCE-1516. Major new feature reported by jnp and fixed by jnp
JobTracker should issue a delegation token only for kerberos authenticated client

Delegation tokens should be issued only if the client is kerberos authenticated.
MAPREDUCE-1505. Major bug reported by devaraj and fixed by dking (client)
Cluster class should create the rpc client only when needed

It will be good to have the org.apache.hadoop.mapreduce.Cluster create the rpc client object only when needed (when a call to the jobtracker is actually required). org.apache.hadoop.mapreduce.Job constructs the Cluster object internally and in many cases the application that created the Job object really wants to look at the configuration only. It'd help to not have these connections to the jobtracker especially when Job is used in the tasks (for e.g., Pig calls mapreduce.FileInputFormat.setInputPath in the tasks and that requires a Job object to be passed).

In Hadoop 20, the Job object internally creates the JobClient object, and the same argument applies there too.
MAPREDUCE-1492. Major improvement reported by rschmidt and fixed by rschmidt (contrib/raid)
Delete or recreate obsolete har files used on hdfs raid

The current code for har on raid doesn't delete or recreate har directories when they become obsolete. We should fix that.
MAPREDUCE-1382. Major improvement reported by schen and fixed by zshao
MRAsyncDiscService should tolerate missing local.dir

Currently when some of the local.dir do not exist, MRAsyncDiscService will fail. It should only fail when all directories don't work.
MAPREDUCE-1376. Major improvement reported by chris.douglas and fixed by chris.douglas (contrib/gridmix)
Support for varied user submission in Gridmix

Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster.
MAPREDUCE-1375. Major bug reported by amar_kamat and fixed by tlipcon (contrib/streaming, test)
TestFileArgs fails intermittently

TestFileArgs failed once for me with the following error
expected:<[job.jar
sidefile
tmp
]> but was:<[]>
sidefile
tmp
]> but was:<[]>
at org.apache.hadoop.streaming.TestStreaming.checkOutput(TestStreaming.java:107)
at org.apache.hadoop.streaming.TestStreaming.testCommandLine(TestStreaming.java:123)
MAPREDUCE-1354. Critical improvement reported by devaraj and fixed by dking (jobtracker)
Incremental enhancements to the JobTracker for better scalability

It'd be nice to have the JobTracker object not be locked while accessing the HDFS for reading the jobconf file and while writing the jobinfo file in the submitJob method. We should see if we can avoid taking the lock altogether.
MAPREDUCE-1288. Critical bug reported by devaraj and fixed by devaraj (distributed-cache, security, tasktracker)
DistributedCache localizes only once per cache URI

As part of the file localization the distributed cache localizer creates a copy of the file in the corresponding user's private directory. The localization in DistributedCache assumes the key as the URI of the cachefile and if it already exists in the map, the localization is not done again. This means that another user cannot access the same distributed cache file. We should change the key to include the username so that localization is done for every user.
MAPREDUCE-1253. Major improvement reported by anirban.dasgupta and fixed by anirban.dasgupta (contrib/mumak)
Making Mumak work with Capacity-Scheduler

In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.
MAPREDUCE-1248. Minor improvement reported by ruibang and fixed by ruibang (contrib/streaming)
Redundant memory copying in StreamKeyValUtil

I found that when MROutputThread collecting the output of Reducer, it calls StreamKeyValUtil.splitKeyVal() and two local byte-arrays are allocated there for each line of output. Later these two byte-arrays are passed to variable key and val. There are twice memory copying here, one is the System.arraycopy() method, the other is inside key.set() / val.set().

This causes double times of memory copying for the whole output (may lead to higher CPU consumption), and frequent temporay object allocation.
MAPREDUCE-1225. Major bug reported by vinodkv and fixed by zhong (tasktracker)
TT successfully localizes a task even though the corresponding cache-file has already changed on DFS.

This happens with the first task of a job being localized on this TT. TT doesn't check if the file on DFS is fresh according to the timestamps set in job-conf during submission. After the first task incorrectly gets localized, all further tasks fail on this TT as expected.

Found this issue while trying to improve test-case for MAPREUCE-913.
MAPREDUCE-1159. Trivial improvement reported by zshao and fixed by qwertymaniac
Limit Job name on jobtracker.jsp to be 80 char long

Sometimes a user submits a job with a very long job name. That made jobtracker.jsp very hard to read.
We should limit the size of the job name. User can see the full name when they click on the job.
MAPREDUCE-1118. Major bug reported by aw and fixed by ramach (contrib/capacity-sched)
Capacity Scheduler scheduling information is hard to read / should be tabular format

The scheduling information provided by the capacity scheduler is extremely hard to read on the job tracker web page. Instead of just flat text, it should be presenting the information in a tabular format, similar to what the fair share scheduler provides. This makes it much easier to compare what different queues are doing.
MAPREDUCE-1092. Major test reported by eli and fixed by eli (test)
Enable asserts for tests by default

See HADOOP-6309. Let's make the tests run with java asserts by default.
MAPREDUCE-1085. Minor bug reported by ravidotg and fixed by tlipcon (tasktracker)
For tasks, "ulimit -v -1" is being run when user doesn't specify mapred.child.ulimit

For tasks, "ulimit -v -1" is being run when user doesn't specify mapred.child.ulimit. Taking -1 as default value and using it in building the command is not right.
MAPREDUCE-927. Major sub-task reported by vinodkv and fixed by amareshwari (security, tasktracker)
Cleanup of task-logs should happen in TaskTracker instead of the Child

Task logs' cleanup is being done in Child now. This is undesirable atleast for two reasons: 1) failures while cleaning up will affect the user's tasks, and 2) the task's wall time will get affected due to operations that TT actually should own.
MAPREDUCE-869. Major bug reported by acmurthy and fixed by tucu00 (documentation, task)
Documentation for config to set map/reduce task environment

HADOOP-2838 added mapred.child.env to allow users to set the map/reduce tasks' environment.

MAPREDUCE-478 will break that into mapred.map.child.env and mapred.reduce.child.env. We need to add documentation (forrest) for these knobs.
MAPREDUCE-714. Major bug reported by tlipcon and fixed by tlipcon
JobConf.findContainingJar unescapes unnecessarily on Linux

In JobConf.findContainingJar, the path name is decoded using URLDecoder.decode(...). This was done by Doug in r381794 (commit msg "Un-escape containing jar's path, which is URL-encoded. This fixes things primarily on Windows, where paths are likely to contain spaces.") Unfortunately, jar paths do not appear to be URL encoded on Linux. If you try to use "hadoop jar" on a jar with a "+" in it, this function decodes it to a space and then the job cannot be submitted.
MAPREDUCE-647. Major bug reported by rschmidt and fixed by rschmidt (documentation)
Update the DistCp forrest doc to make it consistent with the latest changes (5472, 5620, 5762, 5826)

New features have been added to DistCp and the documentation must be updated.
MAPREDUCE-577. Major bug reported by camda03 and fixed by ravidotg (contrib/streaming)
Duplicate Mapper input when using StreamXmlRecordReader

I believe that the problem is in StreamXmlRecordReader.java in the fast-match code. When the bytestream doesn't completely match the begin/end pattern, the code does "pos_ += m" to increment the position in the stream by the number of characters matching the pattern... but it forgets the "c" character which didn't match. Ultimately, the code ends up reading significantly past the end of the split, resulting in the same record (at least the first one right after the split) being detected by multiple StreamXmlRecordReader instances.
MAPREDUCE-572. Minor bug reported by karams and fixed by amareshwari (contrib/streaming)
If #link is missing from uri format of -cacheArchive then streaming does not throw error.

Ran hadoop streaming command as -:
bin/hadoop jar contrib/streaming/hadoop-*-streaming.jar -input in -output out -mapper "xargs cat" -reducer "bin/cat" -cahceArchive hdfs://h:p/pathofJarFile
Streaming submits job to jobtracker and map fails.
For similar with -cacheFile -:
bin/hadoop jar contrib/streaming/hadoop-*-streaming.jar -input in -output out -mapper "xargs cat" -reducer "bin/cat" -cahceFile hdfs://h:p/pathofFile
followinng error is repoerted back -:

You need to specify the uris as hdfs://host:port/#linkname,Please specify a different link name for all of your caching URIs

Streaming should check about present #link after uri of cacheArchive and should throw proper error .
MAPREDUCE-231. Major task reported by owen.omalley and fixed by owen.omalley
Split map/reduce into sub-project
MAPREDUCE-220. Major new feature reported by hong.tang and fixed by schen (task, tasktracker)
Collecting cpu and memory usage for MapReduce tasks

It would be nice for TaskTracker to collect cpu and memory usage for individual Map or Reduce tasks over time.

Changes since Hadoop 0.21.0

HADOOP-7194. Major bug reported by devaraj.k and fixed by devaraj.k (io)
Potential Resource leak in IOUtils.java

try {
copyBytes(in, out, buffSize);
} finally {
if(close) {
out.close();
in.close();
}
}

In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.
HADOOP-7193. Minor improvement reported by umamaheswararao and fixed by umamaheswararao (fs)
Help message is wrong for touchz command.

Help message for touchz command is
-touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
in a file at <path>. An error is returned if the file exists with non-zero length.

Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.

So better to change the help message to give exact meaning.
HADOOP-7183. Blocker bug reported by tlipcon and fixed by tomwhite
WritableComparator.get should not cache comparator objects

HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.
HADOOP-7177. Trivial improvement reported by aw and fixed by aw (native)
CodecPool should report which compressor it is using

Certain native compression libraries are overly verbose causing confusion while reading the task logs. Let's actually say which compressor we got when we report it in the task logs.
HADOOP-7174. Minor bug reported by umamaheswararao and fixed by umamaheswararao (fs)
null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine

When we perform copyToLocal operations from commandLine and if src Path is invalid

srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.

Since we are handling generic exception , it will display null as the message.
HADOOP-7162. Minor bug reported by humanoid and fixed by humanoid (fs)
FsShell: call srcFs.listStatus(src) twice

in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
call method twice:
1. for init variable
2. for getting data
HADOOP-7120. Major bug reported by szetszwo and fixed by szetszwo (test)
200 new Findbugs warnings

ant test-patch on an empty patch over hdfs trunk.

[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] -1 findbugs. The patch appears to introduce 200 new Findbugs (version 1.3.9) warnings.
[exec]
[exec] -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).
[exec]
[exec] +1 system test framework. The patch passed system test framework compile.
HADOOP-7117. Major improvement reported by patrickangeles and fixed by qwertymaniac (conf)
Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml

The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.
HADOOP-7072. Major sub-task reported by cos and fixed by cos (build)
Remove java5 dependencies from build

As the first short-term step let's remove JDK5 dependency from build(s)
HADOOP-7053. Minor bug reported by yaojingguo and fixed by yaojingguo (conf)
wrong FSNamesystem Audit logging setting in conf/log4j.properties

"log4j.logger.org.apache.hadoop.fs.FSNamesystem.audit=WARN" should be "log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=WARN".
HADOOP-7052. Major bug reported by yaojingguo and fixed by yaojingguo (conf)
misspelling of threshold in conf/log4j.properties

In "log4j.threshhold=ALL", threshhold is a misspelling of threshold. So "log4j.threshhold=ALL" has no effect on the control of log4j logging.
HADOOP-7019. Major bug reported by owen.omalley and fixed by vicaya (build)
Refactor build targets to enable faster cross project dev cycles.

The current build always generates fault injection artifacts and pushes them to Maven. Most developers have no need for these artifacts and no users need them.
HADOOP-6993. Major bug reported by atm and fixed by eli (documentation)
Broken link on cluster setup page of docs

The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
http://hadoop.apache.org/common/docs/current/common-default.html
but it should be:
http://hadoop.apache.org/common/docs/current/core-default.html
HADOOP-6971. Major bug reported by cos and fixed by cos (build, test)
Clover build doesn't generate per-test coverage

Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.
HADOOP-6969. Major bug reported by shv and fixed by tomwhite
CHANGES.txt does not reflect the release of version 0.21.0.
HADOOP-6954. Major bug reported by tomwhite and fixed by tomwhite (build)
Sources JARs are not correctly published to the Maven repository

When you try to "close" the staging repository to make it visible to the public the operation fails (see https://issues.apache.org/jira/browse/HDFS-1292?focusedCommentId=12909953&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909953 for the type of error).
HADOOP-6944. Major task reported by vinaythota and fixed by vinaythota (test)
[Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.

Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.

The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.

The following are methods signature,
public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.
HADOOP-6934. Major test reported by oae and fixed by oae (record)
test for ByteWritable comparator

The test for the ByteWritable comparator bug (see HADOOP-6928), which is already fixed in 0.21.
HADOOP-6925. Critical bug reported by tlipcon and fixed by tlipcon (io)
BZip2Codec incorrectly implements read()

HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.
HADOOP-6833. Blocker bug reported by tlipcon and fixed by tlipcon
IPC leaks call parameters when exceptions thrown

HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.
HADOOP-6786. Major improvement reported by cos and fixed by cos (build)
test-patch needs to verify Herriot integrity

Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.
HDFS-1750. Major bug reported by szetszwo and fixed by szetszwo
fs -ls hftp://file not working

hadoop dfs -touchz /tmp/file1 # create file. OK
hadoop dfs -ls /tmp/file1 # OK
hadoop dfs -ls hftp://namenode:50070/tmp/file1 # FAILED: not seeing the file
HDFS-1728. Minor bug reported by szetszwo and fixed by szetszwo (name-node)
SecondaryNameNode.checkpointSize is in byte but not MB.

The unit of SecondaryNameNode.checkpointSize is byte but not MB as stated in the following comment.

//SecondaryNameNode.java
private long checkpointSize; // size (in MB) of current Edit Log
HDFS-1665. Minor bug reported by szetszwo and fixed by szetszwo (balancer)
Balancer sleeps inadequately

The value of {{dfs.heartbeat.interval}} is in seconds. Balancer seems misused it.
HDFS-1612. Minor bug reported by joecrobak and fixed by joecrobak (documentation)
HDFS Design Documentation is outdated

I was trying to discover details about the Secondary NameNode, and came across the description below in the HDFS design doc.

The NameNode keeps an image of the entire file system namespace and file Blockmap in memory. This key metadata item is designed to be compact, such that a NameNode with 4 GB of RAM is plenty to support a huge number of files and directories. When the NameNode starts up, it reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage. This process is called a checkpoint. *In the current implementation, a checkpoint only occurs when the NameNode starts up. Work is in progress to support periodic checkpointing in the near future.*

(emphasis mine).

Note that this directly conflicts with information in the hdfs user guide, http://hadoop.apache.org/common/docs/r0.20.2/hdfs_user_guide.html#Secondary+NameNode
and http://hadoop.apache.org/hdfs/docs/current/hdfs_user_guide.html#Checkpoint+Node

I haven't done a thorough audit of that doc-- I only noticed the above inaccuracy.
HDFS-1598. Major bug reported by szetszwo and fixed by szetszwo (name-node)
ListPathsServlet excludes .*.crc files

The {{.*.crc}} files are excluded by default.
HDFS-1596. Major improvement reported by patrickangeles and fixed by qwertymaniac (documentation, name-node)
Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml

The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.
HDFS-1552. Major bug reported by cos and fixed by cos (build)
Remove java5 dependencies from build

As the first short-term step let's remove JDK5 dependency from build(s)
HDFS-1548. Major bug reported by cos and fixed by cos (build, test)
Fault-injection tests are executed multiple times if invoked with run-test-hdfs-fault-inject target

When invoked with {{run-test-hdfs-fault-inject target}} fault injection tests are getting executed 4 times.
HDFS-1474. Major bug reported by cos and fixed by cos (build)
ant binary-system is broken

Build failed due to unable to copy the commons instrumented jar. I could see the following error in the log.
HDFS-1452. Major bug reported by jghoman and fixed by cos (contrib/hdfsproxy)
ant compile-contrib is broken

ant compile-contrib is broken, looks like commit a0a62d971fb35de7f021ecbd6ceb8d08ef923ed5 HDFS-1444
HDFS-1444. Minor bug reported by cos and fixed by cos (build)
Test related code of build.xml is error-prone and needs to be re-aligned.

Test related parts of build.xml introduce at least two places (effectively different) for test classes destination compilation.
Then some extra logic is applied at say test-jar creation step where the content of one is copied over to another. Etc.

This seems to be overcomplicated and is better be fixed to prevent possible issues with future build modificaitons.
HDFS-1420. Major bug reported by cos and fixed by cos (build, test)
Clover build doesn't generate per-test coverage

Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.
HDFS-1416. Major bug reported by shv and fixed by tomwhite
CHANGES.txt does not reflect the release version 0.21.0.

CHANGES.txt still claims version 0.21.0 - Unreleased. It should show the release date instead and include section for for 0.21.1 - Unreleased. Latest changes, that did not make into 0.21.0, should be moved under 0.21.1 section.
This should also be fixed for common and mapreduce.
HDFS-1413. Major bug reported by shv and fixed by shv (documentation)
Broken links to HDFS Wiki in hdfs site and documentation.

# hdfs/site wiki tab points to "http://wiki.apache.org/hadoop/DFS", should be HDFS.
# hdfs documentation wiki tab points to "http://wiki.apache.org/hadoop/hdfs", should be HDFS.
HDFS-1411. Minor bug reported by chingshen and fixed by chingshen (documentation)
The startup command of the Backup Node is "bin/hdfs namenode -backup"

The startup command of the Backup Node is "bin/hdfs namenode -backup"
HDFS-1377. Blocker bug reported by eli and fixed by eli (name-node)
Quota bug for partial blocks allows quotas to be violated

There's a bug in the quota code that causes them not to be respected when a file is not an exact multiple of the block size.
HDFS-1363. Major bug reported by shv and fixed by shv
startFileInternal should return the last block of the file opened for append as an under-construction block

{{FSNamesystem.startFileInternal}} should convert the last block of the file opened for append to an under-construction block and return it. This will let remove the second synchronized section in {{FSNamesystem.appendFile()}} and avoid redundant computations and potential inconsistencies as stated in HDFS-1152.
HDFS-1343. Minor improvement reported by cos and fixed by cos (build)
Instrumented build should be concentrated in one build area

Current instrumented build is inefficient in a sense that part of it is build under build-fi and another part is under build-fi/system.
This doubles the time of some classes and tests compilation. And needed to be fixed.
HDFS-1206. Major bug reported by szetszwo and fixed by cos (test)
TestFiHFlush fails intermittently

When I was testing HDFS-1114, the patch passed all tests except TestFiHFlush. Then, I tried to print out some debug messages, however, TestFiHFlush succeeded after added the messages.

TestFiHFlush probably depends on the speed of BlocksMap. If BlocksMap is slow enough, then it will pass.
HDFS-1189. Major bug reported by xiaokang and fixed by johnvijoe (name-node)
Quota counts missed between clear quota and set quota

HDFS Quota counts will be missed between a clear quota operation and a set quota.

When setting quota for a dir, the INodeDirectory will be replaced by INodeDirectoryWithQuota and dir.isQuotaSet() becomes true. When INodeDirectoryWithQuota is newly created, quota counting will be performed. However, when clearing quota, the quota conf is set to -1 and dir.isQuotaSet() becomes false while INodeDirectoryWithQuota will NOT be replaced back to INodeDirectory.

FSDirectory.updateCount just update the quota count for inodes that isQuotaSet() is true. So after clear quota for a dir, its quota counts will not be updated and it's reasonable. But when re seting quota for this dir, quota counting will not be performed and some counts will be missed.
HDFS-996. Blocker bug reported by cos and fixed by cos (test)
JUnit tests should never depend on anything in conf

Similar to MAPREDUCE-1369 we need to make sure that nothing in conf is used in the unit tests.
MAPREDUCE-2317. Minor bug reported by devaraj.k and fixed by devaraj.k (harchive)
HadoopArchives throwing NullPointerException while creating hadoop archives (.har files)

While we are trying to run hadoop archive tool in widows using this way,
java org.apache.hadoop.tools.HadoopArchives -archiveName temp.har D:/test/in E:/temp
it is giving java.lang.NullPointerException
MAPREDUCE-2228. Major bug reported by cos and fixed by cos (build)
Remove java5 dependencies from build

As the first short-term step let's remove JDK5 dependency from build(s)
MAPREDUCE-2223. Minor bug reported by cos and fixed by cos (test)
TestMRCLI might fail on Ubuntu with default /etc/hosts

Depending on the order of entries in /etc/hosts, TestCLI can fail. This is because it sets fs.default.name to "localhost", and then the bound IPC socket on the NN side reports its hostname as "foobar-host" if the entry for 127.0.0.1 lists "foobar-host" before "localhost". This seems to be the default in some versions of Ubuntu.
MAPREDUCE-2140. Trivial improvement reported by matei and fixed by matei
Re-generate fair scheduler design doc PDF

The Fair Scheduler contains a design document in src/contrib/fairscheduler/designdoc that is included both as a Latex file and as a PDF. However, the PDF that's currently there is not generated properly and has some question marks for section references. I'd like to regenerate it and commit the new one. There is no patch to attach because this just requires running pdflatex and committing a binary file.
MAPREDUCE-2134. Major bug reported by vinaythota and fixed by cos (build)
ant binary-system is broken in mapreduce project.

Build failed due to unable to copy the commons instrumented jar.
MAPREDUCE-2090. Major bug reported by cos and fixed by cos (build, test)
Clover build doesn't generate per-test coverage

Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.
MAPREDUCE-2086. Major bug reported by shv and fixed by tomwhite
CHANGES.txt does not reflect the release of version 0.21.0.

CHANGES.txt should show the release date for 0.21.0 and include section for for 0.21.1 - Unreleased. Latest changes, that did not make into 0.21.0, should be moved under 0.21.1 section.
MAPREDUCE-2040. Minor new feature reported by sandholm and fixed by sandholm (contrib/dynamic-scheduler)
Forrest Documentation for Dynamic Priority Scheduler

New Forrest documentation for dynamic priority scheduler
MAPREDUCE-2032. Major bug reported by amareshwari and fixed by dking (task)
TestJobOutputCommitter fails in ant test run

TestJobOutputCommitter fails in a "ant test" run with following exception :

Output directory /home/amarsri/mapred/build/test/data/test-job-cleanup/output-2 already exists
org.apache.hadoop.fs.FileAlreadyExistsException: Output directory /home/amarsri/mapred/build/test/data/test-job-cleanup/output-2 already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:141)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:391)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:350)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1037)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1034)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1034)
at org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter.testKilledJob(TestJobOutputCommitter.java:192)
at org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter.testDefaultCleanupAndAbort(TestJobOutputCommitter.java:232)

But it passes when it is run individually.
MAPREDUCE-1984. Major bug reported by balajirg and fixed by balajirg
herriot TestCluster fails because exclusion is not there

restart is part of the test case which causes ioexceptions, and this needs to be ignored. The test case should not be incorrectly failed.
MAPREDUCE-1929. Blocker bug reported by tomwhite and fixed by tomwhite (build)
Allow artifacts to be published to the staging Apache Nexus Maven Repository

MapReduce companion issue to HADOOP-6847.
MAPREDUCE-1905. Blocker bug reported by amareshwari and fixed by amareshwari (task)
Context.setStatus() and progress() api are ignored

TaskAttemptContext.setStatus() and progress() were overriden in TaskInputOutputContext, inbranch 0.20, to call the underlying reporter apis. But the methods are no more over-riden in TaskInputOutputContextImpl after MAPREDUCE-954.
MAPREDUCE-1897. Major bug reported by roelofs and fixed by cos (test)
trunk build broken on compile-mapred-test

...apparently. Fresh checkout of trunk (all three hadoop-*), build.properties project.version fix, ant veryclean mvn-install of common, hdfs, and then mapreduce:
MAPREDUCE-1809. Major task reported by vinaythota and fixed by vinaythota (build)
Ant build changes for Streaming system tests in contrib projects.

Implementing new target( test-system) in build-contrib.xml file for executing the system test that are in contrib projects. Also adding 'subant' target in aop.xml that calls the build-contrib.xml file for system tests.
MAPREDUCE-1501. Major improvement reported by zshao and fixed by zshao
FileInputFormat to support multi-level/recursive directory listing

As we have seen multiple times in the mailing list, users want to have the capability of getting all files out of a multi-level directory structure.
One solution that our users had is to write a new FileInputFormat, but that means all existing FileInputFormat subclasses need to be changed in order to support this feature.
We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}} to recursively go into directory structure.
MAPREDUCE-1280. Major bug reported by kimballa and fixed by alexvk
Eclipse Plugin does not work with Eclipse Ganymede (3.4)

The newest version of Eclipse seems incompatible with the plugin. The plugin as released in 0.16.4 will allow you to add/remove MapReduce servers, and will allow you to browse/manipulate the DFS in the DFS Browser, but will not allow you to run programs. Clicking "Run As * Run On Hadoop" will simply not cause the run-on-hadoop server selection window to appear. No error message is given.

Dropping the 0.17.1 copy of the plugin JAR into the eclipse/plugins/ directory does not fix the issue; it is in fact worse: Eclipse does not seem to regard the 0.17 plugin as real. No "MapReduce Perspective" is made available in the perspectives selection window.