Branch-hadoop-1-win (branched from branch-1 at r1293509 on 2012-02-24 17:46:42) HADOOP-8223 Initial patch for branch-1-win. (David Lao via Sanjay) HADOOP-8234 Enable user group mappings on Windows. (Bikas Saha via Sanjay) MAPREDUCE-4201 Getting PID not working on Windows. Termination of Task/TaskJVM's not working. (Bikas Saha via Sanjay) MAPREDUCE-4204 Refactor ProcfsBasedProcessTree to make the resource collection object pluggable. (Bikas Saha via Sanjay) HADOOP-8412 TestModTime, TestDelegationToken and TestAuthenticationToken fail intermittently on Windows. (Ivan Mitic via Sanjay Radia) HADOOP-8235 Support file permissions and ownership on Windows for RawLocalFileSystem. (Chuan Liu via sanjay) HADOOP-8411 TestStorageDirecotyFailure, TestTaskLogsTruncater, TestWebHdfsUrl and TestSecurityUtil fail on Windows. (Ivan Mitic via Sanjay Radia) HADOOP-8440 HarFileSystem.decodeHarURI fails for URIs whose host contains numbers. (Ivan Mitic via Sanjay Radia) HADOOP-8374 Improve support for hard link manipulation on Windows. (Bikas Saha via Sanjay) HDFS-3424 TestDatanodeBlockScanner and TestReplication fail intermittently on Windows (Bikas Saha via Sanjay) HADOOP-8424. Fix classpath so that web-ui works on Windows. (Bikas Saha via acmurthy) MAPREDUCE-4321. Fix both DCE & LCE to use File.getAbsolutePath of the taskjvm.(sh,cmd) to ensure it works on Windows. (Ivan Mitic via acmurthy) MAPREDUCE-4260 Use JobObject to spawn tasks on Windows (Bikas Saha via Sanjay) HADOOP-8440 HarFileSystem.decodeHarURI 2nd patch. (Ivan Mitic via Sanjay Radia) HADOOP-8409 Fix TestCommandLineJobSubmission and TestGenericOptionsParser to work for windows (Ivan Mitic via Sanjay Radia) HADOOP-8454 Fix the ‘chmod =[perm]’ bug in winutils. (Chuan Liu via sanjay) MAPREDUCE-4203. Added an implementation of the process tree for Windows. (Bikas Saha via acmurthy) HADOOP-8486 Resource leak ... SequenceFile. (Kanna Karanam via Sanjay) HADOOP-8534 Some tests leave a config file open causing failure on windows. (Ivan Mitic via Sanjay Radia) HADOOP-8487 Many HDFS tests use a test path intended for local file system tests. (Ivan Mitic via Sanjay Radia) MAPREDUCE-4332 Fix command-length abort issues on Windows. (Ivan Mitic via acmurthy) MAPREDUCE-4369. Fix streaming test failures caused by WindowsResourceCalculatorPlugin. (Bikas Saha via acmurthy) MAPREDUCE-4368. Fix TaskRunner to deal with java.library.path with a quoted path on Windows. (John Gordon via acmurthy) HADOOP-8414 Address problems related to localhost resolving to 127.0.0.1 on Windows. (Ivan Mitic via Sanjay Radia) HADOOP-8544 Move an assertion location in 'winutils chmod'. (Chuan Liu via sanjay) MAPREDUCE-4598. Add support for NodeHealthCheck script for Windows. (Bikas Saha via acmurthy) MAPREDUCE-4510. Fix needless check/logging of getconf in Windows. (Bikas Saha via acmurthy) HADOOP-8453. Added unit tests for winutils. (Chuan Liu via acmurthy) MAPREDUCE-4374. Completed support for configurable environment for child map/reduce tasks. (Chuan Liu via acmurthy) HADOOP-8734. Modify LocalJobRunner to support private distributed cache. (Ivan Mitic via vinodkv) HDFS-3564. Improve BlockPlacementPolicy interface and change BlockPlacementPolicyDefault to make it more extendible. (Sumadhur Reddy Bolli via szetszwo) HDFS-3566. Add AzureBlockPlacementPolicy to handle fault and upgrade domains in Azure. (Sumadhur Reddy Bolli via szetszwo) HADOOP-8911. CRLF characters in source and text files. HDFS-6527. Backport HADOOP-7389: Use of TestingGroups by tests causes subsequent tests to fail. (Ivan Mitic via szetszwo) HADOOP-8618. Fix build failures due to release 1.0.3 merge. (Bikas Saha via suresh) HDFS-3763. TestNameNodeMXBean fails on Windows. (Brandon Li via suresh) HADOOP-8739. Fix command line parsing on Windows. (Bikas Saha via acmurthy) HDFS-3766. Fix TestStorageRestore on Windows. (Brandon Li via suresh) HADOOP-8657. Fix TestCLI to not hardcode file length since it's encoding/platform dependent. (Bikas Saha via acmurthy) MAPREDUCE-4597. Fix intermittent failures in TestKillSubProcesses. (Bikas Saha via acmurthy) HADOOP-8732. Fix intermittent failures in tests by serializing process creation to get around a bug in MS Windows. (Ivan Mitic via acmurthy) HDFS-3833. TestDFSShell fails on windows due to concurrent file read/write. (Brandon Li via suresh) HADOOP-8664. hadoop streaming job need the full path to commands even when they are in the path. (Bikas Saha via suresh) MAPREDUCE-4564. Fix issues with timing out shell processes. (Bikas Saha via acmurthy) HADOOP-8457. Address file ownership issue for users in Admin group on windows. (Ivan Mitic via suresh) MAPREDUCE-4400. Avoid task finish sleep to improve small job/workflow latency. (llu) HADOOP-8733. Fix the tests TestStreamingTaskLog, TestJvmManager and TestLinuxTaskControllerLaunchArgs to not fail on Windows. (Ivan Mitic via vinodkv) HADOOP-8694. Add support for windows native symbolic links. (Chuan Liu via suresh) HADOOP-8836. UGI should throw exception in case winutils.exe cannot be loaded. (Bikas Saha via suresh) MAPREDUCE-4657. WindowsResourceCalculatorPlugin has NPE. (Bikas Saha via suresh) HADOOP-8879. TestUserGroupInformation fails on Windows when run as Administrator. (Bikas Saha via suresh) HADOOP-8880. Missing jersey jars as dependency in the pom causes hive tests to fail. (gkesavan via suresh) HADOOP-8903. support HADOOP_USER_CLASSPATH_FIRST in windows hadoop cmd. (Thejas Nair via suresh) HADOOP-8908. Refactor winutil.exe related code. (Chuan Liu via suresh) HADOOP-8911. CRLF characters in source and text files. (Raja Aluri via suresh) HADOOP-8912. Add .gitattributes file to prevent CRLF and LF mismatches for source and text files. (Raja Aluri via suresh) HADOOP-8868. FileUtil#chmod should normalize the path before calling into shell APIs. (Ivan Mitic via suresh) HADOOP-8936. Local users should be able to query a domain user's groups on Windows. (Chuan Liu via suresh) HADOOP-8869. Links at the bottom of the jobdetails page do not render correctly in IE9. (Ivan Mitic via suresh) HADOOP-8935. Make 'winutils ls' show the SID if the owner does not exist on the system. (Chuan Liu via suresh) HDFS-4065. TestDFSShell.testGet sporadically fails attempting to corrupt block files due to race condition. (Chris Nauroth via suresh) HADOOP-8899. Classpath exceeds maximum Windows command limit. (Ahmed El Baz via suresh) HADOOP-8763. Implement set-group-owner for Windows correctly. (Chuan Liu via vinodkv) HADOOP-8902. Enable Gridmix v1 & v2 benchmarks on Windows platform. (Mike Liddell via suresh) HADOOP-8564. Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue. (Chuan Liu via suresh) HADOOP-4093. Fix a bug that AzureBlockPlacementPolicy#chooseTarget only returns one DN when replication factor is greater than 3. (Jing Zhao via szetszwo) HADOOP-8907. Provide means to look for zlib1.dll next to hadoop.dll on Windows. (Ivan Mitic via suresh) HADOOP-8872. FileSystem#length returns zero for symlinks on windows+java6. (Ivan Mitic via suresh) HADOOP-8949. Remove FileUtil.CygPathCommand dead code. (Chris Nauroth via suresh) HADOOP-8420. Hadoop Common creating package-info.java must not depend on sh. (Bikas Saha via suresh) HADOOP-8972. Move winutils tests from bat to Java. (Ivan Mitic via suresh) HADOOP-8983. Minor fixes to windows batch scripts to accept alternate config directory location. (Ramya Bharathi Nimmagadda via suresh) HADOOP-9007. TestJobStatusPersistency.testJobStoreDisablingWithInvalidPath failure. (Chuan Liu via suresh) HADOOP-9006. Winutils should keep Administrators privileges intact. (Chuan Liu via suresh) HADOOP-9026. Hadoop.cmd fails to initialize if user's %path% variable has parenthesis. (Ivan Mitic via suresh) HADOOP-9027. Build fails on Windows without sh/sed/echo in the path. (Ivan Mitic via suresh) HADOOP-8847. Change untar to use Java API on Windows instead of spawning tar process. (Bikas Saha via suresh) HADOOP-8456. Support spaces in user names and group names in results returned via winutils. (Ivan Mitic via suresh) HADOOP-9050. Remove java5 dependencies from build. (Raja Aluri via suresh) HADOOP-9062. hadoop-env.cmd overwrites the value of *_OPTS set before install. (Ganeshan Iyer via suresh) HADOOP-9074. Hadoop install scripts for Windows. (Steve Maine, Ramya Nimmagadda, Chuan Liu, Brad Sarsfield and Ivan Mitic via suresh) HADOOP-9110. HADOOP-9110. winutils ls off-by-one error indexing MONTHS array can cause access violation. (Chris Nauroth via suresh) HADOOP-9102. winutils task isAlive does not return a non-zero exit code if the requested task is not alive. (Chris Nauroth via suresh) HADOOP-9061. Java6+Windows does not work well with symlinks. (Ivan Mitic via suresh) HADOOP-8645. HADOOP_HOME and -Dhadoop.home (from hadoop wrapper script) are not uniformly handled. (John Gordon via suresh) HADOOP-9185. TestFileCreation.testFsClose should clean up on exit. (Arpit Agarwal via suresh) HADOOP-9250. Windows installer bugfixes. (Ivan Mitic via suresh) HADOOP-9463. branch-1-win fails to build with OpenJDK7. (Ivan Mitic via suresh) HADOOP-8731. Fixed TrackerDistributedCacheManager to address failure in the test TestTrackerDistributedCacheManager on Windows. (Ivan Mitic via vinodkv) HADOOP-9480. Windows installer should pick the config from src\packages\win\template\conf. (Ivan Mitic via suresh) HADOOP-9452. Windows install scripts bugfixes. (Ivan Mitic via suresh) HADOOP-9472. Cleanup hadoop-config.cmd. (Ivan Mitic via suresh) HADOOP-9043. disallow in winutils creating symlinks with forwards slashes. (Chris Nauroth via suresh) HADOOP-9579. Contrib ant test target not setting the java.library.path. (ivanmi) HADOOP-9552. Windows log4j template should suppress info messages from mortbay.log. (ivanmi) HDFS-4677. Editlog should support synchronous writes. (ivanmi) MAPREDUCE-5224. JobTracker should allow the system directory to be in non-default FS. (Xi Fang via ivanmi) HADOOP-9681. FileUtil.unTarUsingJava() should close the InputStream upon finishing. (Chuan Liu via cnauroth) HADOOP-9677. TestSetupAndCleanupFailure#testWithDFS fails on Windows. (Xi Fang via cnauroth) MAPREDUCE-5330. JVM manager should not forcefully kill the process on Signal.TERM on Windows. (Xi Fang via cnauroth) HADOOP-9687. TestJobTrackerQuiescence and TestFileLengthOnClusterRestart failed caused by incorrect DFS path construction on Windows. (Xi Fang via cnauroth) HADOOP-9685. hadoop-config.cmd: builds a classpath that is too long on windows. (Raja Aluri via cnauroth) MAPREDUCE-5371. TestProxyUserFromEnv#testProxyUserFromEnvironment failed caused by domains of windows users. (Xi Fang via cnauroth) MAPREDUCE-5278. Distributed cache is broken when JT staging dir is not on the default FS. (Xi Fang via cnauroth) HADOOP-9718. Branch-1-win TestGroupFallback#testGroupWithFallback() failed caused by java.lang.UnsatisfiedLinkError. (Xi Fang via cnauroth) HADOOP-9719. Branch-1-win TestFsShellReturnCode#testChgrp() failed caused by incorrect exit codes. (Xi Fang via cnauroth) HADOOP-9722. Branch-1-win TestNativeIO failed caused by Window incompatible test case. (Xi Fang via cnauroth) HDFS-4975. Branch-1-win TestReplicationPolicy failed caused by stale data node handling. (Xi Fang via cnauroth) MAPREDUCE-5391. TestNonLocalJobJarSubmission fails on Windows due to missing classpath entries. (cnauroth) HDFS-5001. TestAzureBlockPlacementPolicy and TestReplicationPolicyWithNodeGroup failed caused by 1) old APIs and 2) incorrect value of depthOfAllLeaves. (Xi Fang via cnauroth) HADOOP-9609. Remove sh dependency of bin-package target. (Chuan Liu via cnauroth) HADOOP-9790. Job token path is not unquoted properly. (Xi Fang via cnauroth) HADOOP-9551. Backport common utils introduced with HADOOP-9413 to branch-1-win. (ivanmi) HADOOP-9802. Support Snappy codec on Windows. (cnauroth) HADOOP-9791. Add a test case covering long paths for new FileUtil access check methods (ivanmi) HADOOP-10094. NPE in GenericOptionsParser#preProcessForWindows(). (Enis Soztutar via cnauroth) Merged from branch-1 HDFS-385. Backport: Add support for an experimental API that allows a module external to HDFS to specify how HDFS blocks should be placed. (Sumadhur Reddy Bolli via szetszwo) HADOOP-8820. Backport HADOOP-8469 and HADOOP-8470: Make NetworkTopology class pluggable and add NetworkTopologyWithNodeGroup, a 4-layer implementation of NetworkTopology. (Junping Du and Jing Zhao via szetszwo) HDFS-3941. Backport HDFS-3498 and HDFS-3601: Support replica removal in BlockPlacementPolicy and make BlockPlacementPolicyDefault extensible for reusing code in subclasses, and add BlockPlacementPolicyWithNodeGroup to support block placement with 4-layer network topology. (Junping Du and Jing Zhao via szetszwo) HADOOP-7868. Hadoop native fails to compile when default linker option is -Wl,--as-needed. (Trevor Robinson via eli) HADOOP-6496. HttpServer sends wrong content-type for CSS files. (Todd Lipcon, Backport by Ivan Mitic via suresh). HADOOP-7827. jsp pages missing DOCTYPE tags. (Dave Vronay, Backport by Ivan Mitic via suresh) HADOOP-8900. BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size. (Andy Isaacson via suresh) HADOOP-9036. Fix racy test case TestSinkQueue (Backport HADOOP-7292). (Luke Lu backport by suresh) HADOOP-9095. Backport HADOOP-8372: NetUtils.normalizeHostName() incorrectly handles hostname starting with a numeric character. (Jing Zhao via szetszwo) HADOOP-9099. TestNetUtils fails if "UnknownHost" is resolved as a valid hostname. (Ivan Mitic via szetszwo) HADOOP-9111. Change some JUnit 3 tests to JUnit 4 so that @Ignore tests can be run with ant 1.8.x. (Jing Zhao via szetszwo) HADOOP-9090. Support on-demand publish of metrics. (Mostafa Elhemali via suresh) HADOOP-8617. Backport HADOOP-6148, HADOOP-6166 and HADOOP-7333 for a pure Java CRC32 calculator implementation. (Brandon Li via szetszwo) HADOOP-7096. Allow setting of end-of-record delimiter for TextInputFormat (Ahmed Radwan, backported by suresh) HDFS-3942. Backport HDFS-3495 and HDFS-4234: Update Balancer to support new NetworkTopology with NodeGroup and use generic code for choosing datanode in Balancer. (Junping Du via szetszwo) HDFS-4320. Add a separate configuration for namenode rpc address instead of using fs.default.name. (Mostafa Elhemali via suresh) HDFS-4341. Set default data dir permission in MiniDFSClusterWithNodeGroup. (Ivan Mitic via szetszwo) HDFS-4337. Backport HDFS-4240: For nodegroup-aware block placement, when a node is excluded, the nodes in the same nodegroup should also be excluded. (Meng Gong via szetszwo) HADOOP-9174. TestSecurityUtil fails with Open JDK 7. (Arpit Agarwal via suresh) HADOOP-9175. TestWritableName fails with Open JDK 7. (Arpit Agarwal via suresh) MAPREDUCE-4909. TestKeyValueTextInputFormat fails with Open JDK 7 on Windows. (Arpit Agarwal via suresh) HDFS-4355. TestNameNodeMetrics.testCorruptBlock fails with open JDK7. (Brandon Li via suresh) HDFS-4358. TestCheckpoint failure with JDK7. (Arpit Agarwal via suresh) HADOOP-9179. TestFileSystem fails with open JDK7. (Brandon Li via suresh) MAPREDUCE-4914. TestMiniMRDFSSort fails with openJDK7. (Brandon Li via suresh) MAPREDUCE-4915. TestShuffleExceptionCount fails with open JDK7. (Brandon Li via suresh) HADOOP-9191. TestAccessControlList and TestJobHistoryConfig fail with JDK7. (Arpit Agarwal via suresh) HADOOP-9177. Address issues that reported by static code analysis on winutils. (Ivan Mitic via suresh) MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7. (Arpit Agarwal via suresh) HADOOP-8973. DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs. (Chris Nauroth via suresh) MAPREDUCE-782. Use PureJavaCrc32 in mapreduce spills. (Todd Lipcon, backport by Brandon Li via sseth) HADOOP-9071. Configure ivy log levels for resolve/retrieve. (Giridharan Kesavan via suresh) HDFS-4413. Secondary namenode won't start if HDFS isn't the default file system. (Mostafa Elhemali via suresh) HADOOP-9467. Metrics2 record filter should check name as well as tags. (Ganeshan Iyler via llu) HADOOP-9450. HADOOP_USER_CLASSPATH_FIRST is not honored; CLASSPATH is PREpended instead of APpended. (Chris Nauroth and harsh via harsh) MAPREDUCE-5161. Merge MAPREDUCE-1806 from branch-1 to branch-1-win. CombineFileInputFormat fix for paths not on default FS (Chris Nauroth via bikas) MAPREDUCE-5066. Added a timeout for the job.end.notification.url. (Ivan Mitic via acmurthy) MAPREDUCE-4737. Ensure that mapreduce APIs are semantically consistent with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that cleanup is now called even if there is an error. The old mapred API already ensures that Mapper.close and Reducer.close are invoked during error handling. Note that it is an incompatible change, however end-users can override Mapper.run and Reducer.run to get the old (inconsistent) behaviour. (acmurthy) HDFS-4782. Backport edit log corruption toleration. (Chris Nauroth via szetszwo) HADOOP-8745. Incorrect version numbers in hadoop-core POM. (Matthias Friedrich via eli) HADOOP-8981. TestMetricsSystemImpl fails on Windows. (Xuan Gong, backported by Chris Nauroth via suresh) HADOOP-9624. TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has "X" in its name. (Xi Fang via cnauroth) HDFS-4261. Fix bugs in Balaner causing infinite loop and TestBalancerWithNodeGroup timeing out. (Junping Du via szetszwo) HDFS-3794. WebHDFS Open used with Offset returns the original (and incorrect) Content Length in the HTTP Header. (Tsz Wo (Nicholas), SZE via cnauroth) MAPREDUCE-5405. Job recovery can fail if task log directory symlink from prior run still exists. (cnauroth) HADOOP-9507. LocalFileSystem rename() is broken in some cases when destination exists. (cnauroth) HADOOP-9665. Fixed BlockDecompressorStream#decompress to return -1 rather than throw EOF at end of file. (Zhijie Shen via acmurthy) HADOOP-9768. chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable. (cnauroth) HADOOP-9801. Configuration#writeXml uses platform defaulting encoding, which may mishandle multi-byte characters. (cnauroth) HDFS-4963. Improve multihoming support in namenode. (Arpit Agarwal via cnauroth) HDFS-5211. Race condition between DistributedFileSystem#close and FileSystem#close can cause return of a closed DistributedFileSystem instance from the FileSystem cache. (cnauroth) MAPREDUCE-5351. Fixed a memory leak in JobTracker due to stable FS objects in FSCache. (Sandy Ryza via acmurthy) MAPREDUCE-5508. JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob. (Xi Fang via cnauroth) HADOOP-10142. Avoid groups lookup for unprivileged users such as "dr.who" (vinay, backported by Xi Fang via cnauroth) HDFS-5516. WebHDFS does not require user name when anonymous http requests are disallowed. (Miodrag Radulovic via cnauroth) MAPREDUCE-5808. Port output replication factor configurable for terasort to Hadoop 1.x. (Chuan Liu via cnauroth)