Hadoop 0.20.2 Release Notes

Changes Since Hadoop 0.20.1

Common

Bug

• [HADOOP-4802] - RPC Server send buffer retains size of largest response ever sent
• [HADOOP-5611] - C++ libraries do not build on Debian Lenny
• [HADOOP-5612] - Some c++ scripts are not chmodded before ant execution
• [HADOOP-5623] - Streaming: process provided status messages are overwritten every 10 seoncds
• [HADOOP-5759] - IllegalArgumentException when CombineFileInputFormat is used as job InputFormat
• [HADOOP-6097] - Multiple bugs w/ Hadoop archives
• [HADOOP-6231] - Allow caching of filesystem instances to be disabled on a per-instance basis
• [HADOOP-6269] - Missing synchronization for defaultResources in Configuration.addResource
• [HADOOP-6315] - GzipCodec should not represent BuiltInZlibInflater as decompressorType
• [HADOOP-6386] - NameNode's HttpServer can't instantiate InetSocketAddress: IllegalArgumentException is thrown
• [HADOOP-6428] - HttpServer sleeps with negative values
• [HADOOP-6460] - Namenode runs of out of memory due to memory leak in ipc Server
• [HADOOP-6498] - IPC client bug may cause rpc call hang
• [HADOOP-6506] - Failing tests prevent the rest of test targets from execution.
• [HADOOP-6524] - Contrib tests are failing Clover'ed build
• [HADOOP-6575] - Tests do not run on 0.20 branch
• [HADOOP-6576] - TestStreamingStatus is failing on 0.20 branch

Task

• [HADOOP-6328] - Hadoop 0.20 Docs - backport changes for streaming and m/r tutorial docs

HDFS

Bug

• [HDFS-101] - DFS write pipeline : DFSClient sometimes does not detect second datanode failure
• [HDFS-185] - Chown , chgrp , chmod operations allowed when namenode is in safemode .
• [HDFS-187] - TestStartup fails if hdfs is running in the same machine
• [HDFS-495] - Hadoop FSNamesystem startFileInternal() getLease() has bug
• [HDFS-579] - HADOOP-3792 update of DfsTask incomplete
• [HDFS-596] - Memory leak in libhdfs: hdfsFreeFileInfo() in libhdfs does not free memory for mOwner and mGroup
• [HDFS-677] - Rename failure due to quota results in deletion of src directory
• [HDFS-723] - Deadlock in DFSClient#DFSOutputStream
• [HDFS-732] - HDFS files are ending up truncated
• [HDFS-734] - TestDatanodeBlockScanner times out in branch 0.20
• [HDFS-745] - TestFsck timeout on 0.20.
• [HDFS-761] - Failure to process rename operation from edits log due to quota verification
• [HDFS-781] - Metrics PendingDeletionBlocks is not decremented
• [HDFS-793] - DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet
• [HDFS-795] - DFS Write pipeline does not detect defective datanode correctly in some cases (HADOOP-3339)
• [HDFS-872] - DFSClient 0.20.1 is incompatible with HDFS 0.20.2
• [HDFS-927] - DFSInputStream retries too many times for new block locations

Test

• [HDFS-784] - TestFsck times out on branch 0.20.1
• [HDFS-907] - Add tests for getBlockLocations and totalLoad metrics.
• [HDFS-919] - Create test to validate the BlocksVerified metric

MapReduce

Bug

• [MAPREDUCE-112] - Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API
• [MAPREDUCE-433] - TestReduceFetch failed.
• [MAPREDUCE-826] - harchive doesn't use ToolRunner / harchive returns 0 even if the job fails with exception
• [MAPREDUCE-979] - JobConf.getMemoryFor{Map|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1
• [MAPREDUCE-1010] - Adding tests for changes in archives.
• [MAPREDUCE-1068] - In hadoop-0.20.0 streaming job do not throw proper verbose error message if file is not present
• [MAPREDUCE-1070] - Deadlock in FairSchedulerServlet
• [MAPREDUCE-1112] - Fix CombineFileInputFormat for hadoop 0.20
• [MAPREDUCE-1147] - Map output records counter missing for map-only jobs in new API
• [MAPREDUCE-1163] - hdfsJniHelper.h: Yahoo! specific paths are encoded
• [MAPREDUCE-1182] - Reducers fail with OutOfMemoryError while copying Map outputs
• [MAPREDUCE-1251] - c++ utils doesn't compile
• [MAPREDUCE-1328] - contrib/index - modify build / ivy files as appropriate

Improvement

• [MAPREDUCE-623] - Resolve javac warnings in mapred

New Feature

• [MAPREDUCE-1145] - Multiple Outputs doesn't work with new API in 0.20 branch
• [MAPREDUCE-1170] - MultipleInputs doesn't work with new API in 0.20 branch

Changes Since Hadoop 0.20.0

Common

Sub-task

• [HADOOP-6213] - Remove commons dependency on commons-cli2

Bug

• [HADOOP-4626] - API link in forrest doc should point to the same version of hadoop.
• [HADOOP-4674] - hadoop fs -help should list detailed help info for the following commands: test, text, tail, stat & touchz
• [HADOOP-4856] - Document JobInitializationPoller configuration in capacity scheduler forrest documentation.
• [HADOOP-4931] - Document TaskTracker's memory management functionality and CapacityScheduler's memory based scheduling.
• [HADOOP-5210] - Reduce Task Progress shows > 100% when the total size of map outputs (for a single reducer) is high
• [HADOOP-5213] - BZip2CompressionOutputStream NullPointerException
• [HADOOP-5349] - When the size required for a path is -1, LocalDirAllocator.getLocalPathForWrite fails with a DiskCheckerException when the disk it selects is bad.
• [HADOOP-5533] - Recovery duration shown on the jobtracker webpage is inaccurate
• [HADOOP-5539] - o.a.h.mapred.Merger not maintaining map out compression on intermediate files
• [HADOOP-5636] - Job is left in Running state after a killJob
• [HADOOP-5641] - Possible NPE in CapacityScheduler's MemoryMatcher
• [HADOOP-5646] - TestQueueCapacities is failing Hudson tests for the last few builds
• [HADOOP-5648] - Not able to generate gridmix.jar on already compiled version of hadoop
• [HADOOP-5654] - TestReplicationPolicy.<init> fails on java.net.BindException
• [HADOOP-5655] - TestMRServerPorts fails on java.net.BindException
• [HADOOP-5688] - HftpFileSystem.getChecksum(..) does not work for the paths with scheme and authority
• [HADOOP-5691] - org.apache.hadoop.mapreduce.Reducer should not be abstract.
• [HADOOP-5711] - Change Namenode file close log to info
• [HADOOP-5718] - Capacity Scheduler should not check for presence of default queue while starting up.
• [HADOOP-5719] - Jobs failed during job initalization are never removed from Capacity Schedulers waiting list
• [HADOOP-5736] - Update CapacityScheduler documentation to reflect latest changes
• [HADOOP-5746] - Errors encountered in MROutputThread after the last map/reduce call can go undetected
• [HADOOP-5796] - DFS Write pipeline does not detect defective datanode correctly in some cases (HADOOP-3339)
• [HADOOP-5828] - Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster
• [HADOOP-5850] - map/reduce doesn't run jobs with 0 maps
• [HADOOP-5863] - mapred metrics shows negative count of waiting maps and reduces
• [HADOOP-5869] - TestQueueCapacitisues.apache.org/jjira/browse/HADOOP-OP-6017] - NameNode and SecondaryNameNode fail to restart because of abnormal filenames.
• [HADOOP-6097] - Multiple bugs w/ Hadoop archives
• [HADOOP-6139] - Incomplete help message is displayed for rm and rmr options.
• [HADOOP-6141] - hadoop 0.20 branch "test-patch" is broken
• [HADOOP-6145] - No error message for deleting non-existant file or directory.
• [HADOOP-6215] - fix GenericOptionParser to deal with -D with '=' in the value

Improvement

• [HADOOP-5726] - Remove pre-emption from the capacity scheduler code base

New Feature

• [HADOOP-3315] - New binary file format
• [HADOOP-5714] - Metric to show number of fs.exists (or number of getFileInfo) calls
• [HADOOP-6080] - Handling of Trash with quota

HDFS

Bug

• [HDFS-26] - HADOOP-5862 for version .20 (Namespace quota exceeded message unclear)
• [HDFS-167] - DFSClient continues to retry indefinitely
• [HDFS-438] - Improve help message for quotas
• [HDFS-442] - dfsthroughput in test.jar throws NPE
• [HDFS-485] - error : too many fetch failures
• [HDFS-495] - Hadoop FSNamesystem startFileInternal() getLease() has bug
• [HDFS-525] - ListPathsServlet.java uses static SimpleDateFormat that has threading issues

Improvement

• [HDFS-504] - HDFS updates the modification time of a file when the file is closed.
• [HDFS-527] - Refactor DFSClient constructors

Map/Reduce

Bug

• [MAPREDUCE-2] - ArrayOutOfIndex error in KeyFieldBasedPartitioner on empty key
• [MAPREDUCE-18] - Under load the shuffle sometimes gets incorrect data
• [MAPREDUCE-40] - Memory management variables need a backwards compatibility option after HADOOP-5881
• [MAPREDUCE-112] - Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API
• [MAPREDUCE-124] - When abortTask of OutputCommitter fails with an Exception for a map-only job, the task is marked as success
• [MAPREDUCE-130] - Delete the jobconf copy from the log directory of the JobTracker when the job is retired
• [MAPREDUCE-179] - setProgress not called for new RecordReaders
• [MAPREDUCE-383] - pipes combiner does not reset properly after a spill
• [MAPREDUCE-421] - mapred pipes might return exit code 0 even when failing
• [MAPREDUCE-430] - Task stuck in cleanup with OutOfMemoryErrors
• [MAPREDUCE-565] - Partitioner does not work with new API
• [MAPREDUCE-657] - CompletedJobStatusStore hardcodes filesystem to hdfs
• [MAPREDUCE-687] - TestMiniMRMapRedDebugScript fails sometimes
• [MAPREDUCE-735] - ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
• [MAPREDUCE-745] - TestRecoveryManager fails sometimes
• [MAPREDUCE-796] - Encountered "ClassCastException" on tasktracker while running wordcount with MultithreadedMapRunner
• [MAPREDUCE-805] - Deadlock in Jobtracker
• [MAPREDUCE-806] - WordCount example does not compile given the current instructions
• [MAPREDUCE-807] - Stray user files in mapred.system.dir with permissions other than 777 can prevent the jobtracker from starting up.
• [MAPREDUCE-818] - org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name doesnt exist.
• [MAPREDUCE-827] - "hadoop job -status <jobid>" command should display job's completion status also.
• [MAPREDUCE-832] - Too many WARN messages about deprecated memorty config variables in JobTacker log
• [MAPREDUCE-834] - When TaskTracker config use old memory management values its memory monitoring is diabled.
• [MAPREDUCE-838] - Task succeeds even when committer.commitTask fails with IOException
• [MAPREDUCE-911] - TestTaskFail fail sometimes
• [MAPREDUCE-924] - TestPipes crashes on trunk

Improvement

• [MAPREDUCE-465] - Deprecate org.apache.hadoop.mapred.lib.MultithreadedMapRunner
• [MAPREDUCE-487] - DBInputFormat support for Oracle
• [MAPREDUCE-767] - to remove mapreduce dependency on commons-cli2

Changes Since Hadoop 0.19.1