Hadoop HDFS Change Log Release 0.21.1 - Unreleased IMPROVEMENTS HDFS-1411. Correct backup node startup command in hdfs user guide. (Ching-Shen Chen via shv) HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects. (todd, nigel) BUG FIXES HDFS-1363. Eliminate second synchronized sections in appendFile(). (shv) HDFS-1413. Fix broken links to HDFS Wiki. (shv) HDFS-1420. Clover build doesn't generate per-test coverage (cos) HDFS-1444. Test related code of build.xml is error-prone and needs to be re-aligned. (cos) HDFS-1343. Instrumented build should be concentrated in one build area (cos) HDFS-1452. ant compile-contrib is broken (cos) HDFS-1474. ant binary-system is broken (cos) HDFS-1292. Allow artifacts to be published to the staging Apache Nexus Maven Repository. (Giridharan Kesavan via tomwhite) HDFS-1206. TestFiHFlush fails intermittently. (cos) HDFS-1548. Fault-injection tests are executed multiple times if invoked with run-test-hdfs-fault-inject target (cos) HDFS-1552. Remove java5 dependencies from build. (cos) HDFS-996. JUnit tests should never depend on anything in conf (cos) HDFS-1612. Update HDFS design documentation for append, quota, symlink, block placement and checkpoint/backup node features. (Joe Crobak via szetszwo) HDFS-1189. Quota counts missed between clear quota and set quota. (John George via szetszwo) HDFS-1665. Balancer misuses dfs.heartbeat.interval as milliseconds. (szetszwo) HDFS-1728. SecondaryNameNode.checkpointSize is in bytes but not in MB. (szetszwo) HDFS-1596. Replace fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations. (Harsh J Chouraria via szetszwo) Release 0.21.0 - 2010-08-13 INCOMPATIBLE CHANGES HDFS-538. Per the contract elucidated in HADOOP-6201, throw FileNotFoundException from FileSystem::listStatus rather than returning null. (Jakob Homan via cdouglas) HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException instead of FileNotFoundException. (Boris Shkolnik via suresh) HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong) HDFS-576. Block report includes under-construction replicas. (shv) HDFS-636. SafeMode counts complete blocks only. (shv) HDFS-644. Lease recovery, concurrency support. (shv) HDFS-570. Get last block length from a data-node when opening a file being written to. (Tsz Wo (Nicholas), SZE via shv) HDFS-657. Remove unused legacy data-node protocol methods. (shv) HDFS-658. Block recovery for primary data-node. (shv) HDFS-660. Remove deprecated methods from InterDatanodeProtocol. (shv) HDFS-512. Block.equals() and compareTo() compare blocks based only on block Ids, ignoring generation stamps. (shv) HDFS-873. Configuration specifies data-node storage directories as URIs. (shv) HDFS-905. Use the new UserGroupInformation from HDFS-6299. (jghoman via omalley) HDFS-984. Persistent delegation tokens. (Jitendra Pandey via shv) HDFS-1016. HDFS side change for HADOOP-6569. This jira changes the error message on the screen when cat a directory or a non-existent file. (hairong) NEW FEATURES HDFS-1134. Large-scale Automated Framework. (cos) HDFS-436. Introduce AspectJ framework for HDFS code and tests. (Konstantin Boudnik via szetszwo) HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas) HDFS-459. Introduce Job History Log Analyzer. (shv) HDFS-461. Tool to analyze file size distribution in HDFS. (shv) HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt blocks/replicas information. (Bill Zeller via szetszwo) HDFS-578. Add support for new FileSystem method for clients to get server defaults. (Kan Zhang via suresh) HDFS-595. umask settings in configuration may now use octal or symbolic instead of decimal. (Jakob Homan via suresh) HADOOP-6234. Updated hadoop-core and test jars to propagate new option dfs.umaskmode in configuration. (Jakob Homan via suresh) HDFS-235. Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. (Bill Zeller via suresh) HDFS-385. Add support for an experimental API that allows a module external to HDFS to specify how HDFS blocks should be placed. (dhruba) HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext file system application interface. (Sanjay Radia via suresh). HDFS-567. Add block forensics contrib tool to print history of corrupt and missing blocks from the HDFS logs. (Bill Zeller, Jitendra Nath Pandey via suresh). HDFS-610. Support o.a.h.fs.FileContext. (Sanjay Radia via szetszwo) HDFS-536. Support hflush at DFSClient. (hairong) HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica states while writing. (shv) HDFS-565. Introduce block committing logic during new block allocation and file close. (shv) HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the support of dfs writes/hflush. It also updates a replica's bytes received, bytes on disk, and bytes acked after receiving a packet. (hairong) HDFS-585. Datanode should serve up to visible length of a replica for read requests. (szetszwo) HDFS-604. Block report processing for append. (shv) HDFS-619. Support replica recovery initialization in datanode for the new append design. (szetszwo) HDFS-592. Allow clients to fetch a new generation stamp from NameNode for pipeline recovery. (hairong) HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup for append. (hairong) HDFS-627. Support replica update in data-node. (Tsz Wo (Nicholas), SZE and Hairong Kuang via shv) HDFS-642. Support pipeline close and close error recovery. (hairong) HDFS-631. Rename configuration keys towards API standardization and backward compatibility. (Jitendra Nath Pandey via suresh) HDFS-669. Add unit tests framework (Mockito) (cos, Eli Collins) HDFS-731. Support new Syncable interface in HDFS. (hairong) HDFS-702. Add HDFS implementation of AbstractFileSystem. (Sanjay Radio via suresh) HDFS-758. Add decommissioning status page to Namenode Web UI. (Jitendra Nath Pandey via suresh) HDFS-814. Add an api to get the visible length of a DFSDataInputStream. (szetszwo) HDFS-654. Add support new atomic rename functionality in HDFS for supporting rename in FileContext. (suresh) HDFS-222. Support for concatenating of files into a single file without copying. (Boris Shkolnik via hairong) HDFS-933. Adds Delegation token based authentication in the NameNode. (Kan Zhang via ddas) HDFS-935. Adds a real user component in Delegation token. (Jitendra Nath Pandey via ddas) HDFS-245. Adds a symlink implementation to HDFS. This complements the new symlink feature added in HADOOP-6421 (Eli Collins via Sanjay Radia) HDFS-1009. Support Kerberos authorization in HDFSProxy. (Srikanth Sundarrajan via szetszwo) HDFS-1091. Implement listStatus that returns an iterator of FileStatus. (hairong) IMPROVEMENTS HDFS-1199. Extract a subset of tests for smoke (DOA) validation. (cos) HDFS-381. Remove blocks from DataNode maps when corresponding file is deleted. (Suresh Srinivas via rangadi) HDFS-377. Separate codes which implement DataTransferProtocol. (szetszwo) HDFS-396. NameNode image and edits directories are specified as URIs. (Luca Telloli via rangadi) HDFS-444. Allow to change probability levels dynamically in the fault injection framework. (Konstantin Boudnik via szetszwo) HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv) HADOOP-6106. Updated hadoop-core and test jars from hudson trunk build #12. (Giridharan Kesavan) HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode. (Jitendra Nath Pandey via szetszwo) HDFS-278. HDFS Outputstream close does not hang forever. (dhruba) HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode. (Jitendra Nath Pandey via szetszwo) HDFS-475. Add new ant targets for fault injection jars and tests. (Konstantin Boudnik via szetszwo) HDFS-458. Create a new ant target, run-commit-test. (Jakob Homan via szetszwo) HDFS-493. Change build.xml so that the fault-injected tests are executed only by the run-test-*-fault-inject targets. (Konstantin Boudnik via szetszwo) HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv) HADOOP-6160. Fix releaseaudit target to run on specific directories. (gkesavan) HDFS-501. Use enum to define the constants in DataTransferProtocol. (szetszwo) HDFS-508. Factor out BlockInfo from BlocksMap. (shv) HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo. (Jakob Homan & Hairong Kuang via shv) HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol. (Jakob Homan via shv) HDFS-514. Change DFSClient.namenode from public to private. (Bill Zeller via szetszwo) HDFS-496. Use PureJavaCrc32 in HDFS. (Todd Lipcon via szetszwo) HDFS-511. Remove redundant block searches in BlockManager. (shv) HDFS-504. Update the modification time of a file when the file is closed. (Chun Zhang via dhruba) HDFS-498. Add development guide and documentation for the fault injection framework. (Konstantin Boudnik via szetszwo) HDFS-524. Further DataTransferProtocol code refactoring. (szetszwo) HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches in BlockManager. (shv) HDFS-530. Refactor TestFileAppend* to remove code duplication. (Konstantin Boudnik via szetszwo) HDFS-451. Add fault injection tests for DataTransferProtocol. (szetszwo) HDFS-409. Add more access token tests. (Kan Zhang via szetszwo) HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv) HDFS-457. Do not shutdown datanode if some, but not all, volumes fail. (Boris Shkolnik via szetszwo) HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong) HDFS-539. Refactor fault injeciton pipeline test util for future reuse. (Konstantin Boudnik via szetszwo) HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new tests. (szetszwo) HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..). (szetszwo) HDFS-581. Introduce an iterator over blocks in the block report array.(shv) HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which allows an execution of non-FI tests in FI-enable environment. (Konstantin Boudnik via szetszwo) HDFS-173. Namenode will not block until a large directory deletion completes. It allows other operations when the deletion is in progress. (suresh) HDFS-551. Create new functional test for a block report. (Konstantin Boudnik via hairong) HDFS-288. Redundant computation in hashCode() implementation. (szetszwo via tomwhite) HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible. (Brian Bockelman via tomwhite) HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design document. (Zhiyong Zhang via cdouglas) HDFS-617. Support non-recursive create(). (Kan Zhang via szetszwo) HDFS-618. Support non-recursive mkdir(). (Kan Zhang via szetszwo) HDFS-574. Split the documentation between the subprojects. (Corinne Chandel via omalley) HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite) HDFS-641. Move all of the components that depend on map/reduce to map/reduce. (omalley) HDFS-509. Redesign DataNode volumeMap to include all types of Replicas. (hairong) HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from un-closed file. (szetszwo) HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary, writeToRBW, ad append. (hairong) HDFS-603. Add a new interface, Replica, which is going to replace the use of Block in datanode. (szetszwo) HDFS-589. Change block write protocol to support pipeline recovery. (hairong) HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv) HDFS-648. Change some methods in AppendTestUtil to public. (Konstantin Boudnik via szetszwo) HDFS-662. Unnecessary info message from DFSClient. (hairong) HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik via szetszwo) HDFS-688. Add configuration resources to DFSAdmin. (shv) HDFS-29. Validate the consistency of the lengths of replica and its file in replica recovery. (szetszwo) HDFS-680. Add new access method to a copy of a block's replica. (shv) HDFS-704. Unify build property names to facilitate cross-projects modifications (cos) HDFS-705. Create an adapter to access some of package-private methods of DataNode from tests (cos) HDFS-710. Add actions with constraints to the pipeline fault injection tests and change SleepAction to support uniform random sleeping over an interval. (szetszwo) HDFS-713. Need to properly check the type of the test class from an aspect (cos) HDFS-716. Define a pointcut for pipeline close and add a few fault injection tests to simulate out of memory problem. (szetszwo) HDFS-719. Add 6 fault injection tests for pipeline close to simulate slow datanodes and disk errors. (szetszwo) HDFS-616. Create functional tests for new design of the block report. (cos) HDFS-584. Fail the fault-inject build if any advices are mis-bound. (cos) HDFS-730. Add 4 fault injection tests to simulate non-responsive datanode and out-of-memory problem for pipeline close ack. (szetszwo) HDFS-728. Create a comprehensive functional test for append. (hairong) HDFS-736. commitBlockSynchronization() updates block GS and length in-place. (shv) HADOOP-5107. Use Maven ant tasks to publish the subproject jars. (Giridharan Kesavan via omalley) HDFS-521. Create new tests for pipeline (cos) HDFS-764. Places the Block Access token implementation in hdfs project. (Kan Zhang via ddas) HDFS-787. Upgrade some libraries to be consistent with common and mapreduce. (omalley) HDFS-519. Create new tests for lease recovery (cos) HDFS-804. New unit tests for concurrent lease recovery (cos) HDFS-813. Enable the append test in TestReadWhileWriting. (szetszwo) HDFS-145. Cleanup inconsistent block length handling code in FSNameSystem#addStoredBlock. (hairong) HDFS-127. Reset failure count in DFSClient for each block acquiring operation. (Igor Bolotin via szetszwo) HDFS-520. Create new tests for block recovery. (hairong) HDFS-1067. Create block recovery tests that handle errors. (hairong) HDFS-1107. Turn on append by default. (shv) HDFS-968. Use StringBuilder instead of StringBuffer for better performance. (Kay Kay via suresh) HDFS-703. Replace current fault injection implementation with one from (cos) HDFS-754. Reduce ivy console output to observable level (cos) HDFS-832. HDFS side of HADOOP-6222. (cos) HDFS-840. Change tests to use FileContext test helper introduced in HADOOP-6394. (Jitendra Nath Pandey via suresh) HDFS-685. Use the user-to-groups mapping service in the NameNode. (boryas, acmurthy) HDFS-755. Read multiple checksum chunks at once in DFSInputStream. (Todd Lipcon via tomwhite) HDFS-786. Implement getContentSummary in HftpFileSystem. (Tsz Wo (Nicholas), SZE via cdouglas) HDFS-587. Add support for specifying queue name in mapreduce tests. (Erik Steffl via suresh) HDFS-902 Move contrib/raid to MapReduce. (Eli Collins via omalley) HDFS-800. The last block of a file under construction may change to the COMPLETE state in response to getAdditionalBlock or completeFileInternal. (hairong) HDFS-899. Delegation Token Implementation and corresponding changes in Namenode and DFS Api to issue, renew and cancel delegation tokens. (jnp via boryas) HDFS-844. Log the filename when file locking fails. (tomwhite) HDFS-914. Refactor DFSOutputStream and DFSInputStream out of DFSClient. (Todd Lipcon via tomwhite) HDFS-949. Move DelegationToken into Common so that it can be used by MapReduce. (omalley) HDFS-930. Better error message for DATA_TRANSFER_VERSION mismatched. (Kay Kay via szetszwo) HDFS-986. Delegation token renewing and cancelling should provide meaningful exceptions when there are failures instead of returning false. (omalley) HADOOP-6579. Upgrade the commons-codec library to 1.4. (omalley) HDFS-991. Allow authentication to the web ui via a delegation token. (omalley) HDFS-994. Allow fetching of delegation token from NameNode for hftp. (Jakob Homan via acmurthy) HDFS-998. Quote blocks streamed through jsps. (cdouglas) HDFS-729. NameNode API to list files that have missing blocks. (Rodrigo Schmidt via dhruba) HDFS-850. The WebUI display more details about namenode memory usage. (Dmytro Molkov via dhruba) HDFS-826. The DFSOutputStream has a API that returns the number of active datanode(s) in the current pipeline. (dhruba) HDFS-985. HDFS should issue multiple RPCs for listing a large directory. (hairong) HDFS-1043. NNThroughputBenchmark modifications to support benchmarking of server-side user group resolution. (shv) HDFS-892. Optionally use Avro reflection for Namenode RPC. This is not a complete implementation yet, but rather a starting point. (cutting) HDFS-854. Datanode should scan devices in parallel to generate block report. (Dmytro Molkov via jhoman) HDFS-1032. fsck has an option to list corrupt files. (Andre Oriai via dhruba) HDFS-1024. SecondaryNameNode verifies size of fsimage and edits file. (Dmytro Molkov via dhruba) HDFS-1011. hdfsproxy: Improve log messages by restoring the previous thread name. (Srikanth Sundarrajan via szetszwo) HDFS-997. Allow datanode storage directory permissions to be configurable. (Luke Lu via cdouglas) HDFS-1012. hdfsproxy: Support for fully qualified HDFS path in addition to simple unqualified path. (Srikanth Sundarrajan via szetszwo) HDFS-933. Namenode should issue a delegation token only for kerberos authenticated clients.(jnp via boryas) HDFS-1087. Modify audit log to use a StringBuilder rather than a Formatter. (cdouglas) HDFS-1083. Update TestHDFSCLI not to expect exception class name in error messages. (suresh) HDFS-1099. Add test for umask backward compatibility. (suresh) HDFS-1092. Use logging rather than System.err in MiniDFSCluster. (Kay Kay via jghoman) HDFS-1047. Install/deploy source jars to Maven repo. (Patrick Angeles via jghoman) HDFS-666. Unit test for FsShell -text. (cdouglas via jghoman) HDFS-1054. Remove unnecessary sleep after failure in nextBlockOutputStream. (Todd Lipcon via jghoman) HDFS-921. Convert TestDFSClientRetries::testNotYetReplicatedErrors to Mockito. (jghoman) HDFS-1100. Override unwrapException in TestFcHdfsSymlink to test symlink API conformance. (Eli Collins via suresh). HDFS-1089. Remove uses of FileContext#isFile, isDirectory, and exists. (Eli Collins via hairong) HDFS-1126. Change HDFS to depend on Hadoop 'common' artifacts instead of 'core'. (tomwhite) HDFS-995. Replace usage of FileStatus#isDir(). (Eli Collins via tomwhite) HDFS-1161. Make DN minimum valid volumes configurable. (Eli Collins via tomwhite) HDFS-1181. Move configuration and script files post split. (tomwhite) HDFS-1170. Add more assertions to TestLargeDirectoryDelete. (Steve Loughran via tomwhite) HDFS-752. Add interfaces classification to to HDFS source code. (suresh) HDFS-1174. New properties for suspend and resume process. (Vinay Thota via cos) HDFS-1277. [Herriot] New property for multi user list. (Vinay Thota via cos) HDFS-806. Add new unit tests to the 10-mins 'run-commit-test' target (cos) OPTIMIZATIONS HDFS-946. NameNode should not return full path name when lisitng a diretory or getting the status of a file. (hairong) BUG FIXES HDFS-76. Better error message to users when commands fail because of lack of quota. Allow quota to be set even if the limit is lower than current consumption. (Boris Shkolnik via rangadi) HADOOP-4687. HDFS is split from Hadoop Core. It is a subproject under Hadoop (Owen O'Malley) HADOOP-6096. Fix Eclipse project and classpath files following project split. (tomwhite) HDFS-195. Handle expired tokens when write pipeline is reestablished. (Kan Zhang via rangadi) HDFS-181. Validate src path in FSNamesystem.getFileInfo(..). (Todd Lipcon via szetszwo) HDFS-441. Remove TestFTPFileSystem. (szetszwo) HDFS-440. Fix javadoc broken links in DFSClient. (szetszwo) HDFS-480. Fix a typo in the jar name in build.xml. (Konstantin Shvachko via gkesavan) HDFS-438. Check for NULL before invoking GenericArgumentParser in DataNode. (Raghu Angadi) HDFS-415. BlockReceiver hangs in case of certain runtime exceptions. (Konstantin Boudnik via rangadi) HDFS-462. loadFSImage should close edits file. (Jakob Homan via shv) HDFS-489. Update TestHDFSCLI for the -skipTrash option in rm. (Jakob Homan via szetszwo) HDFS-445. pread() does not pick up changes to block locations. (Kan Zhang via rangadi) HDFS-463. CreateEditLog utility broken after HDFS-396 (URI for FSImage). (Suresh Srinivas via rangadi) HDFS-484. Fix bin-package and package target to package jar files. (gkesavan) HDFS-490. Eliminate the deprecated warnings introduced by H-5438. (He Yongqiang via szetszwo) HDFS-119. Fix a bug in logSync(), which causes NameNode block forever. (Suresh Srinivas via shv) HDFS-534. Include avro in ivy. (szetszwo) HDFS-532. Allow applications to know that a read request failed because block is missing. (dhruba) HDFS-561. Fix write pipeline READ_TIMEOUT in DataTransferProtocol. (Kan Zhang via szetszwo) HDFS-553. BlockSender reports wrong failed position in ChecksumException. (hairong) HDFS-568. Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. (Amareshwari Sriramadasu via szetszwo) HDFS-15. All replicas end up on 1 rack. (Jitendra Nath Pandey via hairong) HDFS-586. TestBlocksWithNotEnoughRacks sometimes fails. (Jitendra Nath Pandey via hairong) HADOOP-6243. Fixed a NullPointerException in handling deprecated keys. (Sreekanth Ramakrishnan via yhemanth) HDFS-605. Do not run fault injection tests in the run-test-hdfs-with-mr target. (Konstantin Boudnik via szetszwo) HDFS-606. Fix ConcurrentModificationException in invalidateCorruptReplicas() (shv) HDFS-601. TestBlockReport obtains data directories directly from MiniHDFSCluster. (Konstantin Boudnik via shv) HDFS-614. TestDatanodeBlockScanner obtains data directories directly from MiniHDFSCluster. (shv) HDFS-612. Remove the use of org.mortbay.log.Log in FSDataset. (szetszwo) HDFS-622. checkMinReplication should count live nodes only. (shv) HDFS-629. Remove ReplicationTargetChooser.java along with fixing import warnings generated by Eclipse. (dhruba) HDFS-637. DataNode sends a Success ack when block write fails. (hairong) HDFS-640. Fixed TestHDFSFileContextMainOperations.java build failure. (suresh) HDFS-547. TestHDFSFileSystemContract#testOutputStreamClosedTwice sometimes fails with CloseByInterruptException. (hairong) HDFS-588. Fix TestFiDataTransferProtocol and TestAppend2 failures. (shv) HDFS-550. DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas. (hairong) HDFS-659. If the the last block is not complete, update its length with one of its replica's length stored in datanode. (szetszwo) HDFS-649. Check null pointers for DataTransferTest. (Konstantin Boudnik via szetszwo) HDFS-661. DataNode upgrade fails on non-existant current directory. (hairong) HDFS-597. Mofication introduced by HDFS-537 breakes an advice binding in FSDatasetAspects. (Konstantin Boudnik via szetszwo) HDFS-665. TestFileAppend2 sometimes hangs. (hairong) HDFS-676. Fix NPE in FSDataset.updateReplicaUnderRecovery() (shv) HDFS-673. BlockReceiver#PacketResponder should not remove a packet from the ack queue before its ack is sent. (hairong) HDFS-682. Fix bugs in TestBlockUnderConstruction. (szetszwo) HDFS-668. TestFileAppend3#TC7 sometimes hangs. (hairong) HDFS-679. Appending to a partial chunk incorrectly assumes the first packet fills up the partial chunk. (hairong) HDFS-722. Fix callCreateBlockWriteStream pointcut in FSDatasetAspects. (szetszwo) HDFS-690. TestAppend2#testComplexAppend failed on "Too many open files". (hairong) HDFS-725. Support the build error fix for HADOOP-6327. (Sanjay Radia via szetszwo) HDFS-625. Fix NullPointerException thrown from ListPathServlet. (suresh) HDFS-735. TestReadWhileWriting has wrong line termination symbols (cos) HDFS-691. Fix an overflow error in DFSClient.DFSInputStream.available(). (szetszwo) HDFS-733. TestBlockReport fails intermittently. (cos) HDFS-774. Intermittent race condition in TestFiPipelines (cos) HDFS-741. TestHFlush test doesn't seek() past previously written part of the file (cos, szetszwo) HDFS-706. Intermittent failures in TestFiHFlush (cos) HDFS-646. Fix test-patch failure by adding test-contrib ant target. (gkesavan) HDFS-791. Build is broken after HDFS-787 patch has been applied (cos) HDFS-792. TestHDFSCLI is failing. (Todd Lipcon via cos) HDFS-781. Namenode metrics PendingDeletionBlocks is not decremented. (Suresh) HDFS-192. Fix TestBackupNode failures. (shv) HDFS-797. TestHDFSCLI much slower after HDFS-265 merge. (Todd Lipcon via cos) HDFS-824. Stop lease checker in TestReadWhileWriting. (szetszwo) HDFS-823. CheckPointer should use addInternalServlet for image-fetching servlet (jghoman) HDFS-456. Fix URI generation for windows file paths. (shv) HDFS-812. FSNamesystem#internalReleaseLease throws NullPointerException on a single-block file's lease recovery. (cos) HDFS-724. Pipeline hangs if one of the block receiver is not responsive. (hairong) HDFS-564. Adding pipeline tests 17-35. (hairong) HDFS-849. TestFiDataTransferProtocol2#pipeline_Fi_18 sometimes fails. (hairong) HDFS-762. Balancer causes Null Pointer Exception. (Cristian Ivascu via dhruba) HDFS-868. Fix link to Hadoop Upgrade Wiki. (Chris A. Mattmann via shv) HDFS-880. TestNNLeaseRecovery fails on windows (cos, shv) HDFS-699. Primary datanode should compare replicas' on disk lengths. (hairong) HDFS-897. Fix a bug related to generation stamp comparison in ReplicasMap. (suresh) HDFS-793. Data node should receive the whole packet ack message before it constructs and sends its own ack message for the packet. (hairong) HDFS-101. DFS write pipeline: DFSClient sometimes does not detect second datanode failure. (hairong) HDFS-822. Appends to already-finalized blocks can rename across volumes. (hairong) HDFS-1046. Fix Tomcat version in hdfsproxy/build.xml. (Srikanth Sundarrajan via szetszwo) HDFS-1072. Fix TestReadWhileWriting failure. (Erik Steffl via shv) HDFS-132. Fix namenode to not report files deleted metrics for deletions done while replaying edits during startup. (suresh) HDFS-913. Rename fault injection test TestRename.java to TestFiRename.java to include it in tests run by ant target run-test-hdfs-fault-inject. (suresh) HDFS-695. RaidNode should read in configuration from hdfs-site.xml. (dhruba) HDFS-726. Eclipse .classpath template has outdated jar files and is missing some new ones. (cos) HDFS-750. Fix build failure due to TestRename. (suresh) HDFS-712. Move libhdfs from mapreduce subproject to hdfs subproject. (Eli Collins via dhruba) HDFS-757. Enable Unit test for HDFS Raid. (dhruba) HDFS-611. Prevent DataNode heartbeat times from increasing even when the DataNode has many blocks to delete. (Zheng Shao via dhruba) HDFS-751. Fix TestCrcCorruption to pick up the correct datablocks to corrupt. (dhruba) HDFS-763. Fix slightly misleading report from DataBlockScanner about corrupted scans. (dhruba) HDFS-727. bug setting block size hdfsOpenFile (Eli Collins via cos) HDFS-756. libhdfs unit tests do not run. (Eli Collins via cos) HDFS-783. libhdfs tests brakes code coverage runs with Clover (cos) HDFS-785. Add Apache license to several namenode unit tests. (Ravi Phulari via jghoman) HDFS-802. Update Eclipse configuration to match changes to Ivy configuration (Edwin Chan via cos) HDFS-423. Unbreak FUSE build and fuse_dfs_wrapper.sh (Eli Collins via cos) HDFS-825. Build fails to pull latest hadoop-core-* artifacts (cos) HDFS-94. The Heap Size printed in the NameNode WebUI is accurate. (Dmytro Molkov via dhruba) HDFS-767. An improved retry policy when the DFSClient is unable to fetch a block from the datanode. (Ning Zhang via dhruba) HDFS-775. FSDataset calls getCapacity() twice. (stevel) HDFS-885. Datanode toString() NPEs on null dnRegistration. (stevel) HDFS-877. Client-driven block verification not functioning. (Todd Lipcon via hairong) HDFS-630. In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block. (Cosmin Lehene via Stack) HDFS-922. Remove unnecessary semicolon added by HDFS-877 that causes problems for Eclipse compilation. (jghoman) HDFS-927 DFSInputStream retries too many times for new block locations (Todd Lipcon via Stack) HDFS-938. Replace calls to UGI.getUserName() with UGI.getShortUserName() (jghoman) HDFS-894. DatanodeID.ipcPort is not updated when existing node re-registers. (Todd Lipcon via tomwhite) HDFS-965. Split TestDelegationToken in to two parts and fix configuration to allow proxy users in the test. (Jitendra Pandey via omalley) HDFS-999. Secondary namenode should login using kerberos if security is configured (boryas) HDFS-856. Hardcoded replication level for new files in fuse-dfs. (Brian Bockelman via tomwhite) HDFS-857. Incorrect type for fuse-dfs capacity can cause "df" to return negative values on 32-bit machines. (Brian Bockelman via tomwhite) HDFS-858. Incorrect return codes for fuse-dfs. (Brian Bockelman via tomwhite) HDFS-859. fuse-dfs utime behavior causes issues with tar. (Brian Bockelman via tomwhite) HDFS-861. fuse-dfs does not support O_RDWR. (Brian Bockelman via tomwhite) HDFS-961. dfs_readdir incorrectly parses paths. (Eli Collins via tomwhite) HDFS-1015. Fix intermittent failure in TestSecurityTokenEditLog. (Jitendra Nath Pandey via suresh) HDFS-939. libhdfs test is broken. (Eli Collins via tomwhite) HDFS-1074. hdfsproxy: Fix bugs in TestProxyUtil. (Srikanth Sundarrajan via szetszwo) HDFS-481. hdfsproxy: Bug Fixes + HdfsProxy to use proxy user to impresonate the real user. (Srikanth Sundarrajan via szetszwo) HDFS-482. Move HsftpFileSystem's ssl.client.do.not.authenticate.server configuration setting to ssl-client.xml. (Srikanth Sundarrajan via szetszwo) HDFS-1010. hdfsproxy: Retrieve groups from UnixUserGroupInformation instead of LdapEntry. (Srikanth Sundarrajan via szetszwo) HDFS-466. hdfs_write infinite loop when dfs fails and cannot write files > 2 GB. (Pete Wyckoff via tomwhite) HDFS-651. HDFS Docs - fix listing of docs in the doc menu. (Corinne Chandel via tomwhite) HDFS-1014. Error in reading delegation tokens from edit logs. (Jitendra Nath Pandey via jhoman) HDFS-1088. Prevent renaming a symbolik link to its target. (Eli Collins via suresh) HDFS-966. NameNode does not recovers lease when it is in safemode. (dhruba) HDFS-833. Datanode shutdown should log problems with Storage.unlockAll() (Steve Loughran via dhruba) HDFS-1101. TestDiskError.testLocalDirs() fails. (cdouglas via jghoman) HDFS-1031. Enhance the webUi to list a few of the corrupted files in HDFS. (Andre Orian via dhruba) HDFS-1078. Create static and dynamic versions of libhdfs. (Sam Rash via dhruba) HDFS-1104. Fsck triggers full GC on NameNode. (hairong) HDFS-1173. Fix references to 0.22 in 0.21 branch. (tomwhite) HDFS-1000. Updates libhdfs to the new API for UGI (ddas) HDFS-609. Create a file with the append flag does not work in HDFS. (tomwhite) HDFS-1255. Fix failing test-libhdfs.sh test. (tomwhite) HDFS-1256. libhdfs is missing from the tarball. (tomwhite) HDFS-1057. Concurrent readers hit ChecksumExceptions if following a writer to very end of file. (sam rash via hairong) HDFS-1212. Harmonize HDFS JAR library versions with Common. (tomwhite) HDFS-1159. clean-cache target removes wrong ivy cache (cos) HDFS-1193. -mvn-system-deploy target is broken which inturn fails the mvn-deploy task leading to unstable mapreduce build (Giridharan Kesavan via cos) HDFS-1299. 'compile-fault-inject' never should be called directly. (cos) HDFS-1311. Running tests with 'testcase' cause triple execution of the same test case (Cos) HDFS-1267. fuse-dfs does not compile. (Devaraj Das via tomwhite) HDFS-1598. Directory listing on hftp:// does not show .*.crc files. (szetszwo) HDFS-1750. ListPathsServlet should not use HdfsFileStatus.getLocalName() to get file name since it may return an empty string. (szetszwo) Release 0.20.3 - 2011-1-5 IMPROVEMENTS BUG FIXES HDFS-955. New implementation of saveNamespace() to avoid loss of edits when name-node fails during saving. (shv) HDFS-1041. DFSClient.getFileChecksum(..) should retry if connection to the first datanode fails. (szetszwo) HDFS-909. Wait until edits syncing is finishes before purging edits. (Todd Lipcon via shv) HDFS-1258. Clearing namespace quota on "/" corrupts fs image. (Aaron T. Myers via szetszwo) HDFS-1406. TestCLI fails on Ubuntu with default /etc/hosts. (cos) Release 0.20.2 - 2009-09-01 IMPROVEMENTS HDFS-737. Add full path name of the file to the block information and summary of total number of files, blocks, live and deadnodes to metasave output. (Jitendra Nath Pandey via suresh) HDFS-919. Create test to validate the BlocksVerified metric (Gary Murry via cos) HDFS-907. Add tests for getBlockLocations and totalLoad metrics. (Ravi Phulari via cos) BUG FIXES HDFS-686. NullPointerException is thrown while merging edit log and image. (hairong) HDFS-677. Rename failure when both source and destination quota exceeds results in deletion of source. (suresh) HDFS-709. Fix TestDFSShell failure due to rename bug introduced by HDFS-677. (suresh) HDFS-579. Fix DfsTask to follow the semantics of 0.19, regarding non-zero return values as failures. (Christian Kunz via cdouglas) HDFS-723. Fix deadlock in DFSClient#DFSOutputStream. (hairong) HDFS-596. Fix memory leak in hdfsFreeFileInfo() for libhdfs. (Zhang Bingjun via dhruba) HDFS-185. Disallow chown, chgrp, chmod, setQuota, and setSpaceQuota when name-node is in safemode. (Ravi Phulari via shv) HDFS-187. Initialize secondary namenode http address in TestStartup. (Todd Lipcon via szetszwo) HDFS-464. Fix memory leaks in libhdfs. (Christian Kunz via suresh) HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (eli) Release 0.20.1 - 2009-09-01 IMPROVEMENTS HDFS-438. Improve help message for space quota command. (Raghu Angadi) BUG FIXES HDFS-167. Fix a bug in DFSClient that caused infinite retries on write. (Bill Zeller via szetszwo) HDFS-527. Remove/deprecate unnecessary DFSClient constructors. (szetszwo) HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread safe. (Suresh Srinivas and cdouglas) HDFS-761. Fix failure to process rename operation from edits log due to quota verification. (suresh)