(-*- text -*-) Subversion Commandline Client: Test Suite ========================================== The cmdline client test suite doesn't use the C-level testing framework, but is structured similarly. Instead of testing library APIs, it drives the client just like a user would, examining the output and the on-disk results (i.e., the working copy) carefully as it goes. In other words, this is "black box" testing of the command-line client. It has no access to code internals; it never looks inside the .svn/ directory; it only performs actions that a human user would do. These tests require Python 2.7 or later. [ For more general information on Subversion's testing system, please read the README in subversion/tests/. ] How To Run The Tests ==================== To run a test script over ra_local, invoke it from THIS DIRECTORY. $ cd subversion/tests/cmdline/ Invoke the script with no arguments to run all the tests in that script: $ ./basic_tests.py Invoke with one or more numeric arguments to run those particular tests: $ ./basic_tests.py 7 13 17 Invoke with one or more function names to run those particular tests: $ ./basic_tests.py basic_mkdir_wc_with_parents basic_switch basic_import And invoke with the "--list" option to list information about some or all tests available in that script: $ ./basic_tests.py --list 2 3 4 $ ./basic_tests.py --list Note: if you are building Subversion in a directory other than the source directory (q.v. INSTALL), you will have to invoke the tests from within the build directory: $ cd obj/subversion/tests/cmdline $ ../../../../svn/subversion/tests/cmdline/basic_tests.py Running over mod_dav_svn ------------------------ Running a script over mod_dav_svn is basically the same, but you have to set up httpd 2.0 first (on the same machine, since the tests create repositories on the fly), and pass a URL argument to the test scripts. Assuming you have httpd 2.0 installed in /usr/local/apache2, just add two Location directives to /usr/local/apache2/conf/httpd.conf, with paths adjusted appropriately: DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/repositories AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user DAV svn SVNPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp/repos AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On # This may seem unnecessary but granting access to everyone here is necessary # to exercise a bug with httpd 2.3.x+. The "Require all granted" syntax is # new to 2.3.x+ which we can detect with the mod_authz_core.c module # signature. Use the "Allow from all" syntax with older versions for symmetry. Require all granted Allow from all DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user Satisfy Any DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user AuthzSVNNoAuthWhenAnonymousAllowed On DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user AuthzSVNAnonymous Off DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user AuthzForceUsernameCase Lower DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users Require valid-user AuthzForceUsernameCase Lower DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users AuthGroupFile /usr/local/apache2/conf/groups Require group random AuthzSVNAuthoritative Off DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users AuthzSendForbiddenOnFailure On Satisfy All Require valid-user Require expr req('ALLOW') == '1' DAV svn SVNParentPath /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/local_tmp AuthzSVNAccessFile /home/yourusernamehere/projects/svn/subversion/tests/cmdline/svn-test-work/authz SVNListParentPath On AuthType Basic AuthName "Subversion Repository" AuthUserFile /usr/local/apache2/conf/users AuthzSendForbiddenOnFailure On Satisfy All Require valid-user Require expr req('ALLOW') == '1' RedirectMatch permanent ^/svn-test-work/repositories/REDIRECT-PERM-(.*)$ /svn-test-work/repositories/$1 RedirectMatch ^/svn-test-work/repositories/REDIRECT-TEMP-(.*)$ /svn-test-work/repositories/$1 Httpd should be running on port 80. You may also need to ensure that it's running as you, so it has read/write access to the repositories that are probably living in your Subversion working copy. To do this, set the User and Group directives in httpd.conf, something like this: User yourusernamehere Group users You need to run the tests over mod_dav_svn with authentication enabled, so just drop the following 2-line snippet into the /usr/local/apache2/conf/users file [1]: ---------------------------- jrandom:xCGl35kV9oWCY jconstant:xCGl35kV9oWCY JRANDOM:xCGl35kV9oWCY JCONSTANT:xCGl35kV9oWCY ---------------------------- and these lines into the /usr/local/apache/conf/groups file: ---------------------------- random: jrandom constant: jconstant ---------------------------- Now, (re)start Apache and run the tests over mod_dav_svn. You can run a test script over DAV: $ ./basic_tests.py --url http://localhost $ ./basic_tests.py --url http://localhost 3 or $ ./basic_tests.py --url=http://localhost $ ./basic_tests.py --url=http://localhost 3 If you run httpd on a port other than 80, you can specify the port in the URL: "http://localhost:15835" for example. To run all tests over DAV, pass BASE_URL when running 'make check' from the top of the build dir: $ make check BASE_URL=http://localhost BASE_URL=URL can also be used when running individual tests: $ ./basic_tests.py BASE_URL=http://localhost $ ./basic_tests.py BASE_URL=http://localhost 3 Note [1]: It would be quite too much to expect those password entries to work on Windows... Apache httpd on Windows doesn't understand crypted passwords, but it does understand MD5-hashed passwords. The correct password entries for Windows are: ---------------------------- jrandom:$apr1$3p1.....$FQW6RceW5QhJ2blWDQgKn0 jconstant:$apr1$jp1.....$Usrqji1c9H6AbOxOGAzzb0 JRANDOM:$apr1$3p1.....$FQW6RceW5QhJ2blWDQgKn0 JCONSTANT:$apr1$jp1.....$Usrqji1c9H6AbOxOGAzzb0 ---------------------------- As a shorthand to all of the above, ./davautocheck.sh will generate an Apache configuration listening on a random port on localhost and run some tests. Without arguments, or when invoking 'make davautocheck' on the top-level Makefile, it will run all tests. With arguments, it will run just one suite or just one test: $ ./davautocheck.sh $ ./davautocheck.sh basic $ ./davautocheck.sh basic 15 With '--no-tests' argument, it will start httpd but not run any tests. This is useful for manual testing --- create repositories in ./svn-test-work/repositories/<$repo> and they will be accessible at /svn-test-work/repositories/<$repo>. You can also run individual tests by passing the --url option to them, as described above. davautocheck.sh also respects some environment variables; see the comments at the top of the script for details. Running over ra_svn ------------------- It's also easy to run the tests against a local svnserve: $ subversion/svnserve/svnserve -d -r `pwd`/subversion/tests/cmdline $ make check BASE_URL=svn://localhost or, to run individual tests, $ ./basic_tests.py --url=svn://localhost 3 To enable Cyrus SASL on the server side you should either set the ENABLE_SASL variable when calling make: $ make check BASE_URL=svn://localhost ENABLE_SASL=true or if you're running an individual test, $ ./basic_tests.py --url=svn://localhost --enable-sasl 3 Note that to do this you'll have to have a svn.conf file in your SASL lib dir (i.e. something like /usr/lib/sasl2/svn.conf), it should contain something like: pwcheck_method: auxprop mech_list: CRAM-MD5 And then you'll have to add the users jrandom and jconstant to your SASL password db, $ saslpasswd2 -c -u svntest jrandom $ saslpasswd2 -c -u svntest jconstant As usual, both users should use the password 'rayjandom'. To enable DUMP_LOAD_CROSS_CHECK to work a third user is required, $ saslpasswd2 -c -u svntest __dumpster__ with password '__loadster__'. The user running the tests will need read access to the sasl database and on some systems this can be arranged by adding the user to the sasl group. There are 'make svnserveautocheck' and ./svnserveautocheck.sh commands, analogous to davautocheck.sh documented above. Running tests in a RAM disk -------------------------- Test execution can be dramatically sped up by keeping Subversion test data on a RAM disk. On a Linux system, you can mount a RAM disk on the fly with the command: mount -t tmpfs tmpfs /path/to/src/subversion/tests/cmdline/svn-test-work \ -o uid=$USER,mode=770,size=32m Or, for a more permanent solution, add lines like the following in your /etc/fstab file: tmpfs /path/to/src/svn/subversion/tests/cmdline/svn-test-work tmpfs defaults,user,noauto,exec,size=32m The minimum required size for testing ramdisk is approximately 700MB. However, flagging your test targets for cleanup dramatically reduces the space requirements (as shown in the example configuration above), and thus your memory usage. Cleanup means more I/O, but since test data is in-memory, there will be no performance degradation. Example: make check CLEANUP=true See http://svn.haxx.se/dev/archive-2003-02/0068.shtml for the original authoritative discussion on use of RAM disks. Directory Contents ================== *.py The tests themselves. svntest/ Python package, provides test suite framework /main.py: Global vars, utility routines; exports run_tests(), the main test routine. /tree.py: Infrastructure for SVNTreeNode class. - tree constructors, tree comparison routines. - routines to parse subcommand output into specific kinds of trees. - routines to parse a working copy and entries files into specific kinds of trees. /wc.py: Functions for interacting with a working copy, and converting to/from trees. /actions.py: Main API for driving subversion client and using trees to verify results. /verify.py: Verifies output from Subversion. /testcase.py: Control of test case execution - contains decorators for expected failures and conditionally executed tests. /sandbox.py: Tools for manipulating a test's working area ("a sandbox"), those are handy for most simple actions a test might want to perform on a wc. /objects.py: Objects that keep track of state during a test. (not directly used by the test scripts.) /mergetrees.py: Routines that create merge scenarios. /factory.py: Automatically generate a (near-)complete new cmdline test from a series of shell commands. /error.py: Error codes as constants, for convenience. (auto-generated by tools/dev/gen-py-error.py) What the Python Tests are Doing =============================== I. Theory A. Types of Verification The point of this test system is that it's *automated*: that is, each test can algorithmically verify the results and indicate "PASS" or "FAIL". We've identified two broad classes of verification: 1. Verifying svn subcommand output. Most important subcommands (co, up, ci, im, st) print results to stdout as a list of paths. Even though the paths may be printed out in an unpredictable order, we still want to make sure this list is exactly the *set* of lines we expect to get. 2. Verifying the working copy itself. Every time a subcommand could potentially change something on disk, we need to inspect the working copy. Specifically, this means we need to make sure the working copy has exactly the tree-structure we expect, and each file has exactly the contents and properties we expect. II. Practice: Trees Sam TH proposed and began work on a solution whereby all important, inspectable information is parsed into a general, in-memory tree representation. By comparing actual vs. expected tree structures, we get automated verification. A. Tree node structure Each "tree node" in a tree has these fields: - name : the name of the node - children: list of child nodes (if the node is a dir) - contents: textual contents (if the node is a file) - properties: a hash to hold subversion props - atts: a hash of meta-information about tree nodes themselves B. Parsing subcommand output into a tree Special parsers examine lines printed by subcommands, and convert them into a tree of tree-nodes. The 'contents' and 'properties' fields are empty; but prepending on the subcommand, specific attributes in the 'atts' field are set in tree-nodes: - svn co/up: a 'status' attribute is set to a two-character value from the set (A, D, G, U, C, _, ' ') or a 'verb' attribute is set to ('Restored') - svn status: a 'status' attribute (as above), plus 'wc_rev' and 'repos_rev' attributes to hold the wc and repos revision numbers. - svn ci/im: a 'verb' attribute is set to one of (Adding, Sending, Deleting) C. Parsing a working copy into a tree We also have a routines that walks a regular working copy and returns a tree representing disk contents and props. In this case the 'atts' hash in each node is empty, but the 'contents' and 'props' fields are filled in. How to Write New Tests ====================== If you'd like to write a new python test, first decide which file it might fit into; test scripts each contain collections of tests grouped by rough categories. (Is it testing a new subcommand? New enhancement? Tricky use-case? Regression test?) Next, read the long documentation comment at the top of svntest/tree.py. It will explain the general API that most tests use. Finally, try copying-and-pasting a simple test and then edit from there. Don't forget to add your test to the 'test_list' variable at the bottom of the file. To avoid renumbering of existing tests, you should add new tests to the end of the list. Testing Compatibility With Previous Release =========================================== You can run the Python test suite against older installed versions of the Subversion servers. This mail fragment introduces the ability: Message-ID: <1ea387f60804091828q48c9d18ah7bf8d89ef7d39461@mail.gmail.com> Date: Wed, 9 Apr 2008 18:28:40 -0700 From: "David Glasser" To: "Subversion Developers" Subject: backwards-compatibility testing! I've updated the expectations on trunk so that you can cleanly run the test suite against 1.4.x svnserve or DAV. You do this by adding SERVER_MINOR_VERSION=4 with make check, or --server-minor-version 4 to run_tests.py or a specific Python test. [...] We expect that post-1.5, this support will expand in the obvious ways (allowing "--server-minor-version 5" and SERVER_MINOR_VERSION=5).