Hacker's Guide to Subversion

If you are contributing code to the Subversion project, please read this first.

$LastChangedDate$

Table of Contents

Participating in the community

Although Subversion is originally sponsored and hosted by CollabNet (http://www.collab.net), it's a true open-source project under a BSD-style license. A number of developers work for CollabNet, some work for other large companies (such as RedHat), and many others are simply excellent volunteers who are interested in building a better version control system.

The community exists mainly through mailing lists and a Subversion repository. To participate:

Go to http://subversion.tigris.org/ and

There are many ways to join the project, either by writing code, or by testing and/or helping to manage the bug database. If you'd like to contribute, then look at:

To submit code, simply send your patches to dev@subversion.tigris.org. No, wait, first read the rest of this file, then start sending patches to dev@subversion.tigris.org. :-)

To help manage the issues database, read over the issue summaries, looking and testing for issues that are either invalid, or are duplicates of other issues. Both kinds are very common, the first because bugs often get unknowingly fixed as side effects of other changes in the code, and the second because people sometimes file an issue without noticing that it has already been reported. If you are not sure about an issue, post a question to dev@subversion.tigris.org. ("Subversion: We're here to help you help us!")

Another way to help is to set up automated builds and test suite runs of Subversion on some platform, and have the output sent to the svn-breakage@subversion.tigris.org mailing list. See more details at http://subversion.tigris.org/servlets/ProjectMailingListList in the description for the svn-breakage list.

Theory and documentation

  1. Design

    A design spec was written in June 2000, and is a bit out of date. But it still gives a good theoretical introduction to the inner workings of the repository, and to Subversion's various layers.

  2. API Documentation

    See the section on the public API documentation for more information.

  3. Delta Editors

    Karl Fogel wrote a chapter for O'Reilly's 2007 book Beautiful Code: Leading Programmers Explain How They Think covering the design and use of Subversion's delta editor interface.

  4. Network Protocols

    The WebDAV Usage document is an introduction to Subversion's DAV network protocol, which is an extended dialect of HTTP and uses URLs beginning with "http://" or "https://".

    The SVN Protocol document contains a formal description of Subversion ra_svn network protocol, which is a custom protocol on port 3690 (by default), whose URLs begin with "svn://" or "svn+ssh://".

  5. User Manual

    Version Control with Subversion is a book published by O'Reilly that shows in detail how to effectively use Subversion. The text of the book is free, and is actively being revised. On-line versions are available at http://svnbook.red-bean.com. The XML source and translations to other languages are maintained in their own repository at http://svn.red-bean.com/svnbook.

  6. System notes

    A lot of the design ideas for particular aspects of the system have been documented in individual files in the notes/ directory.

Code to read

Before you can contribute code, you'll need to familiarize yourself with the existing code base and interfaces.

Check out a copy of Subversion (anonymously, if you don't yet have an account with commit access) — so you can look at the code.

Within 'subversion/include/' are a bunch of header files with huge doc comments. If you read through these, you'll have a pretty good understanding of the implementation details. Here's a suggested perusal order:

  1. the basic building blocks: svn_string.h, svn_error.h, svn_types.h

  2. useful utilities: svn_io.h, svn_path.h, svn_hash.h, svn_xml.h

  3. the critical interface: svn_delta.h

  4. client-side interfaces: svn_ra.h, svn_wc.h, svn_client.h

  5. the repository and versioned filesystem: svn_repos.h, svn_fs.h

Subversion tries to stay portable by using only ANSI/ISO C and by using the Apache Portable Runtime (APR) library. APR is the portability layer used by the Apache httpd server, and more information can be found at http://apr.apache.org/.

Because Subversion depends so heavily on APR, it may be hard to understand Subversion without first glancing over certain header files in APR (look in 'apr/include/'):

Subversion also tries to deliver reliable and secure software. This can only be achieved by developers who understand secure programming in the C programming language. Please see 'notes/assurance.txt' for the full rationale behind this. Specifically, you should make it a point to carefully read David Wheeler's Secure Programming (as mentioned in 'notes/assurance.txt'). If at any point you have questions about the security implications of a change, you are urged to ask for review on the developer mailing list.

Directory layout

A rough guide to the source tree:

Coding style

Subversion uses ANSI C, and follows the GNU coding standards, except that we do not put a space between the name of a function and the opening parenthesis of its parameter list. Emacs users can just load svn-dev.el to get the right indentation behavior (most source files here will load it automatically, if `enable-local-eval' is set appropriately).

Read http://www.gnu.org/prep/standards.html for a full description of the GNU coding standards. Below is a short example demonstrating the most important formatting guidelines, including our no-space-before-param-list-paren exception:

   char *                                     /* func type on own line */
   argblarg(char *arg1, int arg2)             /* func name on own line */
   {                                          /* first brace on own line */
     if ((some_very_long_condition && arg2)   /* indent 2 cols */
         || remaining_condition)              /* new line before operator */
       {                                      /* brace on own line, indent 2 */
         arg1 = some_func(arg1, arg2);        /* NO SPACE BEFORE PAREN */
       }                                      /* close brace on own line */
     else
       {
         do                                   /* format do-while like this */
           {
             arg1 = another_func(arg1);
           }
         while (*arg1);
       }
   }

In general, be generous with parentheses even when you're sure about the operator precedence, and be willing to add spaces and newlines to avoid "code crunch". Don't worry too much about vertical density; it's more important to make code readable than to fit that extra line on the screen.

Secure coding guidelines

Just like almost any other programming language, C has undesirable features which enables an attacker to make your program fail in predictable ways, often to the attacker's benefit. The goal of these guidelines is to make you aware of the pitfalls of C as they apply to the Subversion project. You are encouraged to keep these pitfalls in mind when reviewing code of your peers, as even the most skilled and paranoid programmers make occasional mistakes.

Input validation is the act of defining legal input and rejecting everything else. The code must perform input validation on all untrusted input.

Security boundaries:

A security boundary in the Subversion server code must be identified as such as this enables auditors to quickly determine the quality of the boundary. Security boundaries exist where the running code has access to information the user does not or where the code runs with privileges above those of the user making the request. Typical examples of such is code that does access control or an application with the SUID bit set.

Functions which make calls to a security boundary must include validation checks of the arguments passed. Functions which themselves are security boundaries should audit the input received and alarm when invoked with improper values.

[### todo: need some examples from Subversion here...]

String operations:

Use the string functions provided in apr_strings.h instead of standard C library functions that write to strings. The APR functions are safer because they do bounds-checking and dest allocation automatically. Although there may be circumstances where it's theoretically safe to use plain C string functions (such as when you already know the lengths of the source and dest), please use the APR functions anyway, so the code is less brittle and more reviewable.

Password storage:

Help users keep their passwords secret: When the client reads or writes password locally, it should ensure that the file is mode 0600. If the file is readable by other users, the client should exit with a message that tells the user to change the filemode due to the risk of exposure.

Destruction of stacked resources

Some resources need destruction to ensure correct functioning of the application. Such resources include files, especially since open files cannot be deleted on Windows.

When writing an API which creates and returns a stream, in the background this stream may be stacked on a file or other stream. To ensure correct destruction of the resources the stream is built upon, it must correctly call the destructors of the stream(s) it is built upon (owns).

At first in http://svn.haxx.se/dev/archive-2005-12/0487.shtml and later in http://svn.haxx.se/dev/archive-2005-12/0633.shtml this was discussed in more general terms for files, streams, editors and window handlers.

As Greg Hudson put it:

On consideration, here is where I would like us to be:

  • Streams which read from or write to an underlying object own that object, i.e. closing the stream closes the underlying object, if applicable.
  • The layer (function or data type) which created a stream is responsible for closing it, except when the above rule applies.
  • Window handlers are thought of as an odd kind of stream, and passing the final NULL window is considered closing the stream.

If you think of apply_textdelta as creating a window handler, then I don't think we're too far off. svn_stream_from_aprfile isn't owning its subsidiary file, svn_txdelta_apply is erroneously taking responsibility for closing the window stream it is passed, and there may be some other deviations.

There is one exception to the rules above though. When a stream is passed to a function as an argument (for example: the 'out' parameter of svn_client_cat2()), that routine can't call the streams destructor, since it did not create that resource.

If svn_client_cat2() creates a stream, it must also call the destructor for that stream. By the above model, that stream will call the destructor for the 'out' parameter. This is however wrong, because the responsibility to destruct the 'out' parameter lies elsewhere.

To solve this problem, at least in the stream case, svn_stream_disown() has been introduced. This function wraps a stream, making sure it's not destroyed, even though any streams stacked upon it may try to do so.

Documentation

Document Everything

Every function, whether public or internal, must start out with a documentation comment that describes what the function does. The documentation should mention every parameter received by the function, every possible return value, and (if not obvious) the conditions under which the function could return an error.

For internal documentation put the parameter names in upper case in the doc string, even when they are not upper case in the actual declaration, so that they stand out to human readers.

For public or semi-public API functions, the doc string should go above the function in the .h file where it is declared; otherwise, it goes above the function definition in the .c file.

For structure types, document each individual member of the structure as well as the structure itself.

For actual source code, internally document chunks of each function, so that an someone familiar with Subversion can understand the algorithm being implemented. Do not include obvious or overly verbose documentation; the comments should help understanding of the code, not hinder it.

For example:

  /*** How not to document.  Don't do this. ***/

  /* Make a foo object. */
  static foo_t *
  make_foo_object(arg1, arg2, apr_pool_t *pool)
  {
     /* Create a subpool. */
     apr_pool_t *subpool = svn_pool_create(pool);

     /* Allocate a foo object from the main pool */
     foo_t *foo = apr_palloc(pool, sizeof(*foo));
     ...
  }

Instead, document decent sized chunks of code, like this:

      /* Trasmit the segment (if its within the scope of our concern). */
      SVN_ERR(maybe_crop_and_send_segment(segment, start_rev, end_rev,
                                          receiver, receiver_baton, subpool));

      /* If we've set CURRENT_REV to SVN_INVALID_REVNUM, we're done
         (and didn't ever reach END_REV).  */
      if (! SVN_IS_VALID_REVNUM(current_rev))
        break;

      /* If there's a gap in the history, we need to report as much
         (if the gap is within the scope of our concern). */
      if (segment->range_start - current_rev < 1)
        {
          svn_location_segment_t *gap_segment;
          gap_segment = apr_pcalloc(subpool, sizeof(*gap_segment));
          gap_segment->range_end = segment->range_start - 1;
          gap_segment->range_start = current_rev + 1;
          gap_segment->path = NULL;
          SVN_ERR(maybe_crop_and_send_segment(gap_segment, start_rev, end_rev,
                                              receiver, receiver_baton,
                                              subpool));
        }

Read over the Subversion code to get an overview of how documentation looks in practice; in particular, see subversion/include/*.h for doxygen examples.

Public API Documentation

We use the Doxygen format for public interface documentation. This means anything that goes in a public header file. Snapshots of the public API documentation are generated nightly from the latest Subversion sources.

We use only a small portion of the available doxygen commands to markup our source. When writing doxygen documentation, the following conventions apply:

  • Use complete sentences and prose descriptions of the function, preceding paramater names with @a, and type and macro names with @c.
  • Use <tt>...<tt> to display multiple words and @p to display only one word in typewriter font.
  • Constant values, such as TRUE, FALSE and NULL should be in all caps.
  • When several functions are related, define a group name, and group them together using @defgroup and @{...@}.

See the Doxygen manual for a complete list of commands.

Using page breaks

We're using page breaks (the Ctrl-L character, ASCII 12) for section boundaries in both code and plaintext prose files. Each section starts with a page break, and immediately after the page break comes the title of the section.

This helps out people who use the Emacs page commands, such as `pages-directory' and `narrow-to-page'. Such people are not as scarce as you might think, and if you'd like to become one of them, then add (require 'page-ext) to your .emacs and type C-x C-p C-h sometime.

Error message conventions

For error messages the following conventions apply:

Also read about Localization.

Other conventions

In addition to the GNU standards, Subversion uses these conventions:

APR pool usage conventions

(This assumes you already basically understand how APR pools work; see apr_pools.h for details.)

Applications using the Subversion libraries must call apr_initialize() before calling any Subversion functions.

Subversion's general pool usage strategy can be summed up in two principles:

  1. The call level that created a pool is the only place to clear or destroy that pool.

  2. When iterating an unbounded number of times, create a subpool before entering the iteration, use it inside the loop and clear it at the start of each iteration, then destroy it after the loop is done, like so:

             apr_pool_t *subpool = svn_pool_create(pool);

         for (i = 0; i < n; ++i)
         {
           svn_pool_clear(subpool);
           do_operation(..., subpool);
         }

         svn_pool_destroy(subpool);

By using a loop subpool for loop-bounded data, you ensure O(1) instead of O(N) memory leakage should the function return abruptly from within the loop (say, due to error). That's why you shouldn't make a subpool for data which persists throughout a function, but instead should use the pool passed in by the caller. That memory will be reclaimed when the caller's pool is cleared or destroyed. If the caller is invoking the callee in a loop, then trust the caller to take care of clearing the pool on each iteration. The same logic propagates all the way up the call stack.

The pool you use also helps readers of the code understand object lifetimes. Is a given object used only during one iteration of the loop, or will it need to last beyond the end of the loop? For example, pool choices indicate a lot about what's going on in this code:

      apr_hash_t *persistent_objects = apr_hash_make(pool);
      apr_pool_t *subpool = svn_pool_create(pool);

      for (i = 0; i < n; ++i)
      {
        const char *intermediate_result;
        const char *key, *val;
        
        svn_pool_clear(subpool);
        SVN_ERR(do_something(&intermediate_result, ..., subpool));
        SVN_ERR(get_result(intermediate_result, &key, &val, ..., pool));
        apr_hash_set(persistent_objects, key, APR_HASH_KEY_STRING, val);
      }
      svn_pool_destroy(subpool);

      return persistent_objects;

Except for some legacy code, which was written before these principles were fully understood, virtually all pool usage in Subversion follows the above guidelines.

One such legacy pattern is a tendency to allocate an object inside a pool, store the pool in the object, and then free that pool (either directly or through a close_foo() function) to destroy the object.

For example:

   /*** Example of how NOT to use pools.  Don't be like this. ***/

   static foo_t *
   make_foo_object(arg1, arg2, apr_pool_t *pool)
   {
      apr_pool_t *subpool = svn_pool_create(pool);
      foo_t *foo = apr_palloc(subpool, sizeof(*foo));

      foo->field1 = arg1;
      foo->field2 = arg2;
      foo->pool   = subpool;
   }

   [...]

   [Now some function calls make_foo_object() and returns, passing
   back a new foo object.]

   [...]

   [Now someone, at some random call level, decides that the foo's
   lifetime is over, and calls svn_pool_destroy(foo->pool).]

This is tempting, but it defeats the point of using pools, which is to not worry so much about individual allocations, but rather about overall performance and lifetime groups. Instead, foo_t generally should not have a `pool' field. Just allocate as many foo objects as you need in the current pool — when that pool gets cleared or destroyed, they will all go away simultaneously.

See also the Exception handling section, for details of how resources associated with a pool are cleaned up when that pool is destroyed.

In summary:

See also Tracking down memory leaks for tips on diagnosing pool usage problems.

APR status codes

Always check for APR status codes (except APR_SUCCESS) with the APR_STATUS_IS_...() macros, not by direct comparison. This is required for portability to non-Unix platforms.

Exception handling

OK, here's how to use exceptions in Subversion.

  1. Exceptions are stored in svn_error_t structures:

    typedef struct svn_error_t
{
  apr_status_t apr_err;      /* APR error value, possibly SVN_ custom err */
  const char *message;       /* details from producer of error */
  struct svn_error_t *child; /* ptr to the error we "wrap" */
  apr_pool_t *pool;          /* place to generate message strings from */
  const char *file;          /* Only used iff SVN_DEBUG */
  long line;                 /* Only used iff SVN_DEBUG */
} svn_error_t;

  2. If you are the original creator of an error, you would do something like this:

    return svn_error_create(SVN_ERR_FOO, NULL, 
                        "User not permitted to write file");

    NOTICE the NULL field... indicating that this error has no child, i.e. it is the bottom-most error.

    See also the section on writing error messages.

    Subversion internally uses UTF-8 to store its data. This also applies to the 'message' string. APR is assumed to return its data in the current locale, so any text returned by APR needs conversion to UTF-8 before inclusion in the message string.

  3. If you receive an error, you have three choices:

    1. Handle the error yourself. Use either your own code, or just call the primitive svn_handle_error(err). (This routine unwinds the error stack and prints out messages converting them from UTF-8 to the current locale.)

      When your routine receives an error which it intends to ignore or handle itself, be sure to clean it up using svn_error_clear(). Any time such an error is not cleared constitutes a memory leak.

    2. Throw the error upwards, unmodified:

              error = some_routine(foo);
        if (error)
          return (error);

      Actually, a better way to do this would be with the SVN_ERR() macro, which does the same thing:

              SVN_ERR(some_routine(foo));

    3. Throw the error upwards, wrapping it in a new error structure by including it as the "child" argument:

              error = some_routine(foo);
        if (error)
          {
           svn_error_t *wrapper = svn_error_create(SVN_ERR_FOO, error,
                                                   "Authorization failed");
           return wrapper;
          }

      Of course, there's a convenience routine which creates a wrapper error with the same fields as the child, except for your custom message:

              error = some_routine(foo);
        if (error)
          {
           return svn_error_quick_wrap(error, 
                                       "Authorization failed");
          }

      The same can (and should) be done by using the SVN_ERR_W() macro:

                SVN_ERR_W(some_routine(foo), "Authorization failed");

    In cases (b) and (c) it is important to know that resources allocated by your routine which are associated with a pool, are automatically cleaned up when the pool is destroyed. This means that there is no need to cleanup these resources before passing the error. There is therefore no reason not to use the SVN_ERR() and SVN_ERR_W() macros. Resources associated with pools are:

    • Memory

    • Files

      All files opened with apr_file_open are closed at pool cleanup. Subversion uses this function in its svn_io_file_* api, which means that files opened with svn_io_file_* or apr_file_open will be closed at pool cleanup.

      Some files (lock files for example) need to be removed when an operation is finished. APR has the APR_DELONCLOSE flag for this purpose. The following functions create files which are removed on pool cleanup:

      • apr_file_open and svn_io_file_open (when passed the APR_DELONCLOSE flag)

      • svn_io_open_unique_file (when passed TRUE in its delete_on_close)

      Locked files are unlocked if they were locked using svn_io_file_lock.

Automated tests

For a description of how to use and add tests to Subversion's automated test framework, please read subversion/tests/README and subversion/tests/cmdline/README.

Various people have arranged for the automated test framework to run at regular intervals on their own machines, sending the results to the svn-breakage@subversion.tigris.org mailing list. The more different platforms the tests run on, the more quickly we can detect portability bugs in Subversion. If you'd like to send svn-breakage messages too, use the svntest framework (start at the README).

Lieven Govaerts has set up a BuildBot build/test farm at http://www.mobsol.be/buildbot/, see his message

   http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=114212
   (Thread URL: http://subversion.tigris.org/servlets/BrowseList?list=dev&by=thread&from=450110)
   Message-ID: 20060326205918.F3C50708B0@adicia.telenet-ops.be
   From: "Lieven Govaerts" <lgo@mobsol.be>
   To: <dev@subversion.tigris.org>
   Subject: Update: Subversion build and test farm with Buildbot.
   Date: Sun, 26 Mar 2006 22:56:11 +0200

for more details. (BuildBot is a system for centrally managing multiple automated testing environments; it's especially useful for portability testing, including of uncommitted changes.)

Writing test cases before code

From: Karl Fogel <kfogel@collab.net>
Subject: writing test cases
To: dev@subversion.tigris.org
Date: Mon, 5 Mar 2001 15:58:46 -0600

Many of us implementing the filesystem interface have now gotten into
the habit of writing the test cases (see fs-test.c) *before* writing
the actual code.  It's really helping us out a lot -- for one thing,
it forces one to define the task precisely in advance, and also it
speedily reveals the bugs in one's first try (and second, and
third...).

I'd like to recommend this practice to everyone.  If you're
implementing an interface, or adding an entirely new feature, or even
just fixing a bug, a test for it is a good idea.  And if you're going
to write the test anyway, you might as well write it first. :-)

Yoshiki Hayashi's been sending test cases with all his patches lately,
which is what inspired me to write this mail to encourage everyone to
do the same.  Having those test cases makes patches easier to examine,
because they show the patch's purpose very clearly.  It's like having
a second log message, one whose accuracy is verified at run-time.

That said, I don't think we want a rigid policy about this, at least
not yet.  If you encounter a bug somewhere in the code, but you only
have time to write a patch with no test case, that's okay -- having
the patch is still useful; someone else can write the test case.

As Subversion gets more complex, though, the automated test suite gets
more crucial, so let's all get in the habit of using it early.

-K

Debugging the server

Debugging the DAV server

'mod_dav_svn.so' contains the main Subversion server logic; it runs as a module within mod_dav, which runs as a module within httpd. Since httpd is probably using dynamic shared modules, you normally won't be able to set breakpoints in advance when you start Apache in a debugger such as GDB. Instead, you'll need to start up, then interrupt httpd, set your breakpoint, and continue:

   % gdb httpd
   (gdb) run -X
   ^C
   (gdb) break some_func_in_mod_dav_svn
   (gdb) continue

The -X switch is equivalent to -DONE_PROCESS and -DNO_DETACH, which ensure that httpd runs as a single thread and remains attached to the tty, respectively. As soon as it starts, it sits and waits for requests; that's when you hit control-C and set your breakpoint.

You'll probably want to watch Apache's run-time logs

   /usr/local/apache2/logs/error_log
   /usr/local/apache2/logs/access_log

to help determine what might be going wrong and where to set breakpoints.

Debugging the ra_svn client and server, on Unix

Bugs in ra_svn usually manifest themselves with one of the following cryptic error messages:

  svn: Malformed network data
  svn: Connection closed unexpectedly

(The first message can also mean the data stream was corrupted in tunnel mode by user dotfiles or hook scripts; see issue #1145.) The first message generally means you to have to debug the client; the second message generally means you have to debug the server.

It is easiest to debug ra_svn using a build with --disable-shared --enable-maintainer-mode. With the latter option, the error message will tell you exactly what line to set a breakpoint at; otherwise, look up the line number at the end of marshal.c:vparse_tuple() where it returns the "Malformed network data" error.

To debug the client, simply pull it up in gdb, set a breakpoint, and run the offending command:

  % gdb svn
  (gdb) break marshal.c:NNN
  (gdb) run ARGS
  Breakpoint 1, vparse_tuple (list=___, pool=___, fmt=___, 
    ap=___) at subversion/libsvn_ra_svn/marshal.c:NNN
  NNN                                 "Malformed network data");

There are several bits of useful information:

  • A backtrace will tell you exactly what protocol exchange is failing.

  • "print *conn" will show you the connection buffer. read_buf, read_ptr, and read_end represent the read buffer, which can show you the data the marshaller is looking at. (Since read_buf isn't generally 0-terminated at read_end, be careful of falsely assuming that there's garbage data in the buffer.)

  • The format string determines what the marshaller was expecting to see.

To debug the server in daemon mode, pull it up in gdb, set a breakpoint (usually a "Connection closed unexpectedly" error on the client indicates a "Malformed network data" error on the server, although it can also indicate a core dump), and run it with the "-X" option to serve a single connection:

  % gdb svnserve
  (gdb) break marshal.c:NNN
  (gdb) run -X

Then run the offending client command. From there, it's just like debugging the client.

Debugging the server in tunnel mode is more of a pain. You'll need to stick something like "{ int x = 1; while (x); }" near the top of svnserve's main() and put the resulting svnserve in the user path on the server. Then start an operation, gdb attach the process on the server, "set x = 0", and step through the code as desired.

Tracking down memory leaks

Our use of APR pools makes it unusual for us to have memory leaks in the strictest sense; all the memory we allocate will be cleaned up eventually. But sometimes an operation takes more memory than it should; for instance, a checkout of a large source tree should not use much more memory than a checkout of a small source tree. When that happens, it generally means we're allocating memory from a pool whose lifetime is too long.

If you have a favorite memory leak tracking tool, you can configure with --enable-pool-debug (which will make every pool allocation use its own malloc()), arrange to exit in the middle of the operation, and go to it. If not, here's another way:

Writing log messages

Every commit needs a log message.

The intended audience for a log message is a developer who is already familiar with Subversion, but not necessarily familiar with this particular commit. Usually when someone goes back and reads a change, he no longer has in his head all the context around that change. This is true even if he is the author of the change! All the discussions and mailing list threads and everything else may be forgotten; the only clue to what the change is about comes from the log message and the diff itself. People revisit changes with surprising frequency, too: for example, it might be months after the original commit and now the change is being ported to a maintenance branch.

The log message is the introduction to the change. Start it off with one line indicating the general nature of the change, and follow that with a descriptive paragraph if necessary. This not only helps put developers in the right frame of mind for reading the rest of the log message, but also plays well with the "CIA" bot that echoes the first line of each commit to realtime forums like IRC. (For details, see http://cia.navi.cx/.) However, if the commit is just one simple change to one file, then you can dispense with the general description and simply go straight to the detailed description, in the standard filename-then-symbol format shown below.

Throughout the log message, use full sentences, not sentence fragments. Fragments are more often ambiguous, and it takes only a few more seconds to write out what you mean. Certain fragments like "Doc fix", "New file", or "New function" are acceptable because they are standard idioms, and all further details should appear in the source code.

The log message should name every affected function, variable, macro, makefile target, grammar rule, etc, including the names of symbols that are being removed in this commit. This helps people searching through the logs later. Don't hide names in wildcards, because the globbed portion may be what someone searches for later. For example, this is bad:

   * subversion/libsvn_ra_pigeons/twirl.c
     (twirling_baton_*): Removed these obsolete structures.
     (handle_parser_warning): Pass data directly to callees, instead
      of storing in twirling_baton_*.

   * subversion/libsvn_ra_pigeons/twirl.h: Fix indentation.

Later on, when someone is trying to figure out what happened to `twirling_baton_fast', they may not find it if they just search for "_fast". A better entry would be:

   * subversion/libsvn_ra_pigeons/twirl.c
     (twirling_baton_fast, twirling_baton_slow): Removed these
      obsolete structures. 
     (handle_parser_warning): Pass data directly to callees, instead
      of storing in twirling_baton_*. 

   * subversion/libsvn_ra_pigeons/twirl.h: Fix indentation.

The wildcard is okay in the description for `handle_parser_warning', but only because the two structures were mentioned by full name elsewhere in the log entry.

Note how each file gets its own entry prefixed with an "*", and the changes within a file are grouped by symbol, with the symbols listed in parentheses followed by a colon, followed by text describing the change. Please adhere to this format, even when only one file is changed — not only does consistency aid readability, it also allows software to colorize log entries automatically.

As an exception to the above, if you make exactly the same change in several files, list all the changed files in one entry. For example:

   * subversion/libsvn_ra_pigeons/twirl.c,
     subversion/libsvn_ra_pigeons/roost.c:
     Include svn_private_config.h.

If all the changed files are deep inside the source tree, you can shorten the file name entries by noting the common prefix before the change entries:

   [in subversion/bindings/swig/birdsong]

   * dialects/nightingale.c (get_base_pitch): Allow 3/4-tone
     pitch variation to account for trait variability amongst
     isolated populations Erithacus megarhynchos.

   * dialects/gallus_domesticus.c: Remove. Unreliable due to
     extremely low brain-to-body mass ratio.

If your change is related to a specific issue in the issue tracker, then include a string like "issue #N" in the log message, but make sure you still summarize what the change is about. For example, if a patch resolves issue #1729, then the log message might be:

   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.

Try to put related changes together. For example, if you create svn_ra_get_ansible2(), deprecating svn_ra_get_ansible(), then those two things should be near each other in the log message:

   * subversion/include/svn_ra.h
     (svn_ra_get_ansible2): New prototype, obsoletes svn_ra_get_ansible.
     (svn_ra_get_ansible): Deprecate.

For large changes or change groups, group the log entry into paragraphs separated by blank lines. Each paragraph should be a set of changes that accomplishes a single goal, and each group should start with a sentence or two summarizing the change. Truly independent changes should be made in separate commits, of course.

See Crediting for how to give credit to someone else if you are committing their patch, or committing a change they suggested.

One should never need the log entries to understand the current code. If you find yourself writing a significant explanation in the log, you should consider carefully whether your text doesn't actually belong in a comment, alongside the code it explains. Here's an example of doing it right:

   (consume_count): If `count' is unreasonable, return 0 and don't
    advance input pointer.

And then, in `consume_count' in `cplus-dem.c':

   while (isdigit((unsigned char)**type))
     {
       count *= 10;
       count += **type - '0';
       /* A sanity check.  Otherwise a symbol like
         `_Utf390_1__1_9223372036854775807__9223372036854775'
         can cause this function to return a negative value.
         In this case we just consume until the end of the string.  */
      if (count > strlen(*type))
        {
          *type = save;
          return 0;
        }

This is why a new function, for example, needs only a log entry saying "New Function" --- all the details should be in the source.

You can make common-sense exceptions to the need to name everything that was changed. For example, if you have made a change which requires trivial changes throughout the rest of the program (e.g., renaming a variable), you needn't name all the functions affected, you can just say "All callers changed". However, if you rename symbols (e.g., to add or remove a library prefix), please mention every affected symbol, both old and new — this is the only way the rename of a particular symbol would be traceable later. See r20946 for an example of this.

In general, there is a tension between making entries easy to find by searching for identifiers, and wasting time or producing unreadable entries by being exhaustive. Use the above guidelines and your best judgment, and be considerate of your fellow developers. (Also, run "svn log" to see how others have been writing their log entries.)

Log messages for documentation or translation have somewhat looser guidelines. The requirement to name every symbol obviously does not apply, and if the change is just one more increment in a continuous process such as translation, it's not even necessary to name every file. Just briefly summarize the change, for example: "More work on Malagasy translation." Please write your log messages in English, so everybody involved in the project can understand the changes you made.

Crediting

It is very important to record code contributions in a consistent and parseable way. This allows us to write scripts to figure out who has been actively contributing — and what they have contributed — so we can spot potential new committers quickly. The Subversion project uses human-readable but machine-parseable fields in log messages to accomplish this.

When committing a patch written by someone else, use "Patch by: " at the beginning of a line to indicate the author:

   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.

   Patch by: J. Random <jrandom@example.com>

If multiple individuals wrote the patch, list them each on a separate line — making sure to start each continuation line with whitespace. Non-committers should be listed by name, if known, and e-mail. Committers may be listed similarly, or by their canonical usernames from COMMITTERS (the leftmost column). Additionally, "me" is an acceptable shorthand for the person actually committing the change.

   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.

   Patch by: J. Random <jrandom@example.com>
             Enrico Caruso <codingtenor@codingtenor.com>
             jcommitter
             me

If someone found the bug or pointed out the problem, but didn't write the patch, indicate their contribution with "Found by: ":

   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.

   Found by: J. Random <jrandom@example.com>

If someone suggested something useful, but didn't write the patch, indicate their contribution with "Suggested by: ":

   Extend the Contribulyzer syntax to distinguish finds from ideas.

   * www/hacking.html (crediting): Adjust accordingly.

   Suggested by: dlr

If someone reviewed the change, use "Review by: " (or "Reviewed by: " if you prefer):

   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.

   Review by: Eagle Eyes <eeyes@example.com>

A field may have multiple lines, and a log message may contain any combination of fields:

   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.

   Patch by: J. Random <jrandom@example.com>
             Enrico Caruso <codingtenor@codingtenor.com>
             me
   Found by: J. Random <jrandom@example.com>
   Review by: Eagle Eyes <eeyes@example.com>
              jcommitter

Further details about a contribution should be listed in a parenthetical aside immediately after the corresponding field. Such an aside always applies to the field right above it; in the following example, the fields have been spaced out for readability, but note that the spacing is optional and not necessary for parseability:

   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.

   Patch by: J. Random <jrandom@example.com>
   (Tweaked by me.)

   Review by: Eagle Eyes <eeyes@example.com>
              jcommitter
   (Eagle Eyes caught an off-by-one-error in the basename extraction.)

Currently, these fields

   Patch by:
   Suggested by:
   Found by:
   Review by:

are the only officially-supported crediting fields (where "supported" means scripts know to look for them), and they are widely used in Subversion log messages. Future fields will probably be of the form "VERB by: ", and from time to time someone may use a field that sounds official but really is not — for example, there are a few instances of "Reported by: ". These are okay, but try to use an official field, or a parenthetical aside, in preference to creating your own. Also, don't use "Reported by: " when the reporter is already recorded in an issue; instead, simply refer to the issue.

Look over Subversion's existing log messages to see how to use these fields in practice. This command from the top of your trunk working copy will help:

svn log | contrib/client-side/search-svnlog.pl "(Patch|Review|Suggested) by: "

Note: The "Approved by: " field seen in some commit messages is totally unrelated to these crediting fields, and is generally not parsed by scripts. It is simply the standard syntax for indicating either who approved a partial committer's commit outside their usual area, or (in the case of merges to release branches) who voted for the change to be merged.

Patch submission guidelines

Mail patches to dev@subversion.tigris.org, starting the subject line with [PATCH]. This helps our patch manager spot patches right away. For example:

   Subject: [PATCH] fix for rev printing bug in svn status

If the patch addresses a particular issue, include the issue number as well: "[PATCH] issue #1729: ...". Developers who are interested in that particular issue will know to read the mail.

A patch submission should contain one logical change; please don't mix N unrelated changes in one submission — send N separate emails instead.

Generate the patch using svn diff from the top of a Subversion trunk working copy. If the file you're diffing is not under revision control, you can achieve the same effect by using diff -u.

Please include a log message with your patch. A good log message helps potential reviewers understand the changes in your patch, and increases the likelihood that it will be applied. You can put the log message in the body of the email, or at the top of the patch attachment (see below). Either way, it should follow the guidelines given in Writing log messages, and be enclosed in triple square brackets, like so:

   [[[
   Fix issue #1729: Don't crash because of a missing file.

   * subversion/libsvn_ra_ansible/get_editor.c
     (frobnicate_file): Check that file exists before frobnicating.
   ]]]

(The brackets are not actually part of the log message, they're just a way to clearly mark off the log message from its surrounding context.)

If possible, send the patch as an attachment with a mime-type of text/x-diff, text/x-patch, or text/plain. Most people's mailreaders can display those inline, and having the patch as an attachment allows them to extract the patch from the message conveniently. Never send patches in archived or compressed form (e.g., tar, gzip, zip, bzip2), because that prevents people from reviewing the patch directly in their mailreaders.

If you can't attach the patch with one of these mime-types, or if the patch is very short, then it's okay to include it directly in the body of your message. But watch out: some mail editors munge inline patches by inserting unasked-for line breaks in the middle of long lines. If you think your mail software might do this, then please use an attachment instead.

If the patch implements a new feature, make sure to describe the feature completely in your mail; if the patch fixes a bug, describe the bug in detail and give a reproduction recipe. An exception to these guidelines is when the patch addresses a specific issue in the issues database — in that case, just refer to the issue number in your log message, as described in Writing log messages.

It is normal for patches to undergo several rounds of feedback and change before being applied. Don't be discouraged if your patch is not accepted immediately — it doesn't mean you goofed, it just means that there are a lot of eyes looking at every code submission, and it's a rare patch that doesn't have at least a little room for improvement. After reading people's responses to your patch, make the appropriate changes and resubmit, wait for the next round of feedback, and lather, rinse, repeat, until some committer applies it.

If you don't get a response for a while, and don't see the patch applied, it may just mean that people are really busy. Go ahead and repost, and don't hesitate to point out that you're still waiting for a response. One way to think of it is that patch management is highly parallizable, and we need you to shoulder your share of the management as well as the coding. Every patch needs someone to shepherd it through the process, and the person best qualified to do that is the original submitter.

The "Patch Manager" Role

Subversion usually has a Patch Manager, whose job is to watch the dev@ mailing list and make sure that no patches "slip through the cracks".

This means watching every thread containing "[PATCH]" mails, and taking appropriate action based on the progress of the thread. If the thread resolves on its own (because the patch gets committed, or because there is consensus that the patch doesn't need to be applied, or whatever) then no further action need be taken. But if the thread fades out without any clear decision, then the patch needs to be saved in the issue tracker. This means that a summary of any discussion threads around that patch, and links to relevant mailing list archives, will be added to some issue in the tracker. For a patch which addresses an existing issue tracker item, the patch is saved to that item. Otherwise, a new issue of type 'PATCH' is filed, and the patch is saved to that new issue.

The Patch Manager needs a basic technical understanding of Subversion, and the ability to skim a thread and get a rough understanding of whether consensus has been reached, and if so, of what kind. It does not require actual Subversion development experience or commit access. Expertise in using one's mail reading software is optional, but recommended :-).

The current patch manager is Hyrum Wright <hyrum_wright@mail.utexas.edu>.

Filing bugs / issues

This pretty much says it all:

   From: Karl Fogel <kfogel@collab.net>
   Subject: Please ask on the list before filing a new issue.
   To: dev@subversion.tigris.org
   Date: Tue, 30 Jul 2002 10:51:24 (CDT)
   
   Folks, we're getting tons of new issues, which is a Good Thing in
   general, but some of them don't really belong in the issue tracker.
   They're things that would be better solved by a quick conversation
   here on the dev list.  Compilation problems, behavior questions,
   feature ideas that have been discussed before, that sort of thing.
   
   *Please* be more conservative about filing issues.  The issues
   database is physically much more cumbersome than email.  It wastes
   people's time to have conversations in the issues database that should
   be had in email.  (This is not a libel against the issue tracker, it's
   just a result of the fact that the issues database is for permanent
   storage and flow annotation, not for real-time conversation.)
   
   If you encounter a situation where Subversion is clearly behaving
   wrongly, or behaving opposite to what the documentation says, then
   it's okay to file the issue right away (after searching to make sure
   it isn't already filed, of course!).  But if you're
   
      a) Requesting a new feature, or
      b) Having build problems, or
      c) Not sure what the behavior should be, or
      d) Disagreeing with current intended behavior, or
      e) Not TOTALLY sure that others would agree this is a bug, or
      f) For any reason at all not sure this should be filed,
   
   ...then please post to the dev list first.  You'll get a faster
   response, and others won't be forced to use the issues database to
   have the initial real-time conversations.
   
   Nothing is lost this way.  If we eventually conclude that it should be
   in the issue tracker, then we can still file it later, after the
   description and reproduction recipe have been honed on the dev list.
   
   Thank you,
   -Karl

Issue triage

When an issue is filed, it goes into the special milestone "---", meaning unmilestoned. This is a holding area that issues live in until someone gets a chance to look at them and decide what to do.

The unmilestoned issues are listed first when you sort by milestone, and issue triage is the process of trawling through all the open issues (starting with the unmilestoned ones), determining which are important enough to be fixed now, which can wait until another release, which are duplicates of existing issues, which have been resolved already, etc. For each issue that will remain open, it also means making sure that the various fields are set appropriately: type, subcomponent, platform, OS, version, keywords (if any), and so on.

Here's an overview of the process (in this example, 1.5 is the next release, so urgent issues would go there):

    for i in issues_marked_as("---"):
      if issue_is_a_dup_of_some_other_issue(i):
        close_as_dup(i)
      elif issue_is_invalid(i):
        # A frequent reason for invalidity is that the reporter
        # did not follow the "buddy system" for filing.
        close_as_invalid(i)
      elif issue_already_fixed(i):
        close_as_fixed(i)
      elif issue_unreproducible(i):
        close_as_worksforme(i)
      elif issue_is_real_but_we_won't_fix_it(i):
        close_as_wontfix(i)
      elif issue_is_closeable_for_some_other_reason(i):
        close_as_it_for_that_reason(i)

      # Else issue should remain open, so DTRT with it...

      # Set priority, platform, subcomponent, etc, as needed.
      adjust_all_fields_that_need_adjustment(i)

      # Figure out where to put it.
      if issue_is_a_lovely_fantasy(i):
        move_to_milestone(i, "blue-sky")
      if issue_is_not_important_enough_to_block_any_particular_release(i):
        move_to_milestone(i, "nonblocking")
      elif issue_resolution_would_require_incompatible_changes(i):
        move_to_milestone(i, "2.0")
      elif issue_hurts_people_somewhat(i):
        move_to_milestone(i, "1.6")  # or whatever
      elif issue_hurts_people_a_lot(i):
        move_to_milestone(i, "1.5-consider")
      elif issue_hurts_and_hurts_and_won't_stop_hurting(i):
        move_to_milestone(i, "1.5")

Commit access

There are two types of commit access: full and partial. Full means anywhere in the tree, partial means only in that committer's specific area(s) of expertise. The COMMITTERS file lists all committers, both full and partial, and says the domains for each partial committer.

How full commit access is granted

After someone has successfully contributed a few non-trivial patches, some full committer, usually whoever has reviewed and applied the most patches from that contributor, proposes them for commit access. This proposal is sent only to the other full committers -- the ensuing discussion is private, so that everyone can feel comfortable speaking their minds. Assuming there are no objections, the contributor is granted commit access. The decision is made by consensus; there are no formal rules governing the procedure, though generally if someone strongly objects the access is not offered, or is offered on a provisional basis.

The primary criterion for full commit access is good judgment.

You do not have to be a technical wizard, or demonstrate deep knowledge of the entire codebase, to become a full committer. You just need to know what you don't know. If your patches adhere to the guidelines in this file, adhere to all the usual unquantifiable rules of coding (code should be readable, robust, maintainable, etc.), and respect the Hippocratic Principle of "first, do no harm", then you will probably get commit access pretty quickly. The size, complexity, and quantity of your patches do not matter as much as the degree of care you show in avoiding bugs and minimizing unnecessary impact on the rest of the code. Many full committers are people who have not made major code contributions, but rather lots of small, clean fixes, each of which was an unambiguous improvement to the code. (Of course, this does not mean the project needs a bunch of very trivial patches whose only purpose is to gain commit access; knowing what's worth a patch post and what's not is part of showing good judgement :-) .)

To assist developers in discovering new committers, we record patches and other contributions in a special crediting format, which is then parsed to produce a browser-friendly contribution list, updated nightly. If you're thinking of proposing someone for commit access and want to look over all their changes, that contribution list might be the most convenient place to do it.

How partial commit access is granted

A full committer sponsors the partial committer. Usually this means the full committer has applied several patches to the same area from the proposed partial committer, and realizes things would be easier if the person were just committing directly. Approval is not required from the full committers; it is assumed that sponsors know what they're doing and will watch the partial committer's first few commits to make sure everything's going smoothly.

Patches submitted by a partial committer may be committed by that committer even if they are outside that person's domain. This requires approval (often expressed as a +1 vote) from at least one full committer. In such a case, the approval should be noted in the log message, like so:

   Approved by: lundblad

Any full committer may offer anyone commit access to an experimental branch at any time. It is not necessary that the experimental branch have a high likelihood of being merged to trunk (although that's always a good goal to aim for). It's just as imporant that the full committer — all the full committers, actually — view such branches as training grounds for new developers, by giving feedback on the commits. The goal of these branches is both to get new code into Subversion and to get new developers into the project. See also the section on lightweight branches, and this mail:

   http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgNo=132746
   From: Karl Fogel <kfogel@red-bean.com>
   To: dev@subversion.tigris.org
   Subject: branch liberalization (was: Elego tree conflicts work)
   Date: Tue, 20 Nov 2007 10:49:38 -0800
   Message-Id: <87y7cswy4d.fsf@red-bean.com>

The contrib/ area

When a tool is accepted into the contrib/ area, we automatically offer its author partial commit access to maintain the tool there. Any full committer can sponsor this. Usually no discussion or vote is necessary, though if there are objections then the usual decision-making procedures apply (attempt to reach consensus first, then vote among the full committers if consensus cannot be reached).

Code under contrib/ must be open source, but need not have the same license or copyright holder as Subversion itself.

The "obvious fix" rule

Any committer, whether full or partial, may commit fixes for obvious typos, grammar mistakes, and formatting problems wherever they may be — in the web pages, API documentation, code comments, commit messages, etc. We rely on the committer's judgement to determine what is "obvious"; if you're not sure, just ask.

Use lightweight branches

If you're working on a feature or bugfix in stages involving multiple commits, and some of the intermediate stages aren't stable enough to go on trunk, then create a temporary branch in /branches. There's no need to ask — just do it. It's fine to try out experimental ideas in a temporary branch, too. And all the preceding applies to partial as well as full committers.

If you're just using the branch to "checkpoint" your code, and don't feel it's ready for review, please put some sort of notice at the top of the log message, such as:

   *** checkpoint commit -- please don't waste your time reviewing it ***

And if a later commit on that branch should be reviewed, then please supply, in the log message, the appropriate 'svn diff' command, since the diff would likely involve two non-adjacent commits on that branch, and reviewers shouldn't have to spend time figuring out which ones they are.

When you're done with the branch — when you've either merged it to trunk or given up on it — please remember to remove it.

See also the section on partial commit access for our policy on offering commit access to experimental branches.

The configuration/build system under unix

Greg Stein wrote a custom build system for Subversion, which had been using `automake' and recursive Makefiles. Now it uses a single, top-level Makefile, generated from Makefile.in (which is kept under revision control). `Makefile.in' in turn includes `build-outputs.mk', which is automatically generated from `build.conf' by the `gen-make.py' script. Thus, the latter two are under revision control, but `build-outputs.mk' is not.

Here is Greg's original mail describing the system, followed by some advice about hacking it:

   From: Greg Stein <gstein@lyra.org>
   Subject:  new build system (was: Re: CVS update: MODIFIED: ac-helpers ...)
   To: dev@subversion.tigris.org
   Date: Thu, 24 May 2001 07:20:55 -0700
   Message-ID: <20010524072055.F5402@lyra.org>

   On Thu, May 24, 2001 at 01:40:17PM -0000, gstein@tigris.org wrote:
   >   User: gstein
   >   Date: 01/05/24 06:40:17
   >
   >   Modified:    ac-helpers .cvsignore svn-apache.m4
   >   Added:       .        Makefile.in
   >   Log:
   >   Switch over to the new non-recursive build system.
   >...

   Okay... this is it. We're now on the build system.

       "It works on my machine."

   I suspect there may be some tweaks to make on different OSs. I'd be
   interested to hear if Ben can really build with normal BSD make. It
   should be possible.

   The code supports building, installation, checking, and
   dependencies. It does *NOT* yet deal with the doc/ subdirectory. That
   is next; I figured this could be rolled out and get the kinks worked
   out while I do the doc/ stuff.  Oh, it doesn't build Neon or APR yet
   either. I also saw a problem where libsvn_fs wasn't getting built
   before linking one of the test proggies (see below).

   Basic operation: same as before.

   $ ./autogen.sh
   $ ./configure OPTIONS
   $ make
   $ make check
   $ make install

   There are some "make check" scripts that need to be fixed up. That'll
   happen RSN. Some of them create their own log, rather than spewing to
   stdout (where the top-level make will place the output into
   [TOP]/tests.log).

   The old Makefile.am files are still around, but I'll be tossing those
   along with a bunch of tweaks to all the .cvsignore files. There are a
   few other cleanups, too. But that can happen as a step two.

   [ $ cvs rm -f `find . -name Makefile.rm`

     See the mistake in that line? I didn't when I typed it. The find
     returned nothing, so cvs rm -f proceeded to delete my entire
     tree. And the -f made sure to delete all my source files, too. Good
     fugging thing that I had my mods in some Emacs buffers, or I'd be
     bitching.

     I am *so* glad that Ben coded SVN to *not* delete locally modified
     files *and* that we have an "undel" command. I had to go and tweak a
     bazillion Entries files to undo the delete...
   ]

   The top-level make has a number of shortcuts in it (well, actually in
   build-outputs.mk):

   $ make subversion/libsvn_fs/libsvn_fs.la

   or

   $ make libsvn_fs

   The two are the same. So... when your test proggie fails to link
   because libsvn_fs isn't around, just run "make libsvn_fs" to build it
   immediately, then go back to the regular "make".

   Note that the system still conditionally builds the FS stuff based
   on whether DB (See 'Building on Unix' below) is available, and
   mod_dav_svn if Apache is available.

   Handy hint: if you don't like dependencies, then you can do:

   $ ./autogen.sh -s

   That will skip the dependency generation that goes into
   build-outputs.mk. It makes the script run quite a bit faster (48 secs
   vs 2 secs on my poor little Pentium 120).

   Note that if you change build.conf, you can simply run:

   $ ./gen-make.py build.conf

   to regen build-outputs.mk. You don't have to go back through the whole
   autogen.sh / configure process.

   You should also note that autogen.sh and configure run much faster now
   that we don't have the automake crap. Oh, and our makefiles never
   re-run configure on you out of the blue (gawd, I hated when automake
   did that to me).

   Obviously, there are going to be some tweaky things going on. I also
   think that the "shadow" builds or whatever they're called (different
   source and build dirs) are totally broken. Something tweaky will have
   to happen there.  But, thankfully, we only have one Makefile to deal
   with.

   Note that I arrange things so that we have one generated file
   (build-outputs.mk), and one autoconf-generated file (Makefile from
   .in).  I also tried to shove as much logic/rules into
   Makefile.in. Keeping build-outputs.mk devoid of rules (thus, implying
   gen-make.py devoid of rules in its output generation) manes that
   tweaking rules in Makefile.in is much more approachable to people.

   I think that is about it. Send problems to the dev@ list and/or feel
   free to dig in and fix them yourself. My next steps are mostly
   cleanup. After that, I'm going to toss out our use of libtool and rely
   on APR's libtool setup (no need for us to replicate what APR already
   did).

   Cheers,
   -g

   --
   Greg Stein, http://www.lyra.org/

And here is some advice for those changing or testing the configuration/build system:

   From: Karl Fogel <kfogel@collab.net>
   To: dev@subversion.tigris.org
   Subject: when changing build/config stuff, always do this first
   Date: Wed 28 Nov 2001

   Yo everyone: if you change part of the configuration/build system,
   please make sure to clean out any old installed Subversion libs
   *before* you try building with your changes.  If you don't do this,
   your changes may appear to work fine, when in fact they would fail if
   run on a truly pristine system.

   This script demonstrates what I mean by "clean out".  This is
   `/usr/local/cleanup.sh' on my system.  It cleans out the Subversion
   libs (and the installed httpd-2.0 libs, since I'm often reinstalling
   that too):

      #!/bin/sh

      # Take care of libs
      cd /usr/local/lib
      rm -f APRVARS
      rm -f libapr*
      rm -f libexpat*
      rm -f libneon*
      rm -f libsvn*

      # Take care of headers
      cd /usr/local/include
      rm -f apr*
      rm -f svn*
      rm -f neon/*

      # Take care of headers
      cd /usr/local/apache2/lib
      rm -f *

   When someone reports a configuration bug and you're trying to
   reproduce it, run this first. :-)

   The voice of experience,
   -Karl

How to release a distribution tarball

See The Subversion Release Procedure.

Release numbering, compatibility, and deprecation

Subversion uses "MAJOR.MINOR.PATCH" release numbers, with the same guidelines as APR (see http://apr.apache.org/versioning.html), plus a few extensions, described later. The general idea is:

  1. Upgrading/downgrading between different patch releases in the same MAJOR.MINOR line never breaks code. It may cause bugfixes to disappear/reappear, but API signatures and semantics remain the same. (Of course, the semantics may change in the trivial ways appropriate for bugfixes, just not in ways that would force adjustments in calling code.)

  2. Upgrading to a new minor release in the same major line may cause new APIs to appear, but not remove any APIs. Any code written to the old minor number will work with any later minor number in that line. However, downgrading afterwards may not work, if new code has been written that takes advantage of the new APIs.

  3. When the major number changes, all bets are off. This is the only opportunity for a full reset of the APIs, and while we try not to gratuitously remove interfaces, we will use it to clean house a bit.

Subversion extends the APR guidelines to cover client/server compatibility questions:

  1. A patch or minor number release of a server (or client) never breaks compatibility with a client (or server) in the same major line. However, new features offered by the release might be unsupported without a corresponding upgrade to the other side of the connection. For updating ra_svn code specifically, please observe these principles:

    1. Fields can be added to any tuple; old clients will simply ignore them. (Right now, the marshalling implementation does not let you put number or boolean values in the optional part of a tuple, but changing that will not affect the protocol.)

      We can use this mechanism when information is added to an API call.

    2. At connection establishment time, clients and servers exchange a list of capability keywords.

      We can use this mechanism for more complicated changes, like introducing pipelining or removing information from API calls.

    3. New commands can be added; trying to use an unsupported command will result in an error which can be checked and dealt with.

    4. The protocol version number can be bumped to allow graceful refusal of old clients or servers, or to allow a client or server to detect when it has to do things the old way.

      This mechanism is a last resort, to be used when capability keywords would be too hard to manage.

  2. Working copy and repository formats are backward- and forward-compatible for all patch releases in the same minor series. They are forward-compatible for all minor releases in the same major series; however, a minor release is allowed to make a working copy or repository that doesn't work with previous minor releases, where "make" could mean "upgrade" as well as "create".

Subversion does not use the "even==stable, odd==unstable" convention; any unqualified triplet indicates a stable release:

   1.0.1  -->  first stable patch release of 1.0
   1.1.0  -->  next stable minor release of 1.x after 1.0.x
   1.1.1  -->  first stable patch release of 1.1.x
   1.1.2  -->  second stable patch release of 1.1.x
   1.2.0  -->  next stable minor release after that

The order of releases is semi-nonlinear — a 1.0.3 might come out after a 1.1.0. But it's only "semi"-nonlinear because eventually we declare a patch line defunct and tell people to upgrade to the next minor release, so over the long run the numbering is basically linear.

Non-stable releases are qualified with "alphaN" or "betaN" suffixes, and release candidates with "-rcN". For example, the prereleases leading to 1.3.7 might look like this:

   subversion-1.3.7-rc1.tar.gz
   subversion-1.3.7-rc2.tar.gz
   subversion-1.3.7-rc3.tar.gz
   subversion-1.3.7.tar.gz

The output of 'svn --version' corresponds in the obvious way:

   version 1.3.7 (Release Candidate 1)
   version 1.3.7 (Release Candidate 2)
   version 1.3.7 (Release Candidate 3)
   version 1.3.7

When you 'make install' subversion-1.3.7-rc1, it still installs as though it were "1.3.7", of course. The qualifiers are metadata on the release; we want each subsequent prerelease release to overwrite the previous one, and the final release to overwrite the last prerelease.

For working copy builds, there is no tarball name to worry about, but 'svn --version' still produces special output:

   version 1.3.7 (dev build)

The version number is the next version that the development was working towards. The important thing is to say "dev build". This indicates that the build came from a working copy, which is useful in bug reports.

We have no mechanism for releasing dated snapshots. If we want code to get wider distribution than just those who build from working copies, we put out a prerelease.

Reuse of release names

If a release or candidate release needs to be quickly re-issued due to some non-code problem (say, a packaging glitch), it's okay to reuse the same name, as long as the tarball hasn't been blessed by signing yet. But if it has been uploaded to the standard distribution area with signatures, or if the re-issue was due to a change in code a user might run, then the old name must be tossed and the next name used.

Deprecation

When a new, improved version of an API is introduced, the old one remains for compatibility, at least until the next major release. However, we mark the old one as deprecated and point to the new one, so people know to write to the new API if at all possible. When deprecating, mention the release after which the deprecation was introduced, and point to the new API. If possible, replace the old API documentation with a diff to the new one. For example:

   /**
    * @deprecated Provided for backward compatibility with the 1.0.0 API.
    *
    * Similar to svn_repos_dump_fs2(), but with the @a use_deltas
    * parameter always set to @c FALSE.
    */
   svn_error_t *svn_repos_dump_fs(svn_repos_t *repos,
                                  svn_stream_t *dumpstream,
                                  svn_stream_t *feedback_stream,
                                  svn_revnum_t start_rev,
                                  svn_revnum_t end_rev,
                                  svn_boolean_t incremental,
                                  svn_cancel_func_t cancel_func,
                                  void *cancel_baton,
                                  apr_pool_t *pool);

When the major release number changes, the "best" new API in a series generally replaces all the previous ones (assuming it subsumes their functionality), and it will take the name of the original API. Thus, marking 'svn_repos_dump_fs' as deprecated in 1.1.x doesn't mean that 2.0.0 doesn't have 'svn_repos_dump_fs', it just means the function's signature will be different: it will have the signature held by svn_repos_dump_fs2 (or svn_repos_dump_fs3, or whatever) in 1.1.x. The numbered-suffix names disappear, and there is a single (shiny, new) svn_repos_dump_fs again.

One exception to this replacement strategy is when the old function has a totally unsatisfying name anyway. Deprecation is a chance to fix that: we give the new API a totally new name, mark the old API as deprecated, point to the new API; then at the major version change, we remove the old API, but don't rename the new one to the old name, because its new name is fine.

Stabilizing and maintaining releases

Minor and major number releases go through a stabilization period before release, and remain in maintenance (bugfix) mode after release. To start the release process, we create an "A.B.x" branch based on the latest trunk, for example:

   $ svn cp http://svn.collab.net/repos/svn/trunk \
            http://svn.collab.net/repos/svn/branches/A.B.x

The stabilization period for a new A.B.0 release normally lasts four weeks, and allows us to make conservative bugfixes and discover showstopper issues. The stabilization period begins with a release candidate tarball with the version A.B.0-rc1. Further release candidate tarballs may be made as blocking bugs are fixed; for example, if a set of language bindings is found to be broken, it is prudent to make a new release candidate when they are fixed so that those language bindings may be tested.

At the beginning of the final week of the stabilization period, a new release candidate tarball should be made if there are any changes pending since the last one. The final week of the stabilization period is reserved for critical bugfixes; fixes for minor bugs should be deferred to the A.B.1 release. A critical bug is a non-edge-case crash, a data corruption problem, a major security hole, or something equally serious.

Under some circumstances, the stabilization period will be extended:

If there are disagreements over whether a change is potentially destabilizing or over whether a bug is critical, they may be settled with a committer vote.

After the A.B.0 release is out, patch releases (A.B.1, A.B.2, etc.) follow when bugfixes warrant them. Patch releases do not require a four week soak, because only conservative changes go into the line.

Certain kinds of commits can go into A.B.0 without restarting the soak period, or into a later release without affecting the testing schedule or release date:

NOTE: The requirements on message translation changes are looser than for text messages in C code. Changing format specifiers in .po files is allowed because their validity can be checked mechanically (with the -c flag on msgfmt of GNU gettext). This is done at build time if GNU gettext is in use.

Core code changes, of course, require voting, and restart the soak or test period, since otherwise the change could be undertested.

The voting system works like this:

A change to the A.B.x line must be first proposed in the A.B.x/STATUS file. Each proposal consists of a short identifying block (e.g., the revision number of a trunk or related-line commit, or perhaps an issue number), a brief description of the change, an at-most-one-line justification of why it should be in A.B.x, perhaps some notes/concerns, and finally the votes. The notes and concerns are meant to be brief summaries to help a reader get oriented; please don't use the STATUS file for actual discussion, use dev@ instead.

Here's an example, probably as complex as an entry would ever get:

   * r98765 (issue #56789)
     Make commit editor take a closure object for future mindreading.
     Justification: API stability, as prep for future enhancement.
     Notes: There was consensus on the desirability of this feature in
       the near future; see thread at http://... (Message-Id: blahblah).
     Concerns: Vetoed by jerenkrantz due to privacy concerns with the
       implementation; see thread at http://... (Message-Id: blahblah)
     Votes:
       +1: ghudson, bliss
       +0: cmpilato
       -0: gstein
       -1: jerenkrantz

A change needs three +1 votes from full committers (or partial committers for the involved areas), and no vetoes, to go into A.B.x.

If you cast a veto (i.e. -1), please state the reason in the concerns field, and include a url / message-id for the list discussion if any. You can go back and add the link later if the thread isn't available at the time you commit the veto.

Voting +1 on a change doesn't just mean you approve of it in principle. It means you have thoroughly reviewed the change, and find it correct and as nondisruptive as possible. When it is committed to the release branch, the log message will include the names of all who voted for it, as well as the original author and the person making the commit. All of these people are considered equally answerable for bugs.

If you've reviewed a patch, and like it but have some reservations, you can write "+1 (concept)" and then ask questions on the list about your concerns. You can write "+0" if you like the general idea but haven't reviewed the patch carefully. Neither of these votes counts toward the total, but they can be useful for tracking down people who are following the change and might be willing to spend more time on it.

There is a somewhat looser voting system for areas that are not core code, and that may have fewer experts available to review changes (for example, tools/, packages/, bindings/, test scripts, etc.). A change in these areas can go in with a +1 from a full committer or a partial committer for that area, at least one +0 or "concept +1" from any other committer, and no vetoes. (If a change affects the build system, however, it is considered a core change, and needs three +1's.) Use your judgment and don't review changes unless you have some competence to do so, of course. The goal is to get at least two pairs of eyes on the change, without demanding that every reviewer have the same amount of expertise as the area maintainer. This way one can review for general sanity, accurate comments, obvious mistakes, etc, without being forced to assert "Yes, I understand these changes in every detail and have tested them."

Before proposing a change in STATUS, you should try merging it onto the branch to ensure that it doesn't produce merge conflicts. If conflicts occur, please create a new temporary branch from the release branch with your changes merged and the conflicts resolved. The branch should be named A.B.x-rYYYY, where YYYY is the first revision of your change in the STATUS file. Add a note in the STATUS file about the existence of the temporary branch. If the change involves further work, you can merge those revisions to the branch. When the entry for this change is removed from STATUS, this temporary branch should also be removed to avoid cluttering the /branches directory.

NOTE: Changes to STATUS regarding the temporary branch, including voting, are always kept on the main release branch.

Signing source distribution packages (a.k.a tarballs)

Before a release or release candidate is officially made public, it is made available in a temporary location for committers to test and sign. The point is to have the tarballs tested on more systems than just that of the person who rolled the release. When there are three signatures from full committers for each of the .tar.bz2, .tar.gz and .zip files, the release (candidate) can go public.

Signing a tarball means that you assert certain things about it. When sending your signature (see below), indicate in the mail what steps you've taken to verify that the tarball is correct. Running make check over all RA layers and FS backends is a good idea, as well as building and testing the bindings.

After having extracted and tested the tarball, you should sign it using gpg. To do so, use a command like:

    gpg -ba subversion-1.3.0-rc4.tar.bz2

This will result in a file with the same name as the signed file, but with a .asc extension in the appropriate format for inclusion in the release announcement. Include this file in a mail, typically in reply to the announcement of the unofficial tarball.

If you've downloaded and tested a .tar.bz2 file, it is possible to sign a .tar.gz file with the same contents without having to download and test it separately. The trick is to extract the .bz2 file, and pack it using gzip like this:

    bzip2 -cd subversion-1.3.0-rc4.tar.bz2 \
    | gzip -9n > subversion-1.3.0-rc4.tar.gz

The resulting file should be identical to the file generated by the release manager, and thus can be signed as described above. To verify that the files are identical, you may use either the MD5 checksums or the release manager's signature, both of which should be provided with the tarballs.

Localization (l10n)

Translation has been divided into two domains. First, there is the translation of server messages sent to connecting clients. This issue has been punted for now. Second there is the translation of the client and its libraries.

The gettext package provides services for translating messages. It uses the xgettext tool to extract strings from the sources for translation. This works by extracting the arguments of the _() and N_() macros. The former is used in context where function calls are allowed (typically anything except static initializers). The latter is used whenever _() isn't. Strings marked with N_() need to be passed to gettext translation routines whenever referenced in the code. For an example, look at how the header and footer are handled in subversion/svn/help-cmd.c.

When using direct calls to gettext routines (*gettext or *dgettext), keep in mind that most of Subversion code is library code. Therefore the default domain is not necessarily Subversion's own domain. In library code you should use the dgettext versions of the gettext functions. The domain name is defined in the PACKAGE_NAME define.

All required setup for localization is controlled by the ENABLE_NLS conditional in svn_private_config.h (for *nix) and svn_private_config.hw (for Windows). Be sure to put

   #include "svn_private_config.h"

as the last include in any file which requires localization.

Also note that return values of _() and *gettext() calls are UTF-8 encoded; this means that they should be translated to the current locale being written as any form of program output.

The GNU gettext manual (http://www.gnu.org/software/gettext/manual/html_node/gettext_toc.html) provides additional information on writing translatable programs in its section "Preparing Program Sources". Its hints mainly apply to string composition.