Revision 1886490

In the update editor, stream data to both pristine and working files.

Previously, the working files were installed in a separate step.  That step
happened per-directory and involved copying and possibly translating the
pristine contents into new working files.  For transports like HTTP having
a separate step with nothing being read from the connection puts an implicit
limit on the size of the directory that can be checked out without hitting
a timeout.

Assuming the default HTTP timeout = 60 seconds of httpd 2.4.x and a relatively
fast disk, the limit was around 6 GB for any directory.

If we stream data to both pristine and the (projected) working file, the
actual install can happen as an atomic rename.  Then we don't have to stop
reading from the connection, and the timeouts should no longer be an issue.

The new approach also has several nice properties, such as not having to
re-read the pristine files, not interfering with the network-level buffering,
TCP slow starts, and etc.  The implementation of the new approach gives up to
25% reduction in the amount of write I/O during checkouts.  On Windows, it
uses the existing mechanism of installing the files by handle, without having
to reopen them.

Several quick benchmarks:

  - Checking out subversion/trunk over https://

    Total time:  3.861 s  →  2.812 s
       Read IO:  57322 KB  →  316 KB
      Write IO:  455013 KB  →  359977 KB

  - Checking out 4 large binary files (7.4 GB) over https://

    Total time:   91.594 s   →  70.214 s
       Read IO:   7798883 KB  →  19 KB
      Write IO:   15598167 KB  →  15598005 KB


From the implementation standpoint, we factor out a utility layer that can
prepare and install working files, svn_wc__working_file_writer_t, and use it
in the part of the workqueue that handles OP_FILE_INSTALL.  We then rework
the update editor to make it stream the data to the working file during
apply_textdelta(), by using the introduced file writer utility layer.

* subversion/include/private/svn_io_private.h
  (svn_stream__install_stream_set_read_only,
   svn_stream__install_stream_set_executable,
   svn_stream__install_stream_set_affected_time,
   svn_io__win_set_file_basic_info): Declare.

* subversion/libsvn_subr/io.c
  (FILE_BASIC_INFO, FileBasicInfo): Declare these Win32 types for older SDKs.
  (APR_DELTA_EPOCH_IN_USEC): Unless defined, declare a copy of the constant
   from apr/arch/win32/apr_arch_atime.h.
  (svn_io__win_set_file_basic_info): New function that sets the basic file
   information using an existing file handle.

* subversion/libsvn_subr/stream.c
  (install_baton_t): Add new fields to remember if the target of the install
   stream should have specific mtime, read-only attribute or the executable
   bit.
  (svn_stream__create_for_install): Initialize the new fields to their
   default values.
  (svn_stream__install_stream_set_read_only,
   svn_stream__install_stream_set_executable,
   svn_stream__install_stream_set_affected_time): Update the new fields.
  (svn_stream__install_stream): Adjust the expected mtime and attributes of
   the to-be-installed file, if necessary.
  (svn_stream__install_get_info): If we are to set a specific mtime on the
   target, return it.  Otherwise, stat the file as before.

* subversion/libsvn_wc/working_file_writer.h: New file.
  (svn_wc__working_file_writer_t,
   svn_wc__working_file_writer_open,
   svn_wc__working_file_writer_get_stream,
   svn_wc__working_file_writer_get_info,
   svn_wc__working_file_writer_install,
   svn_wc__working_file_writer_close): Declare the new utility layer.

* subversion/libsvn_wc/working_file_writer.c: New file.
  (svn_wc__working_file_writer_t,
   cleanup_file_writer,
   svn_wc__working_file_writer_open,
   svn_wc__working_file_writer_get_stream,
   svn_wc__working_file_writer_get_info,
   svn_wc__working_file_writer_install,
   svn_wc__working_file_writer_close): Implement the new utility layer.
   Make use of the svn_stream__install_stream's new functionality.

* subversion/libsvn_wc/workqueue.c
  (): Include working_file_writer.h.
  (run_file_install): Reimplement by creating and delegating most of the
   work to the working file writer.

* subversion/libsvn_wc/wc_db.c
  (): Include working_file_writer.h.
  (svn_wc__db_base_add_file): Accept an optional working file writer.
   If it's passed in, this function will atomically update the db and complete
   the installation of the file.  Accept the new "record_fileinfo" parameter.
   If it's set, also record the fileinfo of the installed file.

* subversion/libsvn_wc/wc_db.c
  (db_record_fileinfo): Move this function so that it can be used without
   a forward declaration.
  (install_working_file): New helper.  Completes an installation of the
   working file and optionally records its fileinfo.
  (svn_wc__db_base_add_file): If the working file writer is passed in,
   atomically update the db and complete the installation of the file.
   If the "record_fileinfo" parameter is set, also record the fileinfo
   of the installed file.

* subversion/libsvn_wc/update_editor.c
  (): Include working_file_writer.h.
  (struct file_baton): Add new fields to hold the cached base properties
   of the file and an optional working file writer.
  (window_handler): In case of an error, explicitly close the working file
   writer.
  (get_file_base_props): New helper, mostly factored out from the
   close_file() function.  Returns the, possibly cached, base properties
   for a file context.
  (open_working_file_writer): New helper.  Opens a working file writer
   according to the properties in the specified file context.
  (lazy_open_target): If necessary, open the working file writer.
   Configure the incoming data to be written to both the pristine and
   the provisioned working file.  Introduce the local variable "stream"
   and rename the output argument to "stream_p" for clarity.
  (close_file): If we need to install the working file, ensure that we have
   an appropriate working file writer.  The writer may have been provisioned
   during apply_textdelta(), but also may need to be opened on-the-go.
   Pass the file writer to svn_wc__db_base_add_file() to complete the
   installation of the working file.  Don't schedule OP_FILE_INSTALL
   workqueue items.

* subversion/tests/libsvn_subr/io-test.c
  (test_install_stream_set_read_only,
   test_install_stream_set_affected_time): New tests.
  (test_funcs): Run the new tests.

* subversion/tests/libsvn_wc/db-test.c
  (test_inserting_nodes): Adjust the call to svn_wc__db_base_add_file().

* subversion/tests/cmdline/update_tests.py
  (update_add_missing_local_add): No longer fails.  Add comment.