============================================================================== BTI Design Document ============================================================================== :Title: BTI Design Document - a Guide for BTI Developers :Author: Alexander Kleymenov :Date: 2007/07/05 This document contains detailed description of each of the functional parts of Harmony BTI. It could be used as a *Guide for BTI Developers*. For quick start with BTI please look at `README`_ file. Architectural parts of BTI covered in this document can be summarized as follows: :`Execution Core`_: Maintains BTI configuration and Tests execution. Acts as a glue for other components of BTI. Provides some auxiliary functionality for implementation of test suites adaptors. :`Test Suite Adaptors`_: The Ant scripts integrating the test suites to BTI by means of implementation *adapting* interface. Consist of ``adaptor.xml`` file maintaining test execution and ``parameters.xml`` file describing configuration parameters. :`CTE Module`_: Performs Continuous Test Execution. Is built on the base of CruiseControl system. :`Execution Status Notifier`_: Sends mail alerts with the status of test execution. :`External Dependencies Fetching Module`_: Provides the unified means for description of external dependencies used by test suites or BTI. Downloads and installs the external libraries. Allows sharing of external libraries between test suites. Execution Core ============== Implemented by the following source files:: ./build.xml ./buildtest.bat ./buildtest.sh ./scripts/exec-adaptor.xml ./scripts/framework.xml ./scripts/main.xml This part of BTI is responsible for BTI configuration and execution. It acts as an execution engine gluing all other functional parts of BTI together. It also provides some functionality supporting the development of Test Suite Adaptors. BTI Configuration ----------------- The BTI configuration is made by means of the property files. The first properties file:: ./scripts/framework.properties contains the properties affecting internals of BTI and *SHOULD NOT* be modified by user. The second file:: ./scripts/parameters.xml is an XML Ant properties file containing the configuration parameters for all of the functional parts of BTI. These parameters can be configured by user through the third configuration file:: ./framework.local.properties By means of this file user can also tune some test suites' specific parameters defined in:: ./adaptors/*/parameters.xml files. The last configuration file is:: ./required-parameters.properties It is generated by BTI and contains the assignments of parameters *required* by test suites selected for execution (more on this in `Test Suite Adaptors`_ section). BTI Execution ------------- BTI execution is launched by means of a command line UI provided by the Core. Currently this UI consists of the following Ant targets expected from user as parameters to BTI command line scripts (``buildtest.bat`` for MS Windows[x]_, and ``buildtest.sh`` for Linux): ``install`` Does an installation of BTI on local workstation from remote SVN repository. Checks out (for initial execution) or updates (for consequent ones) BTI implementation files (``./scripts`` directory) and test suite adaptors (``./adaptors`` directory). This target *MUST* be run at first after initial BTI checkout (see ``./README.txt`` for how to get BTI from SVN repository). Generates ``./framework.local.properties`` property file for local BTI configuration. ``framework-update`` Does an update of current BTI installation from the SVN repository. Updates BTI implementation scripts and test suite adaptors. ``setup`` Does a setup of the test run. Expects the ``test.suites`` property to be set to comma separated list of test suites selected for execution. Produces 3 files: ``./required-parameters.properties`` property file containing the assignments of parameters required by selected test suites. ``./selected-suites.properties`` property file containing the test run configuration. Describes the order in which the selected suites will be run, and dependencies between them. ``./resources.location`` property file describing the locations of external resources needed by configured test run. ``run`` Launches test execution. The selected test suites will be executed in order defined by ``test.suites.names.list`` property in ``./selected-suites.properties`` file. At first, BTI executes ``setup`` targets of adaptor scripts of all of the selected test suites (in defined order). If ``setup`` fails for some of the suite, this suite won't be ``run``, and all dependent test suites won't be ``setup`` and ``run``. BTI will warn about all of the failed and skipped test suites by means of its log messages. At second, after successful ``setup``, BTI launches ``run`` targets of selected test suites. Similarly to ``setup`` BTI will skip the execution of dependent test suites if their superior test suites failed to ``run``. The information about failed and skipped test suites will be printed out by BTI to its log. After all runs of the selected test suites BTI ends with *SUCCESS* if there were no test suites failures, and *ERROR* if some suites could not be set up or failed to run. ``run-cc`` Does the same as ``run``, but test cycle is continuously repeated. Each iteration of test suites execution is triggered whether by modifications happened in some of the controlled by test suites SVN repositories, or by schedule provided by user. See `CTE Module`_ for more information on Continuous Test Execution. ``clean`` Consequently launches ``clean`` targets of adaptors of selected test suites. ``test-mail`` Sends test mail notification. Used to check the operability of the current configurations of `Execution Status Notifier`_ module. The execution of BTI can be configured by the following properties specified in ``./framework.local.properties`` file:: framework.parameters.optional.dolog framework.parameters.repository.noupdate framework.parameters.repository.awaited For descriptions of these parameters look in ``./scripts/parameters.xml`` file. Test Suite Adaptors =================== A Test Suite Adaptor consists of two Ant files placed under ``./adaptors/${suite.name}`` directory: adaptor.xml ----------- Regular Ant project file implementing 3 targets: ``setup``, ``run``, and ``clean``: **setup** has to perform all the works on setting of the test suite on the local workstation: it should download and build the sources for test suite, check required prerequisites, and so on - everything that should be done before first run of the test suite on the workstation. `Execution Core`_ provides some functionality which can be reused here: ```` - macrodefinition checking out the SVN repository located at ``${url}`` to directory pointed by ``${trunk}``. ```` - macrodefinition updating the local ``${path}`` of BTI repository. It is useful when the sources of the test suite are located under BTI's SVN repository. **run** performs the actual run of the test suite. It should "````" if test suite's execution status is FAILED. **clean** cleans the products of test suite's execution The following properties provided by `Execution Core`_ can be useful for implementing these targets: :``suite.name``: holding the name of the suite :``suite.adaptor.dir``: the directory where adaptor is placed :``test.sources.dir``: the root directory containing the test sources. Usually the test sources for each particular test suite are placed under ``${test.sources.dir}/${suite.name}`` directory. :``temp.files.dir``: the directory for temporal files :``checkouts.dir``: the directory for external checkouts. For example *classlib* test suite checks out its sources to ``${checkouts.dir}/classlib`` directory. :``results.dir``: the BTI's directory for storing the test execution results. Usually test suite uses ``${results.dir}/${suite.name}`` directory for this purpose. :``classes.dir``: the directory for compiled java classes. It's needed if the test suite adaptor uses some java classes which should be compiled. parameters.xml -------------- Ant's xml property file containing declarations of parameters describing the test suite. This xml property file is loaded after ``adaptor.xml`` file and, thus, can contain references to the properties defined there. The parameters described in parameters.xml file can be divided into two categories: *execution* and *integration* parameters. Execution Parameters ~~~~~~~~~~~~~~~~~~~~ are the parameters affecting the test suite execution. These parameters includes: *depends* +++++++++ contains comma separated list of all of the test suites on which this test suite depends. Actually this parameter defines the order in which the test suite will be executed: it will be executed after suites defined by this parameter. If some of the listed test suites were not selected for execution, they are removed from this parameter. If default value of this parameter does not correspond to selected set of test suites, User *SHOULD* change this parameter by means of ``${suite.name}.parameters.depends`` property specified in ``./framework.local.properties`` file. *shared* parameters +++++++++++++++++++ are the parameters holding the values which can be used by other test suites. Such parameters affecting the execution of other test suites. The path on built product (as JVM for example) can be an example of such parameter. *required* parameters +++++++++++++++++++++ are the parameters for which the test suite is not able to provide the default values: the values for these parameters can be specified only from the outside of the test suite. For example: such parameters can hold the path to tested JVM - test suite can't guess this information, so it can't provide the default value for this parameter. On the BTI's ``setup`` stage, all the required parameters for the selected test suites are collected in ``./required-parameters.properties`` file and if some of such parameters has no value, BTI stops with *ERROR* message. **Note:** the declaration of *required* parameter should have a ``description`` attribute. Without this attribute such a parameter won't be regarded as *required*. While test suite itself can not provide the default value for such a parameter, it is possible to provide it from BTI's (read "external") side. It is done by means of connecting *required* parameter of test suite with *shared* parameter of test suite which can provide *required* value. Such test suites providing their *shared* parameters should be listed in ``depends`` parameter of dependent test suite - it will allow them to be executed before dependent test suite, and, correspondingly, to build a product pointed by the value of *shared* parameter before dependent test suite will use it. The values for *required* parameters can be specified in three places: * test suites ``./parameters.xml`` file can contain only connections between *required* parameters and *shared* parameters of some other test suites. These test suites should be listed in ``depends`` parameter. **Note:** if these test suites are not selected for execution, these *shared* parameters won't be resolved and *required* parameter won't have the value. BTI will stop with ERROR message requesting to specify the value for this unset *required* parameters. * ``./framework.local.properties`` file can contain connections between the *required* and *shared* parameters, as well as direct evaluation of *required* parameters with some value. The value provided in this file takes precedence over the value in ``./parameters.xml`` file. * ``./required-parameters.properties`` file can contain only direct evaluation of required parameter with some value. It can't contain any property resolutions. The value provided in this file takes precedence on the values provided in ``./framework.local.properties`` and ``./parameters.xml`` files. **Note:** the *required* parameters in this file appears in short format: ``${suite.mane}.${required.parameter.name}``. For example, if *required* parameter has been defined in ``parameters.xml`` as ``suite.parameters.required.some.param``, in ``./required-parameters.properties`` it will be in the following form: ``suite.some.param``. *optional* parameters +++++++++++++++++++++ are the parameters allowing to user to influence how the test suite will be executed. Such parameters have some default values and should have the description. Integration Parameters ~~~~~~~~~~~~~~~~~~~~~~ describe the support from BTI's side needed for correct test suite execution. These parameters includes: *external* parameters +++++++++++++++++++++ are the parameters describing the dependencies of the test suite on external resources located at some point of the web. This parameters are addressed to the `External Dependencies Fetching Module`_ which is responsible for downloading and sharing them between the test suites. *cc* parameters +++++++++++++++ are the parameters describing test suite's specific of continuous execution. These parameters describe files and SVN repositories monitored by test suite, define bootstrapping and publishing procedures, do some test suite's specific configurations of `Execution Status Notifier`_ module. For more information see `CTE Module`_ section. *mail* parameters +++++++++++++++++ are the parameters describing the test suite's specific configuration of `Execution Status Notifier`_ module. Initially these parameters were *cc* parameters, but now they have been moved into special category. The parameters specified here take precedence over corresponding *cc* ones. This parameters allows to specify the summary information to be sent with mail notifications, files to be attached, and so on. See `Execution Status Notifier`_ for more details. CTE Module ========== Continuous Test Execution module consists of the following source files:: ./scripts/cc-project.xml ./scripts/cc-exec-adaptor.xml ./scripts/templates/cc-config-template.xml ./scripts/templates/cc-project-template.xml This module is responsible for performing of continuous test execution launched by ``run-cc`` BTI's target. The work of this module can be divided into 2 steps: `preparation`_ and `execution`_. Preparation ----------- On this stage, CTE Module gathers the parameters describing CruiseControl-related configuration and generates CC project files (CC configurations) for each of selected test suites. The parameters defining these configurations are provided by two sources: - BTI defines *global* CTE parameters common for all test suites. Such parameters include the information about web ports, timeouts, etc. and can be (along with their descriptions) seen in ``./scripts/parameters.xml`` file under ```` section. **Note:** These parameters are *user configurable*. It means that their values can be changed by user through the corresponding properties specified in ``./framework.local.properties`` file. - Each of selected test suites uses its own set of CTE parameters describing the specific of its continuous execution. These parameters appear in test suites' ``parameters.xml`` file under ```` section: :``trigger``: - (no value by default) - path to local file which modification will trigger the execution of test suite. :``usesvn``: - (no value by default) - the URL of SVN repository which modification will trigger the test suite's execution. There can several URLs be set. Every other parameter with ``usesvn.`` prefix is considered as URL for SVN repository. :``publish``: - (default is ``true``) - determines whether do status notification publishing for this test suite or not. The value of this parameter can be changed by user through the ``./framework.local.properties`` configuration file. In future this parameter will go to ```` section of parameters configuration file (so it won't be a CTE configuration parameter). :``results``: - (no value by default) - path to directory containing the test execution result files in XML JUnit format for status notification message. This parameter will go to ```` section. The `Execution Status Notifier`_ configuration parameters may also appear in ```` section. But their normal place has been moved to ```` section. They are: ``summary``, ``message``, and ``attachs``. They describe summary, content of the notification message and files to be attached with the message correspondingly. **Note:** These parameters (except ``publish``) are `Integration Parameters`_ and are *not user configurable*. It means that their values should not be changed by user. Execution --------- After ``setup`` of each of the selected test suites, BTI launches CruiseControl with generated configuration. CC launches test suites in order described by ``test.suites.names.list`` property defined in ``./selected-suites.properties`` file generated on ``setup`` stage. If some of the test suites failed, dependent test suites won't be run. The execution log of each of the test suites can be found in CC's web interface under *build artifacts* section. Web interface of CC is available at localhost with ``framework.parameters.cc.webport`` port number which is 8080 by default. As stated in `README`_ file, there are two modes of continuous execution: SVN modifications triggered and scheduled runs. SVN modifications triggered runs are executed each time monitored SVN of trigger file modifications occur. Scheduled runs are launched at the particular time specified by the following properties in framework.local.properties file: ``framework.parameters.schedule.day=Tuesday`` Do launch test suites execution on particular day. If this property is not defined, the test run will be launched everyday. ``framework.parameters.schedule.time=1530`` Do launch test suites execution at this particular time. Should be defined for scheduled execution mode. Execution Status Notifier ========================= Implemented by the following source file:: ./scripts/publisher.xml This module performs the composition and sending of the message with test suite execution status. The work of this module is regulated by means of particular configuration parameters defined either by BTI or by test suites. The parameters defined by BTI have the global influence and take effect for all of the suites. Such the parameters are (look in ``./scripts/parameters.xml`` file for more info): :``framework.parameters.usemail``: the notification publishing will be in use only if this parameter is set and contains the address of SMTP server to be used for mailing. :``framework.parameters.usemail.always``: defines the notification mode. If it holds ``true`` value - the notifications will be sent regardless of execution status. But if it holds ``false`` - notifications will be sent only after execution status changes (from *PASSED* to *FAILED*, and vice versa) :``framework.parameters.usemail.to``: should contain the address to where notifications should be sent. Use comma for multiple values. :``framework.parameters.usemail.from``: should contain the address from where notifications should be sent Use comma for multiple values. :``framework.parameters.usemail.suppress.exelogs``: controls whether execution logs will be sent with failure messages or not. :``framework.parameters.usemail.subject.prefix``: sets up the prefix for notifications subject line. Also each of the test suite can provide its own parameters configuring the status notifications. They are described in test suite's parameters.xml file under section and their values should not be changed by user: :``summary``: the path to file holding test suite's execution summary. This content will be prepended to the notification message :``message``: the path to file holding test suite's message content. This content will replace one generated by default. :``attachs``: comma separated paths to files to be attached with notification. Sometimes it is necessary for test suite to override some global parameter defined by framework's parameters. It can be done by means of parameters specified in ``./framework.local.properties`` file having ``${suite.name}.parameters.mail`` prefix instead of ``framework.parameters.usemail`` one. For example, if the User needs for *test_suite* to use particular destination mail address, the following property can do that: ``test_suite.parameters.mail.to=different@address.info`` In this case status notifications for all of the test suites will be sent to the address specified by ``framework.parameters.usemail.to`` parameter, but notifications for *test_suite* will be sent to *different@address.info*. External Dependencies Fetching Module ===================================== With this module BTI offers the common way to fetch, use and share external libraries between the test suites. It is done by means of parameters described in ```` section of ``parameters.xml`` files. The description of the external library should contain at least 2 fields: URL from where to take the dependency, and MD5 value to check the dependency. Also there can be some attributes describing the type of the dependency and some installation information. BTI on ``setup`` stage fetches all the dependencies according to their descriptions and sets up corresponding properties to access them. For example: let's look at the following description:: It describes dependency called *my-dependency* which is located at specified URL and has particular MD5 check sum. The dependency will be downloaded only when local operation system is linux. During the local installation of downloaded archive only jar files from `lib` directory will be unzipped and placed under `my-dep-dir` directory. The path to this directory will be provided by ``ext.my-dependency.location`` property.The `if` is an optional parameters. There is no special requirement when it isn't set. This module is under development and provides only simple functionality. For working usage examples, please, look at ``./scripts/parameters.xml`` file. ------------------------------------------------------------------------------ .. [x] Other brands and names are the property of their respective owners .. _README: ./README.txt .. Use "Vim reStructured Text" utility to generate HTML: http://skawina.eu.org/mikolaj/vst.html or online renderer: http://www.hosting4u.cz/jbar/rest/rest.html .. vim:filetype=rst