Derby Functional Tests

Package: org.apache.derbyTesting

created by myrna@golux.com
last updated on 12/02/2004 by: myrna@golux.com

• 1. Introduction
• 2. Quickstart
• 2.1 running tests
  
• 2.2 building derbyTesting package
  
• 3. More details on running the derby functional tests
• 3.1 Running 1 test
• 3.2 Running suites of tests
• 4. Harness internals for developers
• 4.1 Test types
• 4.2 Supporting files for tests
• 4.3 <testname>_app.properties
• 4.4 <testname>_derby.properties
• 4.5 tmp files, out files, master files, and canons
• 4.6 Masking and comparing
• 4.7 Adding a new test
• 4.8 Suites and adding a new suite
• 4.9 Running with a new jvm
• 4.10 Skipping a test
• 4.11 Frameworks
• 4.12 Some test harness properties

1. Introduction

This document describes functionality of the derby functional testing package org.apache.derbyTesting. This package is based on the functional tests in use at IBM for testing the Cloudscape product before its contribution to ASF.

In the following, instructions are geared towards a unix environment. For other environments, some details may need to be adjusted. For instance, the document may refer to $ANT_HOME, for DOS, this would be %ANT_HOME%.

In the following the top directory under which the subversion tree is placed is referred to as ${derby.source} - see also the derby BUILDING.txt.

The version of the classes and supporting files of the derbyTesting package have to match the version of the classes of the derby package. Thus you either need to build all jars yourself, or get all jar files from the incubator site at the same time when available.

2. QuickStart

2.1 running tests

The derbyTesting package enables you to run 1 test or a suite of tests. Before you can run, you need to setup your environment:

• Obtain a jdk or jre (based on jdk 1.3.1 specification or higher). Add the bin directory to your $PATH. Currently supported are:
  

• set $CLASSPATH to include the following jars:
• jakarta-oro-2.0.8.jar
    oromatcher, obtain from http://jakarta.apache.org/oro/index.html, or follow this link for zip file, or tar.gz)
• derbyTesting.jar
  
     test files and classes
• derby.jar
  
    main derby package classes
• derbytools.jar
  
    derby tools classes for tools like ij and dblook
• derbynet.jar
  
    derby network server classes
• db2jcc.jar and db2jcc_license_c.jar
  
    IBM Universal JDBC Driver classes. (See IBM developerworks for download)

• derbyLocale_*.jar
    locale files holding translated messages.

For example:

To run 1 test:

A successful run will have a .pass file, and the output to the console will show no difference between expected and actual test result. A failed test run will have at least a .fail file and the output to the console will show the difference between expected and actual result.

To run a suite:

Each suite run should be started in a clean directory. The test output directory will not be emptied out before testing is begun, although individual test files and result files will be cleaned out and overwritten. 

The suites provided are:

• derbylang:
  
• basic functionality of  language implementation in derby.
  
• Mostly .sql type tests.
  
• tested on a variety of hardware takes from 1.15m to 2.00 hours
  
• derbynetmats
• basic network server tests.
• variety of tests, including some from derbylang suite
• tested on a variety of hardware takes from 15 to 30 minutes
• storeall
• tests for storage area
• includes:
• storemats: most basic quick verification tests.
  
• storemore: more extensive storage tests
• storetests: set of store tests grouped together because they do not each need to create a new database
• tested on a variety of hardware takes from 10 to 25 minutes
  
• xa
• tests the xa implementation. There is both a storage and language element to these tests
• tested on a variety of hardware takes from 2 to 4 minutes
  
• derbytools
• currently only test included is for the dblook utility.
  
• derbyall
• contains all suites typically run by all developers
  
• tested on a variety of hardware takes from 1.50 - 4.00 hours
• See Note1
  

A successful run with all tests passing will have no *.fail files created, the <testsuite>_fail.txt file will be empty, and the <testsuite>_report.txt file will show no failures in the Summary results section.

2.2 building derbyTesting package

To build the derbyTesting package:

• follow all the steps in the derby BUILDING.txt.

This is some typical output for the ant build process.

Once you have built the derbyTesting package built, you can make a derbyTesting.jar use the jar build target at the ${derby.source}level.

This will look something like:

3. More details on running the derby functional tests

The functional tests are run using a class called 'RunTest'. This class calls a number of other classes. A group of tests, called a 'suite' is executed using a class called 'RunSuite'.

3.1 Running 1 test

See section 2.1 for the basic steps to run 1 test.

To pass on system level properties to the test harness, use the test harness property -DtestSpecialFlags. For example, to run a test forcing the message text to be retrieved from the network server:

Tests will be executed in the current directory. When running a test using the network server, i.e. -Dframework=DerbyNet, the test will run in a subdirectory (automatically created) 'DerbyNet'.
See Note2.

The test will normally create the following:

• a database. The default name is 'wombat'. However, the name may be different depending on certain properties passed in to the test harness.
• a .out file: the final result file
• a .tmp file; the initial result file, before any modification to prevent irrelevant differences has been applied (before 'masking').
• a .diff file; the differences between the .out and the master file with expected output it is compared to.
• a .pass or .fail file. This file lists the test if it passes under .pass, and under .fail if the output in .out is different from the expected output in the master.

possibly created:

• additional files used in a specific test may get copied over to the test directory. These normally do not get cleaned up.
• .tmpstr file is created for network server tests and is a possibly massaged copy of the master file the output needs to be compared with.
• .err and .out files in network server database files for any additional error output.

When the test is successful, cleanup will occur unless the test harness property -Dkeepfiles=true is used. Cleanup will attempt to cleanup all files except for .pass.
See Note3.

A successful run (this example is from a dos environment) would look for instance like:

A Test Failure shows the diff, creates a .fail file, does not create a .pass file, and does not cleanup any files upon completion. The output might look like this:

3.2 Running a suite of tests

See section 2.1 for a basic explanation on how to run a suite of tests.

Tests will be run in a subdirectory with the name of the test suite under the current directory. Eg. for derbylang suite, a directory derbylang will be created. While the tests are run, information about the run is inserted into a <testsuite>.sum file. When all tests have completed summary files are created <testsuite>_pass.txt, _fail.txt, and _diff.txt files are created as well as a <testsuite>_report.txt with additional details. Some of the information is duplicate. Also, a .skip file will be created holding a list of the tests that were skipped (for more details on this, see the section on skipping tests).

RunSuite does not empty the top level directory before running. Thus, if another suite was run in the same directory at an earlier time, the resulting summary files might contain results for more than the current run. Therefore it is important to run each suite in a clean directory.

Sample output from RunSuite:

This output does not show whether the tests passed or failed. The Summary section in <testsuite>_report.txt shows the statistics of the passed vs. failed tests, the summary <testsuite>_*.txt files list the tests that passed and failed.

4. Harness internals for developers

The following is intended for people who have the subversion tree available and want to add or modify tests.

The test harness executing one test basically does the following in sequence:

• identify test to run
• identify properties to run with
• copy needed support files
• find the expected output
• if network server, start network server
• run the test, creating the database
• if network server, shutdown the server
• modify the output based on Sed class and _sed.properties file for the test
• compare expected output with actual output
• if pass, cleanup.

4.1 Test types

The test harness recognizes, or will recognize tests with the following extensions:

•  .java    tests that run in a separate jvm.
•  .sql     tests that run using ij
•  .sql2     related to .sql
•  .multi     multi threaded tests. There are no multi threaded tests in the first contribution to apache, and the sections pertaining to such tests have been commented out
•  .unit     unit tests. Currently there are no hooks for unit tests in the harness and no unit tests are contributed initially.

4.2 Supporting files for tests

Various additional files may be used by a test, for instance, to create large data values, to test using of jar files and the like. Any files that need to be accessed by a particular test that are not accessed from the classpath need to be listed under supportfiles= in the <testname>_app.properties file.
Tests can refer to classes without being in the classpath, and sql tests can use the ij command 'run resource ' to execute additional .sql files without changes to the _app.properties files.

For example, in the file (org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:
    supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar

4.3 <testname>_app.properties

Every test directory has a default_app.properties. This file is for system level properties generic to all the tests in that test directory.

If a test requires different system level properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _app.properties

For example, for the test tools/dblook_test.java, there is a properties file called tools/dblook_test_app.properties

4.4 <testname>_derby.properties

Every test directory has a default_derby.properties. This file is for derby specific properties common to all the tests in that test directory.
If a test requires different derby properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _derby.properties

4.5 tmp files, out files, master files, and canons

The test's output will be put into a file testname.tmp. Then the output is modified if masking is required and the result is put into a .out file.
The expected output is found by examining the following directories, based on certain input

• functionTests/master/framework/jcc_version/jvmcode
• functionTests/master/framework/jcc_version/earlier_jvmcode
• functionTests/master/framework/jcc_version
• functionTests/master/framework/jvmcode
• functionTests/master/framework/earlier_jvmcode
• functionTests/master/jvmcode
• functionTests/master

For example, if we are running a test and the flag -Dframework=DerbyNet is used and the jvm we are using is Sun's jdk 142, and the jcc version is 2.4 (not available at time of writing) then the search for the master to compare with starts in the functionTests/derbynet/jcc2.4/jdk14 directory. If a .out file with the same name as the test is found in that directory, that master is taken. If there is no such file in that directory, search continues in the directory functionTests/derbynet/jcc2.4/jdk13 if it exists.

If there is no file there, nor for any other jcc directory, it will continue to derbynet/jdk14, and the search is continued for earlier jvm versions.
If we are not running network server, the DerbyNet and jcc_version directories are not traversed.

The version details do not go into the subversion level, i.e. running with jdk141 or jdk142 is expected to have the same behavior.

This functionality supports dealing with minor differences in behavior caused by minor differences in behavior in the underlying jvms, jcc versions, differences between results returned through network server vs. embedded and minor differences between a debug and non debug (jar) build.

However, having a large number of these files means a maintenance problem. Every time test output changes due to modifications to derby or to the test, all output files in all directories need to be updated accordingly. If at all possible, irrelevant differences should be masked out, or the test should be written so that the output does not reflect such items.

Suggestions to minimize canons:

• create test specific masking
• ensure test data has a specific correct returned order; or an order by should be added to a query
• when writing java tests, ensure only pertinent output is reflected.

4.6 Masking and comparing

Tests often fail because of unimportant differences, such as process ids, statement ids, timestamps. The derby functional test harness provides for masking of these differences at 2 levels:

1. overall level. Masking required in all, or many tests can be achieved using the class Sed in the test harness directory. This class can either delete a reference present in the .tmp file from the .out file, or replace it with a generic string.
2. test specific level. To make masking for only one test, a (testname)_sed.properties file can be created which allows to either remove a string from the output or to replace it.

The diff is executed between the final resulting output and the master file found.

4.7  Adding a new test

To add a new test:

• create the test file (e.g. newfunctest.java or newfunctest.sql) in the appropriate tests subdirectory
• list any files needed that are not .sql or .java files in a supportfiles entry in a test specific _app.properties file. e.g. newfunctest_app.properties:  supportfiles=xyz.jar
  
• list any specific derby properties in a test specific _derby.properties file.
• add the properties files to the copyfiles.ant file in the test specific directory
• run the test. The first time around, the test will fail because no master file will be found.
• if the output is correct, copy it to the master directory. Note that there is no copyfiles.ant file needed for the master directory, all .out files are automatically copied.
• run the test again. Investigate if any differences need to be masked out using a test specific sed.properties file (e.g. newfunctest_sed.properties). If so, ensure this is added to copyfiles.ant.
• add the test to a specific suites/*.xml file, maintaining proper xml syntax.
• run the suite, and correct any problems found.

4.8 Suites and Adding a new suite

A suite constitutes of a <suitename>.properties file and/or a <suitename>.runall file in the org/apache/derbyTesting/functionTests/suites directory. The .properties files hold references to other .properties files, or .runall files, the .runall files are the actual lists of tests.

The lowest level suite always needs to have a .runall file.

For example, the derbyall suite is only a derbyall.properties file that refers to other suites in the 'suites' property:

The derbylang suite is only a derbylang.runall, which lists the tests. The derbynetmats suite has both a .runall and a .properties file, so some additional properties can be specified that are true for all tests in that suite.

To add a suite, you need to create at least a <suite>.runall file, which lists the actual tests, or a properties file that refers to other suites that do have a .runall file. The suite should be added into the directory ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.

4.9 Running with a new jvm

Currently, the supported jvms are:

The classes above are subclasses of org.apache.derbyTesting.functionTests.harness.jvm. The name at the front is just a convention.

To run a test with a jvm that does not have a matching class under org.apache.derbyTesting.functionTests.harness, do the following:

• just run the tests as if there is a jvm class. The harness will default to using the jdk14 class. Unlikely, but possibly there are no differences
  
• if there are failures showing that cannot be explained any other way but genuine, acceptable jvm differences, do the following:
• create a subclass of org.apache.derbyTesting.functionTests.harness.jvm. In this class, specify any jvm specific property settings required
• compile the new jvm class and run the tests
• create a new canon directory for any additional canons that need to be created.
• in rare occasions, other harness changes may be required
• for any tests that should not run with this environment, add a line in the testname_app.properties file indicating this. For instance to add a line for a jvm called jdk29, it would be like this: runwithjdk29=false. Note that the versioning does not currently extend past 2 digits.
• Add code in RunTest.java to switch to the new jvm based on values for system and vendor properties

4.10 Skipping a test

Some tests are written to test specific functionality only available with for instance certain jvms, or, with network server, certain versions of the IBM Universal Driver. To control this, properties can be set for each test, for instance, if a test should not be run when using an ibm jvm, set runwithibmjvm=false. If a test should be run with Sun Hotspot jvm version 14, then set runwithjdk14=true.
The skip setting does not go into the subversion level, i.e. setting runwithjdk141=false has no effect, and setting runwithjdk14 affects runs with jdk141 as well as jdk142.
Other skip reasons are encryption protocols specific to a certain jvm.

The property for skipping a test based on the version of the IBM Universal Driver is "excludeJCC".  The keywords "at-or-before" and "at-or-after" can be used to specify which range of JCC versions should be excluded.  If neither of these keywords is provided, the default is "at-or-before".  For example:

You can also specify an (optional) jvm clause to further tune the exclusion criteria.  This clause starts with the ",when" tag and is followed by a three-part jvm version.  In this case, a test will only be skipped if BOTH the JCC clause AND the jvm clause are true. For example:

To skip a test when running with any version of the IBM Universal Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:
excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1

4.11 Frameworks

Currently, the only framework used is DerbyNet for network server.
Setting framework property setting will invoke the test harness class NetServer, which currently has configuration for connecting to DB2 via jcc (the IBM Universal Driver), and via the older db2 driver. But there are currently no tests to exercise these settings.
Setting this framework also causes the search for expected output to include DerbyNet and jcc version specific subdirectories under master.

4.12 Some test harness properties

For a complete set, refer to comments in RunTest.java, but here are some valuable test properties which can be passed to the RunTest class:

Notes