Derby Functional Tests

Package: org.apache.derbyTesting

created by myrna@golux.com
last updated on 12/02/2004 by: myrna@golux.com


1. Introduction

This document describes functionality of the derby functional testing package org.apache.derbyTesting. This package is based on the functional tests in use at IBM for testing the Cloudscape product before its contribution to ASF.

In the following, instructions are geared towards a unix environment. For other environments, some details may need to be adjusted. For instance, the document may refer to $ANT_HOME, for DOS, this would be %ANT_HOME%.

In the following the top directory under which the subversion tree is placed is referred to as ${derby.source} - see also the derby BUILDING.txt.

The version of the classes and supporting files of the derbyTesting package have to match the version of the classes of the derby package. Thus you either need to build all jars yourself, or get all jar files from the incubator site at the same time when available.

2. QuickStart

2.1 running tests

The derbyTesting package enables you to run 1 test or a suite of tests. Before you can run, you need to setup your environment:

    jdk131 - Sun HotSpot jdk1.3.1
    jdk141 - Sun HotSpot jdk1.4.1
    jdk142 - Sun HotSpot jdk1.4.2
    jdk15 - Sun HotSpot jdk1.5
    ibm131 - IBM Classic jdk1.3.1
    ibm141 - IBM Classic jdk1.4.1
    ibm142 - IBM Classic jdk1.4.2
    j9_13 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition)

For example:

(note that $jardir is only a convenience variable and that the command below has carriage returns for formatting reasons):
set jardir=/local/derbyjar
set CLASSPATH="$jardir/derby.jar:$jardir/derbytools.jar:$jardir/derbynet.jar:$jardir/db2jcc.jar:
$jardir/db2jcc_license_c.jar:$jardir/derbyTesting.jar:/local/derby/tools/java/jakarta-oro-2.0.8.jar:
$jardir/derbyLocale_de_DE.jar:$jardir/derbyLocale_es.jar:$jardir/derbyLocale_fr.jar:
$jardir/derbyLocale_it.jar:$jardir/derbyLocale_ja_JP.jar:$jardir/derbyLocale_ko_KR.jar:
$jardir/derbyLocale_pt_BR.jar:$jardir/derbyLocale_zh_CN.jar:$jardir/derbyLocale_zh_TW.jar:
$CLASSPATH

set PATH=/local/jdk141/bin:$PATH

To run 1 test:

syntax:
    java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunTest <testdir>/<testname>
where
  •    <testproperty> are test specific properties, such as 'framework' for the RunTest class.
  •    <testdir> is one of the directories under functionTests/tests where the actual test is located
  •    <testname> is the actual name of the test
examples:
to run the test supersimple against the embedded driver:
      java org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql

To run a test with network server, add -Dframework=DerbyNet to the run. The test harness will to start network server at port 1527 or connect to a running one, run the test, and stop network server thereafter.
for example:
        java
-Dframework=DerbyNet org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql

A successful run will have a .pass file, and the output to the console will show no difference between expected and actual test result. A failed test run will have at least a .fail file and the output to the console will show the difference between expected and actual result.

To run a suite:

syntax:
  java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunSuite  <testsuite>
where
  •    <testproperty> are test specific properties, such as 'verbose' for the RunSuite class.
  •    <testsuite> is one of the suites under org/apache/derbyTesting/suites
for example for running  the suite derbylang:
   java -Dverbose=true org.apache.derbyTesting.functionTests.harness.RunSuite derbylang

Each suite run should be started in a clean directory. The test output directory will not be emptied out before testing is begun, although individual test files and result files will be cleaned out and overwritten. 

The suites provided are:

A successful run with all tests passing will have no *.fail files created, the <testsuite>_fail.txt file will be empty, and the <testsuite>_report.txt file will show no failures in the Summary results section.

-------snippet from derbylang_report.txt -----
-----------------------------------------------------------
Summary results:

Test Run Started: 2004-11-10 11:27:55.0
Test Run Duration: 00:04:09

129 Tests Run
100% Pass (129 tests passed)
 0% Fail (0 tests failed)
0 Suites skipped


2.2 building derbyTesting package

To build the derbyTesting package:

This is some typical output for the ant build process.

> cd /local/derby/java/testing
> ant.ksh
Searching for build.xml ...
Buildfile: /local/derby/java/testing/build.xml

compile:
    [javac] Compiling 30 source files to /local/derby/classes
...
     [copy] Copying 1 file to /local/derby/classes/org/apache/derbyTesting/funct
ionTests

BUILD SUCCESSFUL
Total time: 10 minutes 3 seconds

Once you have built the derbyTesting package built, you can make a derbyTesting.jar use the jar build target at the ${derby.source}level.

This will look something like:

c:> ant derbytestingjar
Searching for build.xml ...
Buildfile: C:\derby\build.xml

initjars:
    [mkdir] Created dir: C:\derby\jars\
    [mkdir] Created dir: C:\derby\jars\lists
     [echo] Revision number set to exported
     [echo] .

derbytestingjar:
     [echo] Beginning derbytesting.jar build
.....
BUILD SUCCESSFULL


3. More details on running the derby functional tests

The functional tests are run using a class called 'RunTest'. This class calls a number of other classes. A group of tests, called a 'suite' is executed using a class called 'RunSuite'.

3.1 Running 1 test

See section 2.1 for the basic steps to run 1 test.

To pass on system level properties to the test harness, use the test harness property -DtestSpecialFlags. For example, to run a test forcing the message text to be retrieved from the network server:

java -Dframework=DerbyNet -DtestSpecialFlags=RetrieveMessagesFromServerOnGetMessage=true  org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql


Tests will be executed in the current directory. When running a test using the network server, i.e. -Dframework=DerbyNet, the test will run in a subdirectory (automatically created) 'DerbyNet'.
See Note2.

The test will normally create the following:

possibly created:

When the test is successful, cleanup will occur unless the test harness property -Dkeepfiles=true is used. Cleanup will attempt to cleanup all files except for .pass.
See Note3.

A successful run (this example is from a dos environment) would look for instance like:

c:> derbyTesting.functionTests.harness.RunTest lang/supersimple.sql
C:\derby\run2
supersimple
-- listing properties --
derby.locks.deadlockTimeout=3
derby.locks.waitTimeout=3
*** Start: supersimple jdk1.4.2_03 2004-11-10 16:51:02 ***
The test should be running...
MasterFileName = master/supersimple.out
*** End:   supersimple jdk1.4.2_03 2004-11-10 16:51:25 ***

A Test Failure shows the diff, creates a .fail file, does not create a .pass file, and does not cleanup any files upon completion. The output might look like this:

 c:> derbyTesting.functionTests.harness.RunTest lang/supersimple.sql
C:\derby\run2

supersimple
-- listing properties --
derby.locks.deadlockTimeout=3
derby.locks.waitTimeout=3
*** Start: supersimple jdk1.4.2_03 2004-11-10 16:54:39 ***
The test should be running...
MasterFileName = master/supersimple.out
10 del
< 10
10a10
> 1
Test Failed.
*** End:   supersimple jdk1.4.2_03 2004-11-10 16:55:02 ***


3.2 Running a suite of tests

See section 2.1 for a basic explanation on how to run a suite of tests.

Tests will be run in a subdirectory with the name of the test suite under the current directory. Eg. for derbylang suite, a directory derbylang will be created. While the tests are run, information about the run is inserted into a <testsuite>.sum file. When all tests have completed summary files are created <testsuite>_pass.txt, _fail.txt, and _diff.txt files are created as well as a <testsuite>_report.txt with additional details. Some of the information is duplicate. Also, a .skip file will be created holding a list of the tests that were skipped (for more details on this, see the section on skipping tests).

RunSuite does not empty the top level directory before running. Thus, if another suite was run in the same directory at an earlier time, the resulting summary files might contain results for more than the current run. Therefore it is important to run each suite in a clean directory.

Sample output from RunSuite:

c:> $ java org.apache.derbyTesting.functionTests.harness.RunSuite derbylang
Top suite: derbylang
Suite to run: derbylang:derbylang
Now do RunList
Now run the suite's tests
Run the tests...
Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtoprepo
rtdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/altertable.sql
Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtopreportdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/arithmetic.sql
...(.more tests)....
Generated report: derbylang_report.txt

This output does not show whether the tests passed or failed. The Summary section in <testsuite>_report.txt shows the statistics of the passed vs. failed tests, the summary <testsuite>_*.txt files list the tests that passed and failed.


4. Harness internals for developers

The following is intended for people who have the subversion tree available and want to add or modify tests.

The test harness executing one test basically does the following in sequence:


4.1 Test types

The test harness recognizes, or will recognize tests with the following extensions:


4.2 Supporting files for tests

Various additional files may be used by a test, for instance, to create large data values, to test using of jar files and the like. Any files that need to be accessed by a particular test that are not accessed from the classpath need to be listed under supportfiles= in the <testname>_app.properties file.
Tests can refer to classes without being in the classpath, and sql tests can use the ij command 'run resource ' to execute additional .sql files without changes to the _app.properties files.

For example, in the file (org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:
    supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar

4.3 <testname>_app.properties

Every test directory has a default_app.properties. This file is for system level properties generic to all the tests in that test directory.

If a test requires different system level properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _app.properties

For example, for the test tools/dblook_test.java, there is a properties file called tools/dblook_test_app.properties

4.4 <testname>_derby.properties

Every test directory has a default_derby.properties. This file is for derby specific properties common to all the tests in that test directory.
If a test requires different derby properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _derby.properties

4.5 tmp files, out files, master files, and canons

The test's output will be put into a file testname.tmp. Then the output is modified if masking is required and the result is put into a .out file.
The expected output is found by examining the following directories, based on certain input

For example, if we are running a test and the flag -Dframework=DerbyNet is used and the jvm we are using is Sun's jdk 142, and the jcc version is 2.4 (not available at time of writing) then the search for the master to compare with starts in the functionTests/derbynet/jcc2.4/jdk14 directory. If a .out file with the same name as the test is found in that directory, that master is taken. If there is no such file in that directory, search continues in the directory functionTests/derbynet/jcc2.4/jdk13 if it exists.

If there is no file there, nor for any other jcc directory, it will continue to derbynet/jdk14, and the search is continued for earlier jvm versions.
If we are not running network server, the DerbyNet and jcc_version directories are not traversed.

The version details do not go into the subversion level, i.e. running with jdk141 or jdk142 is expected to have the same behavior.

This functionality supports dealing with minor differences in behavior caused by minor differences in behavior in the underlying jvms, jcc versions, differences between results returned through network server vs. embedded and minor differences between a debug and non debug (jar) build.

However, having a large number of these files means a maintenance problem. Every time test output changes due to modifications to derby or to the test, all output files in all directories need to be updated accordingly. If at all possible, irrelevant differences should be masked out, or the test should be written so that the output does not reflect such items.

Suggestions to minimize canons:


4.6 Masking and comparing

Tests often fail because of unimportant differences, such as process ids, statement ids, timestamps. The derby functional test harness provides for masking of these differences at 2 levels:

  1. overall level. Masking required in all, or many tests can be achieved using the class Sed in the test harness directory. This class can either delete a reference present in the .tmp file from the .out file, or replace it with a generic string.
  2. test specific level. To make masking for only one test, a (testname)_sed.properties file can be created which allows to either remove a string from the output or to replace it.

The diff is executed between the final resulting output and the master file found.

4.7  Adding a new test

To add a new test:


4.8 Suites and Adding a new suite

A suite constitutes of a <suitename>.properties file and/or a <suitename>.runall file in the org/apache/derbyTesting/functionTests/suites directory. The .properties files hold references to other .properties files, or .runall files, the .runall files are the actual lists of tests.

The lowest level suite always needs to have a .runall file.

For example, the derbyall suite is only a derbyall.properties file that refers to other suites in the 'suites' property:

-----------------------derbyall.properties---------------
suites=derbylang derbynetmats storeall xa derbytools
derby.debug.true=enableBtreeConsistencyCheck
derby.stream.error.logSeverityLevel=0

The derbylang suite is only a derbylang.runall, which lists the tests. The derbynetmats suite has both a .runall and a .properties file, so some additional properties can be specified that are true for all tests in that suite.

------------------derbynetmats.properties-----------------
framework=DerbyNet
suites=derbynetmats
jdk12test=true
runwithj9=false
timeout=60

To add a suite, you need to create at least a <suite>.runall file, which lists the actual tests, or a properties file that refers to other suites that do have a .runall file. The suite should be added into the directory ${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.

4.9 Running with a new jvm

Currently, the supported jvms are:

    jdk131 - Sun HotSpot jdk1.3.1 - class: jdk13
    jdk141 - Sun HotSpot jdk1.4.1 - class jdk14
    jdk142 - Sun HotSpot jdk1.4.2 - class jdk14
    jdk15 - Sun HotSpot jdk1.5 - class jdk15
    ibm131 - IBM Classic jdk1.3.1  - class ibm13
    ibm141 - IBM Classic jdk1.4.1 - class ibm14
    ibm142 - IBM Classic jdk1.4.1 - class ibm14
    j9_13 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition) - class j9_13

The classes above are subclasses of org.apache.derbyTesting.functionTests.harness.jvm. The name at the front is just a convention.

To run a test with a jvm that does not have a matching class under org.apache.derbyTesting.functionTests.harness, do the following:


4.10 Skipping a test

Some tests are written to test specific functionality only available with for instance certain jvms, or, with network server, certain versions of the IBM Universal Driver. To control this, properties can be set for each test, for instance, if a test should not be run when using an ibm jvm, set runwithibmjvm=false. If a test should be run with Sun Hotspot jvm version 14, then set runwithjdk14=true.
The skip setting does not go into the subversion level, i.e. setting runwithjdk141=false has no effect, and setting runwithjdk14 affects runs with jdk141 as well as jdk142.
Other skip reasons are encryption protocols specific to a certain jvm.

The property for skipping a test based on the version of the IBM Universal Driver is "excludeJCC".  The keywords "at-or-before" and "at-or-after" can be used to specify which range of JCC versions should be excluded.  If neither of these keywords is provided, the default is "at-or-before".  For example:

To skip a test when running with any version of the IBM Universal Driver that is 2.4 or earlier:
excludeJCC=at-or-before:2.4

To skip a test when running with any version of the IBM Universal Driver that is 2.0 or later:
excludeJCC=at-or-after:2.0

You can also specify an (optional) jvm clause to further tune the exclusion criteria.  This clause starts with the ",when" tag and is followed by a three-part jvm version.  In this case, a test will only be skipped if BOTH the JCC clause AND the jvm clause are true. For example:

To skip a test when running with any version of the IBM Universal Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:
excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1

To skip a test when running with any version of the IBM Universal Driver that is 2.0 or earlier, but ONLY if the jvm is 1.5 or later:
excludeJCC=at-or-before:2.0,when-at-or-after:jdk1.5.1

4.11 Frameworks

Currently, the only framework used is DerbyNet for network server.
Setting framework property setting will invoke the test harness class NetServer, which currently has configuration for connecting to DB2 via jcc (the IBM Universal Driver), and via the older db2 driver. But there are currently no tests to exercise these settings.
Setting this framework also causes the search for expected output to include DerbyNet and jcc version specific subdirectories under master.


4.12 Some test harness properties

For a complete set, refer to comments in RunTest.java, but here are some valuable test properties which can be passed to the RunTest class:

runwith<jvm>
          See above section 4.10
framework
    specifies which framework to run with. For example:
        java -Dframework=DerbyNet org.apache.derbyTesting.functionTests.RunTest
lang/supersimple.sql
verbose
    Shows more detailed output. For example:
        java -Dverbose=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql
keepfiles
    Indicates to not clean up any of the files if the test passed.
        java -Dkeepfiles=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql
TestSpecialFlags
    sets additional properties.
         java -D=TestSpecialFlags=derby.infolog.append=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql
excludeJCC
        See above section 4.10





Notes

Note1:
There is one more suite included: the j9derbynetmats suite is a modification of the derbynetmats suite. It is available to test the network server with the jvm available with IBM's WCTME (Workplace Client Technology, Micro Edition; formerly WSDD), and will be run at IBM. Note that the setup for running the j9derbynetmats tests is very specific to the test harness, not even using the WCTME files in their normal location.
The j9derbynetmats suite is included to serve as an example of splitting the network server process to run with a different jvm than the test client. The j9derbynetmats suite will run with another jvm as client (as defined in the suite properties), but start up network server with the 'j9' jvm files (the reference to 'j9' is based on the executable, j9.exe), based on the property 'serverJvm'. Running this suite requires providing the property  bootcp, which is  interpreted from the test harness class j9_13. See also section on adding a new jvm setup.


Note2:
setting RetrieveMessagesFromServerOnGetMessage to true for the test harness is purely for illustration, the test harness actually forces RetrieveMessagesFromServerOnGetMessage to true for the tests so the output is always as expected.

Note3:
occasionally, cleanup is unsuccessful. This does not constitute a problem in any way, as the harness starts with a clean database, and clean copies of all files. However, you will see something like this in the output:
Warning: Cleanup failed on baseDir: /local/myrun1/DerbyNet/supersimple