This document describes functionality of the derby functional testing package org.apache.derbyTesting. This package is based on the functional tests in use at IBM for testing the Cloudscape product before its contribution to ASF.
In the following, instructions are geared towards a unix
environment. For other environments, some details may need to be
adjusted. For instance, the document may
refer to $ANT_HOME, for DOS, this would be %ANT_HOME%.
In the following the top
directory under which the subversion tree is placed is referred to as
${derby.source} - see also the
derby BUILDING.txt.
The version of the classes and supporting files of the derbyTesting
package have to match the version of the classes of the derby package.
Thus you either need to build all jars yourself, or get all jar files
from the Derby site at the same time when available.
The derbyTesting package enables you to run 1 test or a suite of tests.
Before you can run, you need to setup your environment:
| jdk131
- Sun
HotSpot jdk1.3.1 jdk141 - Sun HotSpot jdk1.4.1 jdk142 - Sun HotSpot jdk1.4.2 jdk15 - Sun HotSpot jdk1.5 ibm131 - IBM Classic jdk1.3.1 ibm141 - IBM Classic jdk1.4.1 ibm142 - IBM Classic jdk1.4.2 j9_13 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition) |
For example:
| (note that $jardir is
only a convenience variable and that the set CLASSPATH command below
has carriage
returns for formatting reasons): set jardir=/local/derbyjar set CLASSPATH="$jardir/derby.jar:$jardir/derbytools.jar:$jardir/derbynet.jar:$jardir/derbyclient.jar: $jardir/db2jcc.jar:$jardir/db2jcc_license_c.jar: $jardir/derbyTesting.jar:/local/derby/tools/java/jakarta-oro-2.0.8.jar: $jardir/derbyLocale_de_DE.jar:$jardir/derbyLocale_es.jar:$jardir/derbyLocale_fr.jar: $jardir/derbyLocale_it.jar:$jardir/derbyLocale_ja_JP.jar:$jardir/derbyLocale_ko_KR.jar: $jardir/derbyLocale_pt_BR.jar:$jardir/derbyLocale_zh_CN.jar:$jardir/derbyLocale_zh_TW.jar: $CLASSPATH set PATH=/local/jdk141/bin:$PATH |
To run 1 test:
| syntax: java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunTest <testdir>/<testname> where
to run the test supersimple against the embedded driver: java org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql To run a test with network server, using the derbyclient driver, add -Dframework=DerbyNetClient to the run. The test harness will to start network server at port 1527 or connect to a running one, run the test, and stop network server thereafter. for example: java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql |
A successful run will have a .pass file, and the output to the
console will show no difference between expected and actual test
result. A failed test run will have at least a .fail file and the
output to the console will show the difference between expected and
actual result.
To run a suite:
| syntax: java -D<testproperty> org.apache.derbyTesting.functionTests.harness.RunSuite <testsuite> where
java -Dverbose=true org.apache.derbyTesting.functionTests.harness.RunSuite derbylang |
Each suite run should be started in a clean directory. The test output directory will not be emptied out before testing is begun, although individual test files and result files will be cleaned out and overwritten.
To run a suite with J2ME/CDC/Foundation:
| (note that the command
below
has carriage returns for formatting reasons): syntax: <java> -Xbootclasspath/a:$jardir/<jdbc_jar> -Dbootcp=$jardir/<jdbc_jar> -Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource -Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource" org.apache.derbyTesting.functionTests.harness.RunSuite <testsuite> where
j9 -jcl:foun10 -Xbootclasspath/a:$jardir/jdbc.jar -Dbootcp=$jardir/jdbc.jar -Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource -Djvmflags="-Dij.dataSource=org.apache.derby.jdbc.EmbeddedSimpleDataSource" org.apache.derbyTesting.functionTests.harness.RunSuite derbyall |
The suites provided are:
A successful run with all tests passing will have no *.fail files created, the <testsuite>_fail.txt file will be empty, and the <testsuite>_report.txt file will show no failures in the Summary results section.
| -------snippet from
derbylang_report.txt ----- ----------------------------------------------------------- Summary results: Test Run Started: 2004-11-10 11:27:55.0 Test Run Duration: 00:04:09 129 Tests Run 100% Pass (129 tests passed) 0% Fail (0 tests failed) 0 Suites skipped |
To build the derbyTesting package:
This is some typical
output for the ant build process.
| > cd
/local/derby/java/testing > ant.ksh Searching for build.xml ... Buildfile: /local/derby/java/testing/build.xml compile: [javac] Compiling 30 source files to /local/derby/classes ... [copy] Copying 1 file to /local/derby/classes/org/apache/derbyTesting/funct ionTests BUILD SUCCESSFUL Total time: 10 minutes 3 seconds |
Building using the ant all target places all files, that is,
classes, but also supporting files such as expected output (*.out), sql
test files (*.sql), properties files and any data files used in
individual tests into the classes directory so they can all be found
using the CLASSPATH.
Once you have built the derbyTesting package, you can make a derbyTesting.jar using the jar build target at the ${derby.source}level.
This will look something like:
| c:> ant derbytestingjar Searching for build.xml ... Buildfile: C:\derby\build.xml initjars: [mkdir] Created dir: C:\derby\jars\ [mkdir] Created dir: C:\derby\jars\lists [echo] Revision number set to exported [echo] . derbytestingjar: [echo] Beginning derbytesting.jar build ..... BUILD SUCCESSFULL |
The functional tests are run using a class called 'RunTest'. This class
calls a number of other classes. A group of tests, called a 'suite' is
executed using a class called 'RunSuite'.
See section 2.1 for the basic steps to run 1 test.
To pass on system level properties to the test harness, use the test
harness property -DtestSpecialProps. For example, to ensure extra
information is appended to the log:
| java -Dframework=DerbyNetClient -DtestSpecialProps=derby.infolog.append=true org.apache.derbyTesting.functionTests.harness.RunTest lang/supersimple.sql |
Tests will be executed in the current directory. When
running a test
using the network server and derby client, i.e.
-Dframework=DerbyNetClient, the test will run
in a subdirectory (automatically created) 'DerbyNetClient'.
The test will normally create the following:
possibly created:
When the test is successful, cleanup will occur unless the test harness
property -Dkeepfiles=true is used. Cleanup will attempt to cleanup all
files except for .pass.
See Note3.
A successful run (this example is from a dos environment) would look for instance like:
| c:>
derbyTesting.functionTests.harness.RunTest lang/supersimple.sql C:\derby\run2 supersimple -- listing properties -- derby.locks.deadlockTimeout=3 derby.locks.waitTimeout=3 *** Start: supersimple jdk1.4.2_03 2004-11-10 16:51:02 *** The test should be running... MasterFileName = master/supersimple.out *** End: supersimple jdk1.4.2_03 2004-11-10 16:51:25 *** |
A Test Failure shows the diff, creates a .fail file, does not create
a .pass file, and does not cleanup any files upon completion. The
output might look like this:
| c:>
derbyTesting.functionTests.harness.RunTest lang/supersimple.sql C:\derby\run2 supersimple -- listing properties -- derby.locks.deadlockTimeout=3 derby.locks.waitTimeout=3 *** Start: supersimple jdk1.4.2_03 2004-11-10 16:54:39 *** The test should be running... MasterFileName = master/supersimple.out 10 del < 10 10a10 > 1 Test Failed. *** End: supersimple jdk1.4.2_03 2004-11-10 16:55:02 *** |
See section 2.1 for a basic explanation on how to run a suite of tests.
Tests will be run in a subdirectory with the name of the test suite under the current directory. Eg. for derbylang suite, a directory derbylang will be created. While the tests are run, information about the run is inserted into a <testsuite>.sum file. When all tests have completed summary files are created <testsuite>_pass.txt, _fail.txt, and _diff.txt files are created as well as a <testsuite>_report.txt with additional details. Some of the information is duplicate. Also, a .skip file will be created holding a list of the tests that were skipped (for more details on this, see the section on skipping tests).
RunSuite does not empty the top level directory before running. Thus, if another suite was run in the same directory at an earlier time, the resulting summary files might contain results for more than the current run. Therefore it is important to run each suite in a clean directory.
Sample output from RunSuite:
| c:> $ java
org.apache.derbyTesting.functionTests.harness.RunSuite derbylang Top suite: derbylang Suite to run: derbylang:derbylang Now do RunList Now run the suite's tests Run the tests... Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtoprepo rtdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/altertable.sql Execute command: java -DjavaCmd=java -Doutputdir=C:\derbyt1\derbylang\derbylang -Dtopsuitedir=C:\derbyt1\derbylang -Dtopreportdir=C:\derbyt1\derbylang -Drundir=C:\derbyt1 -Dsuitename=derbylang:derbylang -Dtopsuitename=derbylang org.apache.derbyTesting.functionTests.harness.RunTest lang/arithmetic.sql ...(.more tests).... Generated report: derbylang_report.txt |
This output does not show whether the tests passed or failed. The Summary section in <testsuite>_report.txt shows the statistics of the passed vs. failed tests, the summary <testsuite>_*.txt files list the tests that passed and failed.
The following is intended for people who have the subversion tree available and want to add or modify tests.
The test harness executing one test basically does the following in sequence:
The test harness recognizes, or will recognize tests with the following
extensions:
Various additional files may be used by a test, for instance, to create
large data values, to test using of jar files and the like. Any files
that need to be accessed by a particular test that are not accessed
from the classpath need to be listed under supportfiles= in the
<testname>_app.properties file.
Tests can refer to classes without being in the classpath, and sql
tests can use the ij command 'run resource ' to execute additional .sql
files without changes to the _app.properties files.
For example, in the file
(org/apache/derbyTesting/functionTests/tests/)tools/dblook_test_app.properties:
supportfiles=tools/dblook_makeDB.sql,tools/dblook_test.jar
To support running on non-ISO-5889 systems, the harness copies files
ending in sql, .view, .multi, .properties, .txt and .policy into local
encoding; all other files are copied into UTF-8 encoding.
Every test directory has a default_app.properties. This file is for system level properties generic to all the tests in that test directory.
If a test requires different system level properties, a test specific properties file can be created to overwrite the defaults. The test specific properties file needs to have a name starting with the test file name, followed with _app.properties
For example, for the test tools/dblook_test.java, there is a
properties file called tools/dblook_test_app.properties
Every test directory has a default_derby.properties. This file is for
derby specific properties common to all the tests in that test
directory.
If a test requires different derby properties, a test specific
properties file can be created to overwrite the defaults. The test
specific properties file needs to have a name starting with the
test file name, followed with _derby.properties
The test's output will be put into a file testname.tmp. Then the output
is modified if masking is required and the result is put into a .out
file.
The expected output is found by examining the following directories,
based on certain input
For example, if we are running a test and the flag -Dframework=DerbyNet is used, to use network server and the IBM Universal JDBC Driver ('jcc'), and the jvm we are using is Sun's jdk 142, and the jcc version is 2.4 (not available at time of writing) then the search for the master to compare with starts in the functionTests/derbynet/jcc2.4/jdk14 directory. If a .out file with the same name as the test is found in that directory, that master is taken. If there is no such file in that directory, search continues in the directory functionTests/derbynet/jcc2.4/jdk13 if it exists.
If there is no file there, nor for any other jcc directory, it will
continue to derbynet/jdk14, and the search is continued for earlier jvm
versions.
If we are not running network server, the DerbyNet and
jcc_version directories are not traversed.
The version details do not go into the subversion level, i.e. running with jdk141 or jdk142 is expected to have the same behavior.
This functionality supports dealing with minor differences in behavior caused by minor differences in behavior in the underlying jvms, jcc versions, differences between results returned through network server vs. embedded and minor differences between a debug and non debug (jar) build.
However, having a large number of these files means a maintenance problem. Every time test output changes due to modifications to derby or to the test, all output files in all directories need to be updated accordingly. If at all possible, irrelevant differences should be masked out, or the test should be written so that the output does not reflect such items.
Suggestions to minimize canons:
Tests often fail because of unimportant differences, such as process
ids, statement ids, timestamps. The derby functional test harness
provides for masking of these differences at 2 levels:
The diff is executed between the final resulting output and the master
file found copied into local encoding as .tmpmstr.
To add a new test:
A suite constitutes of a <suitename>.properties file and/or a <suitename>.runall file in the org/apache/derbyTesting/functionTests/suites directory. The .properties files hold references to other .properties files, or .runall files, the .runall files are the actual lists of tests.
The lowest level suite always needs to have a .runall file.
For example, the derbyall suite is only a derbyall.properties file that refers to other suites in the 'suites' property:
| -----------------------derbyall.properties--------------- suites=derbylang derbynetclientmats derbynetmats storeall xa derbytools derby.debug.true=enableBtreeConsistencyCheck derby.stream.error.logSeverityLevel=0 |
The derbylang suite is only a derbylang.runall, which lists the tests. The derbynetclientmats suite has both a .runall and a .properties file, so some additional properties can be specified that are true for all tests in that suite.
| ------------------derbynetclientmats.properties----------------- framework=DerbyNetClient suites=derbynetclientmats derbynetmats jdk12test=true runwithj9=false timeout=60 |
To add a suite, you need to create at least a <suite>.runall
file, which lists the actual tests, or a properties file that refers to
other suites that do have a .runall file. The suite should be added
into the directory
${derby.source}/java/testing/org/apache/derbyTesting/functionTests/suites.
Currently, the supported jvms are:
| jdk131
- Sun
HotSpot jdk1.3.1 - class: jdk13 jdk141 - Sun HotSpot jdk1.4.1 - class jdk14 jdk142 - Sun HotSpot jdk1.4.2 - class jdk14 jdk15 - Sun HotSpot jdk1.5 - class jdk15 ibm131 - IBM Classic jdk1.3.1 - class ibm13 ibm141 - IBM Classic jdk1.4.1 - class ibm14 ibm142 - IBM Classic jdk1.4.1 - class ibm14 j9_13 - WCTME jvm (available with IBM Websphere Client Technology Micro Edition) - class j9_13 |
The classes above are subclasses of
org.apache.derbyTesting.functionTests.harness.jvm. The name at the
front is just a convention.
To run a test with a jvm that does not have a matching class under
org.apache.derbyTesting.functionTests.harness, do the following:
There are 2 skipping mechanisms in place for different kinds of
skipping of a test.
Some tests are written to test specific functionality only available
with for instance certain jvms, or, with network server, certain
versions of the IBM Universal Driver. To control this, properties can
be set for each test, for instance, if a test should not be run when
using an ibm jvm, set runwithibmjvm=false. If a test should be run with
Sun Hotspot jvm version 14, then set runwithjdk14=true.
The skip setting does not go into the subversion level, i.e. setting
runwithjdk141=false has no effect, and setting runwithjdk14 affects
runs with jdk141 as well as jdk142.
Other skip reasons are encryption protocols specific to a certain jvm.
The property for skipping a test based on the version of the IBM
Universal Driver is "excludeJCC". The keywords "at-or-before" and "at-or-after" can be used to specify
which range of JCC versions should be excluded. If neither of
these keywords is provided, the default is "at-or-before". For example:
You can also specify an (optional) jvm clause to further tune the
exclusion criteria. This clause starts with the ",when" tag and is followed by a
three-part jvm version. In this case, a test will only be skipped
if BOTH the JCC clause AND the jvm clause are true. For example:
To skip a test when running with any version of the IBM Universal
Driver that is 2.4 or later, but ONLY if the jvm is 1.3 or earlier:
excludeJCC=at-or-after:2.4,when-at-or-before:jdk1.3.1
Another skipping mechanism works on entire 'frameworks'. Currently there are only 3 supported in the harness, embedded, network server with the derbyclient driver ('DerbyNetClient') and network server with the IBM Universal JDBC Driver ('DerbyNet'). In the suites directory there are .exclude files for each of the frameworks - if for some reason an exclude file were not there, you would see a warning message show up for every test run. In this 'framework'.exclude file tests can be placed that for some reason need to be excluded from running with that framework. This mechanism enables adding of suites to a framework run even if a few of the tests are not appropriate for that particular framework.
Note that at this time, only skipped suites show up in the .skip result file. This still needs to be corrected.
Currently, there are two frameworks used for network server, DerbyNetClient,
which uses the derby Client driver, and DerbyNet, which uses the IBM
Universal JDBC Driver.
Setting the framework property will invoke the test harness class
NetServer which has the actual configuration (driver name, portnumber
etc.) used for the individual frameworks. See Note4.
Setting this framework also causes the search for expected output to
include appropriate DerbyNetClient or DerbyNet and jcc version
specific subdirectories under the master directory.
For a complete set, refer to comments in RunTest.java, but here are some valuable test properties which can be passed to the RunTest class:
| runwith<jvm> See above section 4.10 framework specifies which framework to run with. For example: java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.RunTest lang/supersimple.sql verbose Shows more detailed output. For example: java -Dverbose=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql keepfiles Indicates to not clean up any of the files if the test passed. java -Dkeepfiles=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql testSpecialProps sets additional properties. Several can be set using '^' as separator: -DtestSpecialProps=<prop-1>=<value-1>^ ... ^<prop-n>=<value-n> java -DTestSpecialProps=derby.infolog.append=true org.apache.derbyTesting.functionTests.RunTest lang/arithmetic.sql jvmflags sets specific jvm properties for the jvm used in the test harness, for instance initial memory, and heap size, or properties normally passed on with a -D. For instance: java -Djvmflags=ms32M -mx128M org.apache.derbyTesting.functionTests.RunTest lang/streamingColumn.java excludeJCC See above section 4.10 useprocess (default=true) Controls whether RunTest runs the test in a separate VM or in a thread in harness VM. It is potentially useful for debugging tests. Unit tests are not (yet) runnable with "useprocess=false", though. |