README FILE

1. Goal of this document
2. Description of product
3. Configuration files
   • cfg_env.xml
   • cfg_ts.xml
   • "test_name".xml
4. Default plugins
5. Instruction for product launch
6. How to pass the environment variables into the tests
7. Examples
   • cfg_env.xml
   • cfg_ts.xml
   • "test_name".xml
8. How to
   • create report without test run (based on previous results)
   • use TH for tests running
   • use TH for JUnit tests running

This document describes what is Test Harness tool and how to use it. up

Description.

Test Harness tool (TH) is a product for tests execution . It has flexible and configurable settings. TH consists of core, data and plugins. TH core is the "Main" class, Configuration module, Dispatcher module and micro core to execute tests on remote hosts.

There are some plugins in TH product:

• TestFinder plugin, finds test in pointed directories and passes it to the parser;
• ResFinder plugin, finds results of testing and passes it to the parser;
• Selector plugin, selects tests for execution according to defined parameters;
• Storage plugin, saves testing results into a file and/or passes it to Reporter;
• Reporter plugin, generates reports;
• Logger plugin, writes log messages into the log (stream out, file etc.);
• ExecUnit plugin, executes tests.

Important note: Configuration files must have ASCII coding. The xercesImpl.jar is needed for parsing xml files. You must point location xercesImpl.jar and th.jar in "classpath" variable to run tests (see example below in the "Instruction for product launch." section).

There are two main configuration .xml files for the whole test suite and one description .xml file for every test.

Main configuration files are:

• cfg_env.dtd - file, where the general structure of xml file is defined, thus cfg_env.xml file must correspond with this structure.
• cfg_env.xml - this file contains the general settings for TH launching.
• cfg_ts.dtd - file, where the structure of testsuite  .xml file is defined, thus cfg_ts.xml file must correspond with this structure.
• cfg_ts.xml - this file contains test suite settings.

Test description files are:

• test.dtd - file, where the structure of <testname>.xml file is defined, thus <testname>.xml file must correspond with this structure. <Testname>.xml file is kind of test description.
• <Testname>.xml - file contains the test settings. <Testname> must be the same as the test class name.

Configuration files details:

cfg_env.xml:

<property>

common properties for TH:

TestSuiteRoot

- directory of test suite. It consists of three subdirectories:

1. bin - contains test classes  and libraries required for execution
2. config - contains test suite configuration files - cfg_ts.dtd, cfg_ts.xml, java.policy
3. src - contents sources of test and sources of plugins, which are specific for this suite

Note, if the structure of the test suite differ from pointed the next properties can be used (if needed) to describe the structure:

• TestSuiteTestRoot - path to the test description (if exist)/ sources
• TestSuiteClassRoot - path to the executable part of tests
• TestSuiteConfigRoot - path to the configuration files for the suite
• TestSuiteLibRoot - path to the libraries

TestResultRoot

directory for testing results,
for example - C:\results\TEST_BASE\today

TestedRuntime

location of Java runtime file,
for example - C:\jrockit-j2sdk1.4.2_04\bin\java.exe

TestedCompile

location of compiler,
for example - C:\jrockit-j2sdk1.4.2_04\bin\javac.exe

ReferenceRuntime

location of Java runtime reference ,
for example - C:\java\bin\java.exe

TempStorage

temporary directory,
for example - C:\tmp

GenTimeout

timeout
for example - 100 (means 100 sec)

LogLevel

level of the Logger (java.util.logging.Level), ALL | CONFIG | FINE | FINER | FINEST | INFO | OFF | SEVERE | WARNING
for example - INFO (default). The level of output stream details for tests.

<execution>

contains modes for test execution:

1. exec - SAME/OTHER, SAME means that tests and TH are executed in the same Java machine, OTHER - different JVM
2. location - LOCAL/REMOTE, LOCAL means that tests are executed on the local machine, REMOTE means the execution on the remote machine
3. concurrent - 1-N, the number of tests to execute at the same time
4. syncChanneHost - the host to run the server which provides the synchronize channel possibility (at present time the "localhost" supported only)
5. syncChannePort - the port number to run the sync channel server (default value is 21001)
6. generalVMOption - the string with options to pass to VM for all tests
7. bootClassPath - the value to set the class path for boot class loader, usually -Xbootclasspath/a:$CP or something like that. The harness add nothing to this string.

<run-remote>

list of parameters for remote execution

<host>

host where MCore and waiting command must be launched or already are launched/

<name>

name of the host where MCore was launched

<port>

port number where TH/MCore expect a connection

<mode>

client/server mode for the MCore launched

<run>

true/false should the TH tries to run the MCore. Note, now it works for the local host only.

<totest>

list of subsuites for execution

<subsuite>

the full name of subsuite for TestFinder module

<plugin-list>

default list of plugins

<plugin>

plugin name and plugin class name. Class can be substituted by class from cfg_ts.xml file

<parameter>

parameter for plugin

<value>

values of parameters, every value has name and content

• Logger - plugin, required for execution test logging, by default without parameters
• TestFinder - plugin, required  for tests finding, by default without parameters
• ResFinder - plugin, required  for results finding,  by default without parameters
• Selector - plugin, required  for selecting tests for execution, by default without parameters
• Storage - plugin, required for saving results of testing, by default without parameters
• ExecUnit - plugin, required  for execution units, by default without parameters
• Reporter - plugin, required  for making reports, it has at least 1 default parameter with 4 possible values:
  1. "passed" = 104
  2. "failed" = 105
  3. "error" = 106
  4. "moderror" = 107 (test does not support current configuration)

NOTE: All names of plugins are reserved, because it's parsed by TH. "ExecUnit" is also reserved name for execution units parameters .

<mapping>

the list of the mapping runners to class that executes test. Runners must have names corresponding with names for ExecUnit plugin from env_cfg.xml file

<runner-mapping>

name and corresponding class name of runner

<plugin-list>

user's plugins list, specific for this test suite

<plugin>

plugin with the own class of realization and parameters, which can redefine  the same named plugin, defined in cfg_env.xml.

<property-list>

list of the test suite properties

<property>

name and value of test suite property

ID

name of test (the last syllable of full test name)

date-of-creation

date of creation for this test YYYY-MM-DD

timeout

timeout factor, to multiply GetTimeout (from cfg.env.xml) by it to calculate the real timeout for this test.
for example:

   <Test ID="test1" date-of-creation="2004-10-10" timeout="1">

<Copyright>

tag for copyright information, must be at least one, for example:

   <Copyright value="xxxx"/>

<Author>

tag for information about author, should be at least one, for example:

   <Author value="AuthorName"/>

description

there are three types of test and three types of description accordingly. The description must be only one of the following:

<MulticaseTestDescription>

description of multicase test, for example:

   <MulticaseTestDescription>
        <TestedClass name="java.lang.Byte"/>
        <Case ID="testByteValue0001">
            <TestedMethod name="public byte byteValue()"/>
            <Precondition>create the Byte(123) object</Precondition>
            <Description>checks that Byte(123).byteValue() returns the 123</Description>
            <Expected>the values are equals</Expected>
        </Case>
        <Case ID="testByteValue0002">
            <TestedMethod name="public byte byteValue()"/>
            <Precondition>create the Byte(-123) object</Precondition>
            <Description>checks that Byte(-123).byteValue() returns the -123</Description>
            <Expected>the values are equals</Expected>
        </Case>
        <Case ID="testValueOf0001">
            <TestedMethod name="public static Byte valueOf(String s) throws NumberFormatException"/>
            <Precondition></Precondition>
            <Description>checks that Byte.valueOf("123") returns the same object as Byte(123)</Description>
            <Expected>the values are equals</Expected>
        </Case>
        <Case ID="testValueOf0002">
            <TestedMethod name="public static Byte valueOf(String s) throws NumberFormatException"/>
            <Precondition></Precondition>
            <Description>checks that Byte.valueOf("-123") returns the same object as Byte(-123)</Description>
            <Expected>the values are equals</Expected>
        </Case>
    </MulticaseTestDescription>
        

<APITestDescription>

description of API test, for example:

   <APITestDescription>
        <TestedClass name="java.lang.Boolean"/>
        <TestedMethod name="public boolean booleanValue()"/>
        <Description>
            This is a test for the booleanValue() method.
            This method checks that the boolean value for Boolean('TRUE') is true and the value Boolean('false') is false.
        </Description>
    </APITestDescription>

<Description>

description of simple test, for example:

   <Description>mega description</Description>
        

<Keyword>

tag for keyword, using by Selector module for include into or exclude from the execution list, for example:

   <Keyword name="stress"/>

<Source>

points to test source file, must be at least one, for example:

   <Source name="test1.java"/>

<Modification>

contains date and author of the last test files modification, for example:

   <Modification date="2004-10-20" author="Author3"/>

<Runner>

runner ID, options for execution and list of executors with proper options, for example:

   <Runner ID="Runtime">
        <Option name="-D" value="qwerty"/>
        <Param name="toRun" value="api.java.lang.Boolean.test.test1">
            <Option value="true"/>
        </Param>
        <Param name="toRun" value="api.java.lang.Boolean.test.test2">
            <Option value="false"/>
            <Option name="-cp" value="classpath"/>
        </Param>
    </Runner>
        

Note, the interpretation of options before the first 'Param' depends on the execuiton unit (usually it interpreted as product options. VM, for example).
Also note, the any from 'name' or 'value' for options can be missed.

<Resource>

resources for test execution, for example:

   <Resource name="printer"/>

<Restriction>

restrictions for test execution, for example:

   <Restriction name="SameVMOne"/>

• TestFinder plugin. By default finds all .xml files in the <root>\src\tests directory and tries to parse them as test descriptions;
• ResFinder plugin. By default finds all .xml files in the <results> directory and tries to parse them as test result descriptions;
• Selector plugin. By default rejects all tests with unknown execution unit;
• Storage plugin. By default saves testing results to .xml file and passes it to Reporter plugin;
• Reporter plugin. By default generates reports in the .xml file (with a style sheet). Please, use the IE browser to see it;
• Logger plugin. By default writes all messages to console;
• ExecUnit plugins:
• Runtime - runs the single case tests (vm, for example) on the runtime pointed by TestedRuntime variable in the cfg_env.xml file. Return 'passed' result if all commands are passed.
• RuntimeNegative - runs the single case tests (vm, for example) on the runtime pointed by TestedRuntime variable in the cfg_env.xml file. Return 'passed' result:
  • if exit value of process specified and exit value as expected
  • if output message (out+err) specified and ouput contains message as
    • simple string (first check)
    • output has string that match to specified pattern (second check)
  • both
  • if exit value of process is not zero and no exit value or output message specified
  
  For example to check the exit code and message the following predefined parameters should be pointed (or any of it for single check):

      <Runner ID="RuntimeNegative">
          <Option name="runnerECode" value="12345"/>
          <Option name="runnerEMsg" value="Hello word"/>
          <Param name="toRun" value="test"/>
      </Runner>

• MultiCase - runs the multicase test (api, for example) on the runtime pointed by TestedRuntime variable in the cfg_env.xml file. The test cases to run are defined through the reflection as methods that return the org.apache.harmony.share.Result type and whose name started from the word 'test'. Return 'passed' result if all commands are passed.
• ComilerRuntime - compiles the pointed source on the compiler defined by TestedCompile variable in the cfg_env.xml file and tries to run the resulting class file on the defined ReferenceRuntime. Return 'passed' result if all commands are passed.
• CompilerNegative - runs the compiler tests that are not to be compiled on the compiler pointed by TestedCompile variable in the cfg_env.xml file. Return 'passed' result if last command is failed.
• Execute - runs the user defined commands. Return 'passed' result if all commands are passed.
• ExecuteNegative - runs the user defined commands. Return 'passed' result if last command is failed.
• DistributedRunner - runs all command in parallel. Return 'passed' result if all commands are passed.
  To support the parallel run each <Param> may have the option with predefined name "vmParam" that will be passed to VM to run this command only.
• JPDARunner - the subclass of DistributedRunner. Add the JPDA specific parameters only. The other behavior is the same as DistributedRunner.
• JUExec - run the test in the jUnit format.

%JAVA_HOME%\java [-options] "org.apache.harmony.harness.Main" [-parameters]
%JAVA_HOME%\java [-options] -jar th.jar [-parameters]

This is an example of run.bat file (located in the testsuite directory) for product launch:

------------------
#!sh

TOOLS=C:/jrockit-j2sdk1.4.2_04/lib/tools.jar
TESTBASE=I:
------------------
java.exe -showversion -cp "$TOOLS;$TESTBASE/xercesImpl.jar;$TESTBASE/th.jar" org.apache.harmony.harness.Main -cfp "usr.config"

%JAVA_HOME%

directory where Java is installed

-options

used Java options. Use help for more details, please.

-parameters

aplied in command line parameters of TH. They can be (Use help for more details, please):

-ln <name>

logger name

-cn <name>

configurator name

-cfp <name>

path to the configuration files

-gui

GUI mode ("-gui"/"-batch")

-subs

paths to the subsuites divided by space

-logl <val>

harness log level: 0 - off, 1 - finest, 2 - finer, ... 7 - severe

-include "name1 name2..."

defines test or file names with the test list to include in this execution

-exclude "name1 name2..."

defines test or file names with the test list to exclude from this execution

-version

print the harness version

-property <name> <val>

defines new value for the configuration property

-execopt <name> <val>

defines new value for the execution options

The TH uses the shell notation for the environment variable - '$'+"variable name". In order to pass the value of some environment variable, needed to test compilation or execution (for example as an option value for java or javac options), it must be defined as an option of "RunnerID" parameter in the <testname>.xml file.

The following variable names are reserved in harness:

• TestDirectory - full name of the path to the test description (test.xml)
• FileSeparator - the system file separator character
• PathSeparator - the system path separator character
• CP - the current class path
  The class path is formed as
  CP = TestSuiteClassRoot + TempStorage + all_jars_from_bin_lib + System.getProperty("java.class.path")

Also other variables which are defined in cfg_env.xml and are common for the whole test suite are accessible. For the particular test you can define the compilation or execution option, using these variables.

For example, the line of <testname>.xml file:

<Option name="-cp" value="$TempStorage$PathSeparator$TestSuiteClassRoot"/>

-cp C:/tmp;C:/TEST_BASE/bin/classes

<Option name="-d" value="$TempStorage"/>

-d C:/tmp

Structure of tests:

• TEST_BASE - directory that contains test sources, classes and libraries.
• TEST_BASE\bin\classes - directory for classes.
• TEST_BASE\bin\lib - directory for libraries.
• TEST_BASE\config - directory for testsuite configuration files.
• TEST_BASE\src\plugin\harness - directory for user's plugin source files.
• TEST_BASE\src\test - directory for test sources.
• results - directory for test execution results.
• config - directory for users configuration files.

************************************* cfg_env.xml *************************************

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE env_configuration SYSTEM "cfg_env.dtd">
<env_configuration>

    <properties>
        <property name="TestSuiteRoot">TEST_BASE</property>
        <property name="TestResultRoot">results</property>
        <property name="TestedRuntime">c:/jrockit-j2sdk1.4.2_04/bin/java.exe</property>
        <property name="TestedCompile">c:/jrockit-j2sdk1.4.2_04/bin/javac.exe</property>
        <property name="ReferenceRuntime">c:/jrockit-j2sdk1.4.2_04/bin/java.exe</property>
        <property name="TempStorage">C:\tmp</property>
        <property name="GenTimeout">500</property>
        <property name="LogLevel">OFF</property>
    </properties>
    
    <execution>
        <exec-mode name="exec">other</exec-mode>
        <exec-mode name="location">local</exec-mode>
        <exec-mode name="concurrent">1</exec-mode>
    </execution>
    
    <run-remote>
        <host name="localhost">
            <port>5678</port>
            <mode>passive</mode>
            <run>true</run>
        </host>
    </run-remote>
    
    <totest>
        <subsuite name="lang">api</subsuite>
        <subsuite name="jls">jls</subsuite>
    </totest>

    <plugin-list>
        <plugin name="Logger" class-name="org.apache.harmony.harness.plugins.THLogger">
        </plugin>
        <plugin name="TestFinder" class-name="org.apache.harmony.harness.plugins.TestFinder">
        </plugin>
        <plugin name="Selector" class-name="org.apache.harmony.harness.Selector">
            <parameter name="Runner">
                <value name="run">run</value>
                <value name="compile">compile</value>
            </parameter>
        </plugin>
        <plugin name="ExecUnit" class-name="org.apache.harmony.harness.ExecUnit">
            <parameter name="Runtime">
                <value name="host">localhost</value>
                <value name="action">doit</value>
            </parameter>
            <parameter name="CompileNegative">
                <value name="class-compiler">javac</value>
            </parameter>
            <parameter name="CompileRuntime">
                <value name="class-compiler">c:/jrockit-j2sdk1.4.2_04/bin/javac.exe</value>
            </parameter>
            <parameter name="DistributedRunner">
                <value name="host1">host1</value>
                <value name="host2">host2</value>
            </parameter>
        </plugin>
        <plugin name="Storage" class-name="org.apache.harmony.harness.plugins.StoreRes">
        </plugin>
        <plugin name="ResFinder" class-name="org.apache.harmony.harness.plugins.ResFinder">
        </plugin>
        <plugin name="Reporter" class-name="org.apache.harmony.harness.plugins.Reporter">
            <parameter name="write">
                <value name="over">true</value>
            </parameter>
            <parameter name="status">
                <value name="passed">104</value>
                <value name="failed">105</value>
                <value name="error">106</value>
                <value name="moderror">107</value>
            </parameter>
        </plugin>
    </plugin-list>

</env_configuration>
    

************************************* cfg_ts.xml **************************************

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ts_configuration SYSTEM "cfg_ts.dtd">
<ts_configuration>

    <mapping>
        <runner-mapping name="Runtime" class-name="org.apache.harmony.harness.plugins.Run"/>
        <runner-mapping name="CompileNegative" class-name="org.apache.harmony.harness.plugins.Comp"/>
        <runner-mapping name="CompileRuntime" class-name="org.apache.harmony.harness.plugins.CompRun"/>
        <runner-mapping name="JUExec" class-name="org.apache.harmony.harness.plugins.JUExec"/>
    </mapping>
    
</ts_configuration>
    

********************************** <testname>.xml ************************************

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Test SYSTEM "test.dtd">
<Test
    ID="bValue_test"
    date-of-creation="2004-10-10"
    timeout="1.1"   >
    <Copyright value="xxxx"/>
    <Copyright value="somebody else"/>
    <Author value="Author1"/>
    <Author value="Author2"/>
    <Description>
        mega description
        mega description
    </Description>
    <Keyword name="stress"/>
    <Source name="bValue.java"/>
    <Modification date="2004-10-20" author="Author3"/>
    <Runner ID="Runtime">
        <Option name="-showversion"/>
        <Option name="-cp" value="classpath2"/>
        <Param name="toRun" value="api.java.lang.Boolean.bValue.bValue">
            <Option value="true"/>
            <Option name="-cp" value="classpath"/>
        </Param>
        <Param name="toRun" value="api.java.lang.Boolean.bValue.bValue2">
            <Option value="false"/>
            <Option name="-cp" value="classpath"/>
        </Param>
    </Runner>
    <Resource name="printer"/>
    <Restriction name="SameVMOne"/>
</Test>
    

1. Define the directory with test run results (you should run tests before)
2. Define the directory with configuration for this run
3. Create "report.bat" file, for example:

       set TESTBASE=C:\HarnessTest
       java.exe -cp "%TESTBASE%\xercesImpl.jar;%TESTBASE%\th.jar" org.apache.harmony.harness.ReportTool.Report -cfp "usr.config"

   Or create "report.sh" shell file, for example:

       #!sh
       TESTBASE=C:/HarnessTest
       java.exe -cp "$TESTBASE/xercesImpl.jar;$TESTBASE/th.jar" org.apache.harmony.harness..ReportTool.Report -cfp "usr.config"

4. Then execute report.bat or report.sh. See the result where pointed.

1. Create test source .java file, for example, with method "test(Logger, String[])" that returns integer value:
   • 104 - "passed"
   • 105 - "failed"
   • 106 - "moderror"
   • 107 - "unknown error"
2. Then add method "public static void main(String[] args)" like below:

       public static void main(String[] args) {
           System.exit(new ().test(Logger, args));
       }

3. Create .xml file with description and parameters of the test. Put the source and .xml file of your test in the <TESTBASE>\test directory.
4. If you need, put required libraries into <TESTBASE>\bin\lib directory.
5. If you need, put your plugin source file into TESTBASE\src\plugins directory.
6. Create configuration files for environment, harness and testsuite and put them into usr.config , TESTBASE\config, and <TESTBASE>\test directories correspondingly.
7. Create "run.bat" file, for example:

       set TESTBASE=C:\HarnessTest
       set TEST=%TESTBASE%\Test
       java.exe -showversion -cp "%TESTBASE%\xercesImpl.jar;%TESTBASE%\th.jar" org.apache.harmony.harness.Main -cfp "usr.config"

   Or create "run.sh" shell file, for example:

       #!sh
       TESTBASE=C:/HarnessTest
       TEST=$TESTBASE/Test
       java.exe -showversion -cp "$TESTBASE/xercesImpl.jar;$TESTBASE/th.jar" org.apache.harmony.harness.Main -cfp "usr.config"

8. Then execute run.bat or run.sh.

1. Create JUnit test files. Put them into the source directory. Compiled source files and put them into the class directory.
2. Create configuration files for the environment, harness and testsuite, like above. Just point the JUExec module and its parameter (CLASSPATH: -cp="...") in the cfg_env.xml and obligatory JUExec mapping in cfg_ts.xml and change finder for JUnit tests.
   For example,
   • in environment or testsuite configuration change finder:

         <plugin name="TestFinder" class-name="org.apache.harmony.harness.plugins.JUFinder"></plugin>
                         

   • and in test harness configuration make mapping for executor:

         <runner-mapping name="JUExec" class-name="org.apache.harmony.harness.plugins.JUExec"/>
                         

3. Put required libraries into lib directory. You need "junit.jar" so you have to put it in your library directory.
4. Create "run.bat" file, for example:

       set TESTBASE=C:\HarnessTest
       set TEST=%TESTBASE%\Test
       java.exe -cp "%TESTBASE%\xercesImpl.jar;%TESTBASE%\junit.jar;%TESTBASE%\th.jar" org.apache.harmony.harness.Main -cfp "usr.config"

   Or create "run.sh" shell file, for example:

       #!sh
       TESTBASE=C:/HarnessTest
       TEST=$TESTBASE/Test
       java.exe -cp "$TESTBASE/xercesImpl.jar;$TESTBASE/junit.jar;$TESTBASE/th.jar" \
           org.apache.harmony.harness.Main -cfp "usr.config"

5. Then execute run.bat or run.sh.
6. To analyse test results look at the TestResultRoot directory, which were defined in cfg_env.xml.