performance test framework summary

introduction

This document describes the Jena performance testing framework.

performance tests

A performance test is a class that implements the PerformanceTest interface. That interface has the methods:

void setUp(): called to perform any needed preparation for the test (eg, set up data). This method is not timed.
void tearDown(): called to clear up any debris from the test (eg, big models). This method is not timed.
execute(): called to execute the test body. The test is run repeatedly until the time budget is exhausted. Since the framework uses System.currentTimeMillis(), tests should be moderately fat.
void warmup(): called before execute to allow the JVM to be warmed-up (ie the JIT encouraged to compile code). This method is not timed.
void report( Resource root ): called to add any additional test result data to the item root. (The framework has already added timing data.)
String name(): the name of the test, to be displayed in the test result data.
String description(): a description of the test, to be displayed on request in the test data.
Collection dependsOn(): The engine sets-up and runs each test that is depended upon, then sets-up and runs this test. Then tears down this test. Then tears down the tests that it depends upon.

running tests

The usual way to run the tests is from the command line. The test-running class is com.hp.hpl.jena.perf.Run. The quick help is:

Run -help
Run [-noisy] [-title <TITLE>] [-tests <TESTNAME>] [-config <CONFIG-FILE>]

Run in the top-level directory of the CVS jena-perf download.
Arguments can be repeated, to add more tests, or more config.
When not specified, the default tests are 'legacy10'.
When not specified, the default config is etc/config.n3.
File names are specified in etc/location-mapping.n3.

It takes the following options, in any order:

-title: the title of this test run.
-tests T1...: the names of the tests to run. A test name is either a full class name, or a shortName defined in a config file.
-config F1...: the names of config files to be loaded and used to expand test descriptions.
-noisy: if specified, the framework logs its progress to standard error.

The output RDF/N3 is written to standard output.

config file

The config file(s) define test groups by their short names. A test group is a resource with at least one perf:shortName property; the string value of that property is the group's short name.

A test group has zero or more perf:className properties; each such string value is the full class name of a test class to be run.
A test group has zero or more perf:include properties; each such resource value is another test group whose test classes are to be included into this ones.
A test group has zero ro more perf:includeByName properties; each such string value is the short name of another test group whose test classes are to be included into this one.

Note that a test may be empty, ie contain no classes either directly or indirectly.

Note also that it is permitted for tests to cross-refer to one another, in which case they are equivalent and contain all of each other's tests.

classpath issues

When running the tests, the classpath must include both the framework classes and the Jena libraries of the Jena release being tested.

test result schema

The tests results are RDF (expressed as N3) according to a schema not yet exposed. The test run is represented as a bnode with two properties: an environment describing the test environment:

perf:JVM - the JVM the test was run under
perf:Jena - the version of Jena used (broken)
perf:OS - the OS the test was run on
perf:date - the date the test was run
perf:time - the time the test was run
perf:title - the arguments of the title option
perf:user - the user who ran the tests

and a suite whose elements describe each test run:

perf:averageTime - the average time of the test (in ms)
perf:description - the test description
perf:nTimes - the number of times the test was run
perf:name - the test name
perf:totalTime - the total time the test occupied

See com.hp.hpl.jena.perf.Perf for the vocabulary.

displaying test results

The test results can be rendered by running the class com.hp.hpl.jena.perf.Render, supplying the names of test result files on the command line. Render generates an HTML table from the combined results.