This directory is a repository for test files and test automation for Xalan.
Special Note: the documentation for the tests is still in the process of being migrated into Apache, so expect changes and improvements over the next couple of weeks. The eventual plan is to have both doc here, in the source tree, as well as a copy of pre-built docs up on the xml.apache.org website, to make it easier to start diving in and running (or submitting!) tests. The doc will also be much better organized - for the time being, I'm simply putting most everything into this one file.
Purpose: These tests are provided for Xalan contributors to evaluate the impact of code changes. Run the tests on the unchanged code, make the change and rebuild, then run the tests again and compare the results. Results should also be compared to the files in the "-gold" directory trees. Even though not all tests have "gold" files, it's still valuable to run the tests before and after a code change. That way you can at least ensure yourself that your changes didn't cause any regressions to the code before checking in.
Test Standards: There are a number of coding and design standards we've been using for the tests - both for Java files (whether they're framework or utility classes, specific API test files, or test drivers for bunches of xsl stylesheets), and for the various XSL stylesheet test files we have, along with their corresponding XML data files, etc.
Some tests are ones that Xalan has not passed to date, but we know the correct ("gold") result by analysis or by trying the test on other processors. A number of tests may also be missing matching "gold" files, if we haven't yet had time to confirm the correct output. It's still useful to run these tests (although the ConformanceTest driver will report an AMBG or 'Ambiguous' result) because you can still see if the output looks basically correct, and compare the output to previous test runs before your code changes, etc. At Lotus/IBM, we have some additional test cases that we can't give away, so a patch may pass all these tests and fail others, but those cases should be rare.
Test cases in the "conf" group will ultimately be submitted to OASIS as part of their vendor-independent project on XSLT/XPath Conformance Testing. For more information about this project, visit http://www.oasis-open.org/committees/xslt/index.html
The project will combine test cases from different sources and provide a way to customize the set of tests according to design decisions made for each processor under test. We expect that when the OASIS test system is delivered, it will supplant most or all of the "conf" group provided here. Conformance tests are designed as streamlined "atomic" (or at least "molecular") tests, so there are several hundred of them. While
You are invited to submit additional test cases by checking them into the "contrib" area or mailing them to David_Marston@lotus.com. If you want to help with testing Xalan and wish to be assigned to write some cases, send email to David Marston stating your interest. Contributed tests may be sent along to the OASIS conformance project if they test conformance. Guidelines for comments in tests are still evolving as part of the OASIS project, so it may be necessary to add more comments or other annotation after the test is submitted.
(This builds both the test harness/framework/etc. and the specific Java API test files) Works the same for DOS/Windows or UNIXes
1: Checkout the whole xml-xalan repository locally
e:\builds\xml-xalan
/home/builds/xml-xalan
2: Set an environment variable JARDIR to point to the directory where you have all applicable jars.
Be sure that you delete the testxsl.jar there, since you want to rebuild a new one!
Required: xalan.jar, xerces.jar (or your parser's jar), testxsl.jar , bsf.jar,
js.jar (see Xalan-J doc for how to get this from Mozilla)
set JARDIR=e:\builds\myjars
export JARDIR=/home/builds/myjars
3: Copy all product jars to the aforementioned JARDIR, obviously
4: cd to the test directory and be safe: cleanup the tests first
cd e:\builds\xml-xalan\test\java
build.bat clean
cd /home/builds/xml-xalan/test/java
build.sh clean
5: Build the tests appropriate to your product
Xalan-J 1.x:
build.bat package.xalan
build.sh package.xalan
Xalan-J 2.x:
build.bat package.trax
build.sh package.trax
This will create build\testxsl.jar, which you should manually copy into your JARDIR set above before running any tests.
Note that ProcessorWrapper subclasses for XT and SAXON are currently checked in both as .java source files and as precompiled .class files - the .class files are merely copied into the jar by default, so you don't need the other processors on the classpath when building the jar. These classes aren't strictly necessary unless you're running tests with those flavors.
1: Build a fresh copy of testxsl.jar as described above,
including setting the environment variable JARDIR and putting all the needed .jar files
there, including testxsl.jar
2: cd xml-xalan\test
3: (either) Run Any of the 'convenience' batch files provided:
traxapitest.bat TRAXAPITestClassName [options] (runs TRAX interface tests with Xalan-J 2.x)
(equivalent to runtest trax.TRAXAPITestClassName -load APITest.properties [options]
xalanj1apitest.bat ParamTest [options] (runs Xalan-J 1.x API tests)
(equivalent to runtest xalanj1.ParamTest -load APITest.properties [options]
contribtest.bat [options] (runs ConformanceTest driver over contrib test tree)
runtest.bat end_pkg.AnyTestClassName [options] (see batch file for comments) This is
a generic way to run any tests - it assumes org.apache.qetest as the package; you may
need to provide the last part of the end_pkg name as well as the classname - as well
as providing any extra arguments for the test too.
3: (or) Run java.exe and the desired test class yourself:
Note: the above batch and shell files, and the JARDIR environment variable, are simply conveniences for running the tests. The Java API tests and test drivers can also be executed by hand, for those who wish to manage their own classpaths and/or simply pass all arguments to the tests on the command line, etc.
Sorry! We don't have .sh equivalents for all the convenience .bat files - submissions of ports of these files are welcomed!
Most tests can either accept options from a properties file, via:
TestName -load file.properties
or simply from the command line (which ovverrides the properties file) like:
TestName -arg1 value1 -arg2 -arg3 value3
To see all options, call a test with an illegal argument to force it to print out it's .usage().
For another description of options, see xml-xalan\test\ContribTest.properties, which describes most of them as used in the context of running the ConformanceTest driver over the xml-xalan\tests\contrib tree of stylesheets. Simply change inputDir and goldDir to run over a different set of files (like a conformance test suite, which we hope to add soon).
-load file.properties (read in a .properties file, that can set any/all of the other opts)-logFile resultsFileName sends test results to file-inputDir path to tests input *.xml/*.xsl files-outputDir path to output area - where all output is sent-goldDir path to gold reference output-category name of single subdir within testDir to run-excludes 'test1.xsl;test2.xsl' (will skip running any specifically named tests)-flavor xalan|trax|trax.d2d - which kind/flavor of Processor to test>-noReuse (will force recreate processor each time)-debug (prints extra test debugging info everywhere)-precompile (will use precompiled stylesheet, if applicable)-noErrTest (will skip running 'err' tests, if applicable)Use the existing framework! It provides lots of useful functionality, and will help us to maintain your tests.
To write new TRAX/javax.xml.transform API tests:
- open org.apache.qetest.trax.REPLACE_template_for_new_tests.java
- follow directions to rename the file (and put in the correct dom/sax/stream subdir, if needed) and search-and-replace all REPLACE_* tokens
- write one-time-only setup code in testFileInit()
- write a number of testCasen methods. Each one should be independent from the other test cases. Try to test between one and ten or so individual test points (or calls to reporter.check(...) for each testCase method.
- Never use System.out/.err - always use reporter.log*Msg() (to report general messages), or reporter.check() (to validate a specific test point)
- Build the tests, including your new one, as described
- Put your test's supporting xml/xsl files in xml-xalan/test/tests/api/trax or subdirectories
- Use xml-xalan\test\traxapitest.bat (and APITest.properties) to run your test! Results will be placed by default into xml-xalan\test\results-api\APITest.xml
You can pretty-print the results by using the viewResults.xsl stylesheet to turn the XML into an HTML format.
TBD - in progress. A few C++ API tests are checked into the xml-xalan\c repository area already. To execute any set of 'conformance' tests with the Xalan-C processor, we currently use the org.apache.qetest.xsl.CConformanceTest driver. This is written in Java to take advantage of the framework and results reporting, but basically constructs a command line for each test and then shells out to 'TestXSLT.exe -in file.xsl...' to run the test; it then uses the same validation routines as the Java ConformanceTest.
inputDir to iterate over (using ConformanceDirRules/ConformanceFileRules),
or an explicit -fileList. Processes all files using a specific
-flavor of a ProcessorWrapper, so identical test runs can be done
using different processors (eg. -flavor xalan = XalanWrapper = Xalan-J 1.x;
-flavor trax = TraxWrapper = Trax interface using streams by default;
-flavor trax.d2d = TraxWrapper = Trax interface using DOMs)
-iterations
parameter to iterate over each file a bunch of times to get average timing data