Adding test tools to DEFT

Understanding packages

Packages are gzipped tarballs of a test tool, it's configuration and data files and an instantiator script. The naming scheme is more than just a convention.

"jtest" is the package name
"SunOS" is the architecture of the package
"" is the user who created the package
"tb" is arbitrary versioning tag for the package
"02.07.14.22" is the date and time the package was made

Do NOT put extra hyphens withing components. It breaks everything!

Packages are made using the mkrelease.tcl script which is checked into the qa tools at sqa/traffic/ts_test in cvs. The Makefile.am that has example targets for building the package.

Each packages must have an instantiator script. The instantiator must be named <pkg_name>-instantiate.<arbitrary_suffix>. For jtest, the instantiator is named: jtest-instantiate.pl . Currently the suffix is not used to make any decisions within the proc_manager. The instantiator can be written in any language (although /bin/sh has limitations on Solaris that keep it from working but bash is fine) and could be compiled program instead not a script.

Writing an instantiator

The instantitator must setup test process for running. Processes are run out of a temporary area for testing know as the run_dir . The instantiator must make any necessary binaries, config files and data available from the run_dir . The location where the package was unpacked (or localpath was specified) is known as bin_dir. It's not necessary for the instantiator to copy everything to the run directory. It can symlink binaries or add command line arguments to tell the process where to find read-only data files.

Communication between the proc_manager and the instantiator happens three ways:

name value input pairs are sent to the instantiator on stdin by the proc_manager
name value output pairs from the instantiator to the proc_manager are sent on a pipe that opened by the proc_manager prior to forking the instantiator. The file descriptor for the pipe is sent as first and only command line argument to the instantiator. Stdout is not used for return information since subprocesses may send diagnostics to stdout.
The exit status of the instantitor is read by the proc_manager. The proc_manager intreprets a non-zero exit status, as failure and the creation of the instance fails.

The name value pairs both for input and ouput are of the form:

  <name>: <value>LF

The name can consist of any ascii character other than '\0', ':', CR or LF. The value can be any ascii character other than '\0', CR, or LF. However, the use of printable 7bit characters is recommend in both cases.

Input Value Names
config_file	Absolute to path the file containing "config" argument given to the create command. Not prsent if the config argument is not sent.
bin_dir	Absolute path to where the tool package was unpacked or where localpath was set to. Always sent.
run_dir	Aboslute path to where the instantiator should prepare the test process to be run from. Sent unless "no_run_dir" argument was made to the create command.
no_run_dir	Informs the instantiator that there is no run directory and the instantitor should prepare the process to be run from the "bin_dir"
ports_avail	Hyphen separated continugous range of ports available for the instantiator to allocate. The instantiator must allocate ports in order, starting with the lowest first and report the number of ports used in it's output if any ports are used.

Output Value Names
cmd_line	Command line to run the test process with starting with absolute path the test process binary. Command line arguments to the process follow. Arguments are broken up along whitespace although double quotes and standarding escaping rules can be used to create a argument contaning white space.
port_binding	White space separted list of name value pairs of which the name consists of alphanumeric characters and the port is 16 bit unsigned number in ascii. More than one pair can be given. Example: port_binding: server 12310 rafPort 12311\n rafPort is a special name which indicates the port raf_instance() will use for querying the process via RAF.
ports_used	positive interger value (in ascii) of ports used from the port range passed as an input parameter to the instantiator.
env_vars	Space separated list of environment value pairs (of the form name=value) to add to the test processes enviroment before it is execed. Standard quoting rules apply for allowing spaces in the values.

Parsing the output

All stderr and stdout output from every test process is stamped, and sent to the test_log_collate processes which merges all the streams into one file. In addition, any calls add_to_log() cause entries to be added to the log file. Parsing the log file is the ONLY place where a pass or fail determination is made for a test.

In order to facilitate easy testing writing but allow for complex parsing log, the log parser makes use of perl modules. When adding a tool to the framework, it's generally necessary to add a parsing module to it though for very simple tools the generic parser may do the job.

A parser is perl module which defines the function process_test_log_line(String $instance_id, String $level, StringRef $line). process_test_line must return one of the following three values:

"ok" - the line is neither a warning on an error
"warning" - the line indicates a warning
"error" - the line indicates an error

The easiest way to write your own parser is to use an existing one from traffic/test/parsers as template.

In order to prevent each test from having to specify a parser for each instance, instance name prefixes can be mapped to a parser by default. The prefix is all alpha characters before an numeric characters in the name. For example, if the instance name is syntest1 the prefix is syntest. The default mappings are made in parse_dispatcher.pm in the %predefined_mappings hash. If the there is no entry for the instance name prefix is %predefined_mappings, the parse_generic.pm is used for that instance.

In the cases where the default parser isn't appropriate, the test script can use set_log_parser() to override the default.

The End

Back to the index.