Go to the previous, next section.

# Testing Framework for ILU

This document describes a framework for testing ILU in various configurations, written in Python.

## Introduction

There are two components to this testing framework. The first is the individual test script, written in Python. Each script can make use of a set of Python classes which encapsulate standard testing behavior, and have access to the configuration information for ILU. The classes provided encapsulate things like a `ClientServerTest`, which runs a server, then runs a client against the server, and looks for an error-free completion of the client. The testing framework takes care of things like establishing a temporary binding directory, or running a temporary simple binding server.

The second major element is the automated framework that builds ILU and runs all the test scripts. This component reads descriptions of ILU configurations from a configuration description file, then builds and installs an ILU tree, either from an ILU tar file, or from an RCS tree, then runs the test scripts in the various ``example'` subdirectories of the installed ILU, once for every configuration tested.

## Test Scripts

Typically, there are zero or one test scripts in each subdirectory of ``ILUSRC/examples/'`. However, there may be more than one in a directory. These scripts are installed into the ``ILUHOME'` tree during the `make Install` step, and are used for testing the installed examples.

Each script is written in Python, and will be invoked as script input to the Python interpreter. Typically, a script creates one or more instances of a test object, an object which at its most typical binds together the running of a client and server. After creating a number of instances, the script will run each test object, which causes the test to be run. If all of the test objects run successfully, the script exits with a status of 0; if any of the tests exits unsuccessfully, it should exit with a non-zero status.

### Python Classes and Data Structures Provided

The test scripts may make use of a number of data structures and classes defined in the Python module ``ILUTesting``. The simplest way to access them in a testing script is to `import ILUTesting` at the top of your script.

A number of test classes are defined. They are all subclasses of the abstract class `ILUTestingMachinery.Test`. Each test should be defined by creating an instance of one of these classes. The test can then be run by calling the `run` method on the test instance (though this is usually left to the function ``run_tests``). The following kinds of tests can be defined:

- `SimpleClientServerTest (testname, client command, server command)` -- this object encapsulates the normal client server test. The server is started by executing the server command in a subprocess; a pause then ensues, for the server to start up; the client is then run. If the client exits with a zero status, the test is successful. If it exits with a non-zero status, the test is unsuccessful, and the log files of both the server and client are written to standard output. This object has two instance variables, `server_startup_delay`, which defaults to `5` (seconds), and `server_shutdown_delay`, which defaults to `0` seconds. The first is tthe time after starting a server that the code will wait before attempting to run the client. The second is the time that the system will wait after killing a server, before doing anything else. Both values may be changed after the instantiation of the object if a longer delay is required for a specific test.
- `ClientSBHServerTest (testname, client command, server command)` -- this class embodies a client-server test where the server explicitly outputs a string binding handle to standard output, and the client then uses this SBH to bind to the server. The client command may use the string `"$SBH$"` in its arguments, and that string will be replaced by the generated SBH before the client command is executed. This object type also supports the `server_startup_delay` and `server_shutdown_delay` instance variables, as described for `SimpleClientServerTest`.
- `SimpleSingleProgramTest (testname, command [, environment])` -- this object type encapsulates a single program, which should run and exit with a zero status. The command used to start the program is given with command; optionally, an environment may be specified which can be used to augment or override the default setting of various environment variables when the program is executed.
- `MultiClientMultiServerTest (testname, client commands, server commands)` -- this object captures a test where several servers, or several clients, may have to be run. It will start all of the servers, then run each of the clients sequentially. If all clients complete successfully, the test terminates successfully. If any client terminates unsuccessfully (exits with a non-zero status), the test itself terminates unsuccessfully. This test also supports the `server_startup_delay` and `server_shutdown_delay` instance variables, but they apply to each of the servers of the test.

Several dictionaries of configuration information are also available to the test script:

- `iluconf_dict` contains the contents of the file ``ILUHOME/imake/iluconf.h'`, which is mainly configuration options and settings for the C compilations. Each macro defined in that file with a `#define` construct is mapped to a key-value pair, with the macro name being the key, and the macro expansion being the value. This can be tested for various optional attributes; for example, you can check to see if a particular protocol has been configured into ILU.
- `iludefs_dict` contains the contents of the file ``ILUHOME/imake/ilu.defs.new'`, mapped as for `ILUTesting.iluconf_dict`, with only those constructs defined as C macros, with `#define` statements, being mapped to key-value pairs. Note that any constructs defined in ``ilu.defs.new'` with the form `KEY = VALUE` will not appear in `iludefs_dict`. This can be used to test for configuration attributes which do not appear in the ``iluconfs.h'` file.
- `makefile_dict` contains either the values defined in the ``Makefile'` in the current directory with the construct `KEY = VALUE`, or the values defined in in ``ILUHOME/imake/ilu.defs.new'`, if no ``Makefile'` exists in the current directory. Note that the contents of the Makefile have already been passed through the C pre-processor, so the values here are the actual values used in the build phase of ILU. This can be used to check for various values, such as the location of a particular library.

A number of strings, which abstract commands for running a particular scripting language interpreter, are exported:

- java_command -- an invocation of the Java interpreter;
- python_command -- an invocation of the Python interpreter;
- lisp_command -- an invocation of the Common Lisp interpreter;
- guile_command -- an invocation of the Guile interpreter

The boolean value `ilu_threaded` will be TRUE if ILU has been built with support for threads.

Finally, the function ``run_tests``, which takes a list of test objects as an argument, will run through the tests one at a time, printing the name and completion status of the test to standard output. If a test terminates unsuccessfully, the logs of the client and server will be copied to the standard output.

### Writing a Test Script

Most of the testing scripts simply create a list of test objects (instances of ``ILUTesting.SimpleClientServerTest`` or ``ILUTesting.SimpleSingleProgramTest`` or ``ILUTesting.ClientSBHServerTest`` or ``ILUTesting.MultiClientMultiServerTest``), then call the Python routine ``ILUTesting.run_tests``, which takes a list of test objects, and runs them, one after another. When a test terminates unsuccessfully, the logs of the client and server are written to standard output. Here's an example:

    # we import a number of symbols from ILUTesting, including
    # "run_tests", "iluconf_dict", "python_command",
    # and "SimpleClientServerTest"
    from ILUTesting import *

    tests = []

    if iluconf_dict.has_key("ILU_CORBA_PYTHON_MAPPING"):
    	tests.append(SimpleClientServerTest ("excn: Python (with CORBA mapping)",
    				             (python_command, "clientCORBA.py",),
    				             (python_command, "serverCORBA.py"))
    	tests.append(SimpleClientServerTest ("excn: Python (with CORBA mapping, threaded)",
    				             (python_command, "clientCORBA.py", "-mt",),
    				             (python_command, "serverCORBA.py", "-mt"))
    else:
    	# the ILU mapping has no provision for raising system exceptions, so
    	#  pass the -nosys flag to the client to tell it not to test them
    	tests.append(SimpleClientServerTest ("excn: Python (with ILU mapping) (no system exceptions)",
    				             (python_command, "client.py", "-nosys",),
    				             (python_command, "server.py"))

    if __name__ == "__main__":
    	run_tests(tests)

The scripts may however execute arbitrarily complex Python code, if necessary. An example of a more complicated testing script can be found in ``ILUSRC/examples/test1/Test.py'`; this defines and uses a Python function and relies heavily on configuration tests.

### Imake Considerations

The imake macro `PythonTestScript` is provided to define testing scripts. It takes two arguments. The first is the name of the file containing the test script. The second is a list of dependencies; files that must exist or be brought up to date before the test script can be run. The expansion of `PythonTestScript` will define the test script as a target for `"make ptest"`, which will cause the dependent files to be built, then run the Python interpreter on the test script. So, suppose the testing script ``Test.py'` ran various combinations of the two programs `server` and `client`. You would add a line to the ``Imakefile'` (or, in an ``examples'` subdirectory, the ``runImakefile'`), like this:

    PythonTestScript(Test.py, client server)

which says that (1) the programs `client` and `server` must exist before running ``Test.py'`, and (2) that "make ptest" in this directory should run ``Test.py'`.

## The Automated Testing Framework

There are two major elements to this framework, the configurations file and the program `run-ilu-tests`. `run-ilu-tests` reads the configurations file, and then builds ILU and tests it with one or more configurations.

### Configurations Files

A configurations file has the general syntax of a sequence of Python dictionary literals. That is, it consists of a left parenthesis, followed by zero or more comma-separated dictionary literals specifying testing configurations:

    ( test-config-1, test-config-2, ... )

Each dictionary literal begins with a left brace, ends with a right brace, and contains a number of key-value pairs, separated by commas. A colon character separates each key from its associated value. Each key must be a quoted string. Each value is either a quoted string, a sequence of quoted strings, or another dictionary literal. The following keys must be defined:

- `"name"` -- the name of the testing configuration, as a quoted string

The following keys may also be defined:

- `"copy-ilu-tree"` -- specifies that the sources should be checked out of the PARC ILU source RCS tree using the program `/project/rpc/tools/copy-ilu-tree`. The associated value for this key should be a quoted string giving the version argument for the `copy-ilu-tree` program, such as `"2.0alpha15"`.
- `"enables"` -- in the PARC environment, ``/import'` or ``/project'` packages to enable before configuring the sources. The value for this key should be a comma-separated, parenthesis-enclosed, list of quoted strings.
- `"config-switches"` -- switches to pass to the `ILUSRC/imake/configure` program when configuring the sources. The value for this key should be a comma-separated, parenthesis-enclosed, list of quoted strings.
- `"environment"` -- a python dictionary of key-value pairs, used to augment and override the default set of environment variable settings when the configuration of the sources is done. Each key and value should be a quoted string.
- `"tarfile"` -- file to unpack the ILU sources from. This key cannot be used when the `copy-ilu-tree` key is used. The value is a quoted string giving the file name.
- `"patchfile"` -- used in conjunction with `tarfile`, this key indicates a patch file to be applied to the sources after unpacking them. The value is a quoted string giving the file name.
- `"directory"` -- specifies directory to build the ILU build in. The value is a quoted string giving the name of the directory file.

Here's an example of a configuration:

    {"name" : "basic-solaris-2",
      "copy-ilu-tree" : "2.0alpha15",
      "enables" : ("sunpro-4.2",
    	       "python-1.5",
    	       "franz-4.3",
    	       "java/jdk-1.2",
    	       "texinfo-3.11",
    	       "bison-1.25",
    	       "flex-2.5.3",
    	       "ghostscript-5.03",
    	       "tex",
    	       "pbmplus",
    	       "guile-1.2"),
      "config-switches" : ("--with-x=/usr/openwin/include",
    		       "--enable-http-protocol",
    		       "--enable-w3mux-transport",
    		       "--enable-cplusplus-support",
    		       "--with-cplusplus-libs=/project/sunpro-4.2/SUNWspro/lib/libC.so",
    		       "--enable-w3ng-protocol",
    		       "--enable-w3mux-transport",
    		       "--enable-fixed-point-support",
    		       "--enable-version-2-type-uids",
    		       "--enable-new-keywords-plain",
    		       "--enable-w3ng-relative-ih-passing"),
      "environment" : {
    	  "CC" : "/project/sunpro-4.2/SUNWspro/bin/cc -xs -Xt -v",
    	  "PATH" : "/project/rpc/tools:/usr/ccs/bin:/usr/openwin/bin:/usr/bin:/bin" },
      }

### The `run-ilu-tests` Script

The program `run-ilu-tests` is used to invoke the automated test programs. This program uses the testing script classes discussed in the previous section, and is installed in the ``ILUHOME/bin'` directory as part of the normal ILU installation process. This creates a bit of a bootstrapping situation; you must have successfully built and installed ILU, with Python support, in order to have access to `run-ilu-tests`.

Basically, `run-ilu-tests` reads a configuration file, then builds and tests ILU one or more times, depending on the command-line options specified. The syntax for the invocation of `run-ilu-tests` is

    % run-ilu-tests [command-line switches]

Output from the tests, if any, will be sent to standard output.

The following command-line switches are available:

- -dir DIRECTORY -- Build ILU in ``DIRECTORY/src'`, and install in ``DIRECTORY'`. If no directory is specified, the directory specified in the configuration will be used. If no directory is specified in the configuration, a temporary directory will be used.
- -verbose -- Send messages to standard output on what's happening periodically.
- -tarfile TARFILE -- Use the specified TARFILE as the source for ILU. Automatically `ungzip` the file if the extension is ``.tar.gz'` instead of just ``.tar'`. Overrides any configuration specification.
- -patchfile FILE -- Apply the specified FILE as a patch to the sources after unpacking them. Can only be used in conjunction with -tarfile. Overrides any configuration specification.
- -copy-ilu-tree VERSION-STAMP -- Unpack the sources using `/project/rpc/tools/copy-ilu-tree`, using VERSION-STAMP as the stamp for the command. Only available at PARC.
- -configsfile FILE -- Read this file as the configuration file.
- -temproot DIRECTORY -- If no directory is specified, and a temporary directory is being used, create it under this DIRECTORY.
- -config CONFIG-NAME -- Test the configuration indicated by CONFIG-NAME, which specified one of the configurations defined in the configuration file specified by -testfile. A number of configurations to test may be specified on the same command line; they will be tested in the order in which they are specified on the command line. Each test will be against a fresh build and install of ILU.
- -doall -- Test all of the configurations, even if one of them produces an error. The default is to halt at the first error; specifying this switch will cause the erroring configuration to be passed over and for the other configurations to be tested. `run-ilu-tests` will exit with a non-zero error status if any of the configurations revealed an error.
- -noremove -- Leave the built and tested configuration or configurations in place. Normally they are removed if the tests are all successful.

Here's an example of running `run-ilu-tests` successfully:

    % run-ilu-tests -configsfile /project/rpc/miscdoc/ILU/ilu-testing-configs
        -temproot /usr/tmp -config basic-solaris-2 -verbose
    Reading configuration file /project/rpc/miscdoc/ILU/ilu-testing-configs
    +++ Configuration 'basic-solaris-2'
    Starting at Mon Jun  7 15:27:30 1999
    Unpacking sources into /usr/tmp/usr/tmp/@774.1...
    Using 'copy-ilu-tree -noconfirm 2.0alpha15' to unpack...
    Enabling sunpro-4.2 python-1.5 franz-4.3 java/jdk-1.2 texinfo-3.11
     bison-1.25 flex-2.5.3 ghostscript-5.03 tex pbmplus guile-1.2...
    Configuring with
      --with-x=/usr/openwin/include
      --enable-http-protocol
      --enable-w3mux-transport
      --enable-cplusplus-support
      --with-cplusplus-libs=/project/sunpro-4.2/SUNWspro/lib/libC.so
      --enable-w3ng-protocol
      --enable-w3mux-transport
      --enable-fixed-point-support
      --enable-version-2-type-uids
      --enable-new-keywords-plain
      --enable-w3ng-relative-ih-passing...
    deleting config script file /usr/tmp/@774.3...
    Doing make Install...
    Doing make Ptest in examples subdirectory...
    Done at Mon Jun  7 16:07:06 1999.
    %

Each of the different steps here is done in a subprocess, with output, both regular and error, going to a log file. If the step completes successfully, the log file is deleted. If it completes abnormally, the log file is sent to standard output of the main process.

Go to the previous, next section.