--- ray/test/README.txt 2008/12/26 17:00:32 1.3 +++ ray/test/README.txt 2016/03/28 17:48:43 1.4 @@ -9,10 +9,10 @@ specification. Limitations -For the moment, we use PyUnit to run our tests. This means that -we're restricted to test only complete programs, and not actual -units (since PyUnit was designed to test Python units, not C). -A C-level testing framework may be added later. +We use the Python unittest module to run our tests. This means +that we're currently restricted to test only complete programs, +and not actual units (since unittest was designed to test Python +units, not C). A C-level testing framework may be added later. There's no good way to automatically test GUI programs like rview. We have to rely on good human testers to check whether @@ -21,59 +21,62 @@ those work correctly or not. Requirements -You need a working installation of Python 2.1 (or newer) on your -system. The reason for this is that the PyUnit framework isn't -included with earlier versions. If you prefer to use an older -Python (back to 1.5.2), you can get PyUnit here, and install it -somewhere on your PYTHONPATH: - http://pyunit.sourceforge.net/ +You need a working installation of Python 2.7 or 3.x on your +system. Radiance must be either built with the executables still +in the source tree (preferrable to test before installing), or as +a live installation. -Our testing framework currently assumes that the Radiance files -reside in the following local file tree (seen from the "test/" -subdirectory where this file resides): - executables: ../bin/*[.exe] - support files: ../lib/* - data files: ./test data/* - -This is the location where the SCons build system places -everything, which means we're testing the software after building -but before installing. -The space character in the name of the test data directory is -deliberate, because it is a design requirement that all our -executables can handle path names with spaces. - - How to run tests The simplest way to run tests is to use the SCons build system. The file ray/INSTALL.scons explains the requirements and details. Once you have SCons working, go to the ray directory and type + $> scons build $> scons test -which will automatically execute all available tests in the -correct environment. +The first command will build Radiance, and place the executables +in a platform-specific directory below ray/scbuild/. +The second command will automatically execute all available tests +in the environment created by the build. -You can also run the tests manually: +Other build systems may chose to integrate the tests in a similar +way. The file "run_tests.py" can either be invoked as a script or +imported as a module. Note that in either case, you probably need +to supply the correct paths to the Radiance binaries and library. -On unix systems, just type "run_all.py" in this directory to -test everything. If that file doesn't have execute rights, you -can supply it to the python interpreter as its single argument: -"python run_all.py". You can also run individual test suites from -the "py_tests" directory directly: "python test_getinfo.py". +As a script: -On Windows, this should usually work as well. As an alternative, -use the "winrun.bat" script. WARNING: You need to change the -paths in this script to match your Python installation first. + usage: run_tests.py [-V] [-H] [-p bindir] [-l radlib] [-c cat] + optional arguments: + -V Verbose: Print all executed test cases to stderr + -H Help: print this text to stderr and exit + -p bindir Path to Radiance binaries + -l radlib Path to Radiance library + -c cat Category of tests to run (else all) +As a module: + +Call the class run_tests.RadianceTests(...) with suitable arguments: + bindir=[directory ...] - will be prepended to PATH during tests + radlib=[directory ...] - will be prepended to RAYPATH during tests + cat=[category ...] - only test those categories (else TESTCATS) + V=False - if True, verbose listing of executed tests + +Both methods will run all the tests, or just the category given +as the value of the "cat" argument. + + What gets tested -There are several test groups, each containing a number of test -suites, each containing one or more tests. When running tests, -the name of the test groups and test suites will get printed to -the console, the latter with an "ok" if they succeeded. +There are several test categories, each containing a number of test +suites, each containing one or more tests. When running tests, each +test category will be printed to the console. Depending on the +settings, the individual test cases may also be listed, or just +indicated with a dot. And last the total results for each category are +shown. If any test fails, there will be diagnostic output about the nature of the failure, but the remaining tests will continue to @@ -96,10 +99,11 @@ what went wrong. How to contribute test cases -The list of tests run is still very much incomplete, but will -hopefully grow quickly. You can contribute by creating tests too! -Please ask on the code development mailing list first, so that we -can avoid overlaps between the work of different contributors. +The selection of tests to run is still very much incomplete, but +will hopefully grow over time. You can contribute by creating +tests too! Please ask on the code development mailing list first, +so that we can avoid overlaps between the work of different +contributors. There are two classes of tests to be considered: @@ -124,17 +128,28 @@ If the result is a picture, we'll use ttyimage to pick scan lines for comparison (the image dimensions must be less than 128 pixels). Other binary data needs to be converted into a suitable text representation as well. If you're not sure what to -use, the developers can help you about that point. They will then +use, the developers will be happy to assist you. They will then also wrap your test case into a Python module for integration with the framework. Contributors sufficiently familiar with the Python programming -language and the PyUnit test framework can also submit complete -test suites in Python. Please use the existing tests in the -"py_tests" directory as a template, and check out the helper -modules in "py_tests/unit_tools". +language and the unittest framework can also submit complete +test suites in Python. Please use the existing tests below the +"testcases" directory as a template, and check out the helper +modules in ".../lib/pyradlib" (where ".../lib" is the location of +the Radiance support library). -In any case, please note that we can't use any shell scripts or +Also note the pseudo-builtin module "testsupport" temporarily +created by the RadianceTests() class (see the docstrings there), +which provides information about the various required directory +locations. + +And lastly you'll find that we have deliberately included a space +character in the name of the "test data" directory, because it is +a design requirement that all our executables can handle path +names with spaces. + +In any case, remember that we can't use any shell scripts or similar tools in our tests. All tests should be able to run on all supported platforms, where your favourite shell may not be available. The Python programming language is available for