| 9 |
|
|
| 10 |
|
Limitations |
| 11 |
|
|
| 12 |
< |
For the moment, we use PyUnit to run our tests. This means that |
| 13 |
< |
we're restricted to test only complete programs, and not actual |
| 14 |
< |
units (since PyUnit was designed to test Python units, not C). |
| 15 |
< |
A C-level testing framework may be added later. |
| 12 |
> |
We use the Python unittest module to run our tests. This means |
| 13 |
> |
that we're currently restricted to test only complete programs, |
| 14 |
> |
and not actual units (since unittest was designed to test Python |
| 15 |
> |
units, not C). A C-level testing framework may be added later. |
| 16 |
|
|
| 17 |
|
There's no good way to automatically test GUI programs like |
| 18 |
|
rview. We have to rely on good human testers to check whether |
| 21 |
|
|
| 22 |
|
Requirements |
| 23 |
|
|
| 24 |
< |
You need a working installation of Python 2.1 (or newer) on your |
| 25 |
< |
system. The reason for this is that the PyUnit framework isn't |
| 26 |
< |
included with earlier versions. If you prefer to use an older |
| 27 |
< |
Python (back to 1.5.2), you can get PyUnit here, and install it |
| 28 |
< |
somewhere on your PYTHONPATH: |
| 29 |
< |
http://pyunit.sourceforge.net/ |
| 24 |
> |
You need a working installation of Python 2.7 or 3.x on your |
| 25 |
> |
system. Radiance must be either built with the executables still |
| 26 |
> |
in the source tree (preferrable to test before installing), or as |
| 27 |
> |
a live installation. |
| 28 |
|
|
| 31 |
– |
Our testing framework currently assumes that the Radiance files |
| 32 |
– |
reside in the following local file tree (seen from the "test/" |
| 33 |
– |
subdirectory where this file resides): |
| 29 |
|
|
| 35 |
– |
executables: ../bin/*[.exe] |
| 36 |
– |
support files: ../lib/* |
| 37 |
– |
data files: ./test data/* |
| 38 |
– |
|
| 39 |
– |
This is the location where the SCons build system places |
| 40 |
– |
everything, which means we're testing the software after building |
| 41 |
– |
but before installing. |
| 42 |
– |
The space character in the name of the test data directory is |
| 43 |
– |
deliberate, because it is a design requirement that all our |
| 44 |
– |
executables can handle path names with spaces. |
| 45 |
– |
|
| 46 |
– |
|
| 30 |
|
How to run tests |
| 31 |
|
|
| 32 |
|
The simplest way to run tests is to use the SCons build system. |
| 33 |
|
The file ray/INSTALL.scons explains the requirements and details. |
| 34 |
|
Once you have SCons working, go to the ray directory and type |
| 35 |
|
|
| 36 |
+ |
$> scons build |
| 37 |
|
$> scons test |
| 38 |
|
|
| 39 |
< |
which will automatically execute all available tests in the |
| 40 |
< |
correct environment. |
| 39 |
> |
The first command will build Radiance, and place the executables |
| 40 |
> |
in a platform-specific directory below ray/scbuild/. |
| 41 |
> |
The second command will automatically execute all available tests |
| 42 |
> |
in the environment created by the build. |
| 43 |
|
|
| 44 |
< |
You can also run the tests manually: |
| 44 |
> |
Other build systems may chose to integrate the tests in a similar |
| 45 |
> |
way. The file "run_tests.py" can either be invoked as a script or |
| 46 |
> |
imported as a module. Note that in either case, you probably need |
| 47 |
> |
to supply the correct paths to the Radiance binaries and library. |
| 48 |
|
|
| 49 |
< |
On unix systems, just type "run_all.py" in this directory to |
| 61 |
< |
test everything. If that file doesn't have execute rights, you |
| 62 |
< |
can supply it to the python interpreter as its single argument: |
| 63 |
< |
"python run_all.py". You can also run individual test suites from |
| 64 |
< |
the "py_tests" directory directly: "python test_getinfo.py". |
| 49 |
> |
As a script: |
| 50 |
|
|
| 51 |
< |
On Windows, this should usually work as well. As an alternative, |
| 67 |
< |
use the "winrun.bat" script. WARNING: You need to change the |
| 68 |
< |
paths in this script to match your Python installation first. |
| 51 |
> |
usage: run_tests.py [-V] [-H] [-p bindir] [-l radlib] [-c cat] |
| 52 |
|
|
| 53 |
+ |
optional arguments: |
| 54 |
+ |
-V Verbose: Print all executed test cases to stderr |
| 55 |
+ |
-H Help: print this text to stderr and exit |
| 56 |
+ |
-p bindir Path to Radiance binaries |
| 57 |
+ |
-l radlib Path to Radiance library |
| 58 |
+ |
-c cat Category of tests to run (else all) |
| 59 |
|
|
| 60 |
+ |
As a module: |
| 61 |
+ |
|
| 62 |
+ |
Call the class run_tests.RadianceTests(...) with suitable arguments: |
| 63 |
+ |
bindir=[directory ...] - will be prepended to PATH during tests |
| 64 |
+ |
radlib=[directory ...] - will be prepended to RAYPATH during tests |
| 65 |
+ |
cat=[category ...] - only test those categories (else TESTCATS) |
| 66 |
+ |
V=False - if True, verbose listing of executed tests |
| 67 |
+ |
|
| 68 |
+ |
Both methods will run all the tests, or just the category given |
| 69 |
+ |
as the value of the "cat" argument. |
| 70 |
+ |
|
| 71 |
+ |
|
| 72 |
|
What gets tested |
| 73 |
|
|
| 74 |
< |
There are several test groups, each containing a number of test |
| 75 |
< |
suites, each containing one or more tests. When running tests, |
| 76 |
< |
the name of the test groups and test suites will get printed to |
| 77 |
< |
the console, the latter with an "ok" if they succeeded. |
| 74 |
> |
There are several test categories, each containing a number of test |
| 75 |
> |
suites, each containing one or more tests. When running tests, each |
| 76 |
> |
test category will be printed to the console. Depending on the |
| 77 |
> |
settings, the individual test cases may also be listed, or just |
| 78 |
> |
indicated with a dot. And last the total results for each category are |
| 79 |
> |
shown. |
| 80 |
|
|
| 81 |
|
If any test fails, there will be diagnostic output about the |
| 82 |
|
nature of the failure, but the remaining tests will continue to |
| 99 |
|
|
| 100 |
|
How to contribute test cases |
| 101 |
|
|
| 102 |
< |
The list of tests run is still very much incomplete, but will |
| 103 |
< |
hopefully grow quickly. You can contribute by creating tests too! |
| 104 |
< |
Please ask on the code development mailing list first, so that we |
| 105 |
< |
can avoid overlaps between the work of different contributors. |
| 102 |
> |
The selection of tests to run is still very much incomplete, but |
| 103 |
> |
will hopefully grow over time. You can contribute by creating |
| 104 |
> |
tests too! Please ask on the code development mailing list first, |
| 105 |
> |
so that we can avoid overlaps between the work of different |
| 106 |
> |
contributors. |
| 107 |
|
|
| 108 |
|
There are two classes of tests to be considered: |
| 109 |
|
|
| 128 |
|
scan lines for comparison (the image dimensions must be less than |
| 129 |
|
128 pixels). Other binary data needs to be converted into a |
| 130 |
|
suitable text representation as well. If you're not sure what to |
| 131 |
< |
use, the developers can help you about that point. They will then |
| 131 |
> |
use, the developers will be happy to assist you. They will then |
| 132 |
|
also wrap your test case into a Python module for integration |
| 133 |
|
with the framework. |
| 134 |
|
|
| 135 |
|
Contributors sufficiently familiar with the Python programming |
| 136 |
< |
language and the PyUnit test framework can also submit complete |
| 137 |
< |
test suites in Python. Please use the existing tests in the |
| 138 |
< |
"py_tests" directory as a template, and check out the helper |
| 139 |
< |
modules in "py_tests/unit_tools". |
| 136 |
> |
language and the unittest framework can also submit complete |
| 137 |
> |
test suites in Python. Please use the existing tests below the |
| 138 |
> |
"testcases" directory as a template, and check out the helper |
| 139 |
> |
modules in ".../lib/pyradlib" (where ".../lib" is the location of |
| 140 |
> |
the Radiance support library). |
| 141 |
|
|
| 142 |
< |
In any case, please note that we can't use any shell scripts or |
| 142 |
> |
Also note the pseudo-builtin module "testsupport" temporarily |
| 143 |
> |
created by the RadianceTests() class (see the docstrings there), |
| 144 |
> |
which provides information about the various required directory |
| 145 |
> |
locations. |
| 146 |
> |
|
| 147 |
> |
And lastly you'll find that we have deliberately included a space |
| 148 |
> |
character in the name of the "test data" directory, because it is |
| 149 |
> |
a design requirement that all our executables can handle path |
| 150 |
> |
names with spaces. |
| 151 |
> |
|
| 152 |
> |
In any case, remember that we can't use any shell scripts or |
| 153 |
|
similar tools in our tests. All tests should be able to run on |
| 154 |
|
all supported platforms, where your favourite shell may not be |
| 155 |
|
available. The Python programming language is available for |
