1 |
|
2 |
Radiance Testing Framework |
3 |
-------------------------- |
4 |
|
5 |
A toolkit to test all (eventually) components of the Radiance |
6 |
synthetic image generation system for conformance to their |
7 |
specification. |
8 |
|
9 |
|
10 |
Limitations |
11 |
|
12 |
For the moment, we use PyUnit to run our tests. This means that |
13 |
we're restricted to test only complete programs, and not actual |
14 |
units (since PyUnit was designed to test Python units, not C). |
15 |
A C-level testing framework may be added later. |
16 |
|
17 |
There's no good way to automatically test GUI programs like |
18 |
rview. We have to rely on good human testers to check whether |
19 |
those work correctly or not. |
20 |
|
21 |
|
22 |
Requirements |
23 |
|
24 |
You need a working installation of Python 2.1 (or newer) on your |
25 |
system. The reason for this is that the PyUnit framework isn't |
26 |
included with earlier versions. If you prefer to use an older |
27 |
Python (back to 1.5.2), you can get PyUnit here, and install it |
28 |
somewhere on your PYTHONPATH: |
29 |
http://pyunit.sourceforge.net/ |
30 |
|
31 |
Our testing framework currently assumes that the Radiance files |
32 |
reside in the following local file tree (seen from the "test/" |
33 |
subdirectory where this file resides): |
34 |
|
35 |
executables: ../bin/*[.exe] |
36 |
support files: ../lib/* |
37 |
data files: ./test data/* |
38 |
|
39 |
This is the location where the SCons build system places |
40 |
everything, which means we're testing the software after building |
41 |
but before installing. |
42 |
The space character in the name of the test data directory is |
43 |
deliberate, because it is a design requirement that all our |
44 |
executables can handle path names with spaces. |
45 |
|
46 |
|
47 |
How to run tests |
48 |
|
49 |
The simplest way to run tests is to use the SCons build system. |
50 |
The file ray/INSTALL.scons explains the requirements and details. |
51 |
Once you have SCons working, go to the ray directory and type |
52 |
|
53 |
$> scons test |
54 |
|
55 |
which will automatically execute all available tests in the |
56 |
correct environment. |
57 |
|
58 |
You can also run the tests manually: |
59 |
|
60 |
On unix systems, just type "run_all.py" in this directory to |
61 |
test everything. If that file doesn't have execute rights, you |
62 |
can supply it to the python interpreter as its single argument: |
63 |
"python run_all.py". You can also run individual test suites from |
64 |
the "py_tests" directory directly: "python test_getinfo.py". |
65 |
|
66 |
On Windows, this should usually work as well. As an alternative, |
67 |
use the "winrun.bat" script. WARNING: You need to change the |
68 |
paths in this script to match your Python installation first. |
69 |
|
70 |
|
71 |
What gets tested |
72 |
|
73 |
There are several test groups, each containing a number of test |
74 |
suites, each containing one or more tests. When running tests, |
75 |
the name of the test groups and test suites will get printed to |
76 |
the console, the latter with an "ok" if they succeeded. |
77 |
|
78 |
If any test fails, there will be diagnostic output about the |
79 |
nature of the failure, but the remaining tests will continue to |
80 |
be executed. Note that several utility programs may be used to |
81 |
access the results of other calculations, so if eg. getinfo is |
82 |
broken, that may cause a number of seemingly unrelated tests to |
83 |
fail as well. |
84 |
|
85 |
|
86 |
How to report failures |
87 |
|
88 |
If any of the tests fail on your platform, please report your |
89 |
results (and as much ancilliary information about your system and |
90 |
Radiance version as possible) to the radiance code development |
91 |
mailing list on http://www.radiance-online.org/ |
92 |
The developers will then either try to fix the bug, or instruct |
93 |
you on how to refine your testing to get more information about |
94 |
what went wrong. |
95 |
|
96 |
|
97 |
How to contribute test cases |
98 |
|
99 |
The list of tests run is still very much incomplete, but will |
100 |
hopefully grow quickly. You can contribute by creating tests too! |
101 |
Please ask on the code development mailing list first, so that we |
102 |
can avoid overlaps between the work of different contributors. |
103 |
|
104 |
There are two classes of tests to be considered: |
105 |
|
106 |
- Testing individual executables |
107 |
This means that an individual program like ev, xfom, or getinfo |
108 |
is tested with typical input data, and the output is compared |
109 |
against the expected result. |
110 |
|
111 |
- Testing specific calculations |
112 |
This will mainly affect the actual simulation programs rpict |
113 |
and rtrace. For example, there should be a test suite for every |
114 |
material (and modifier) type, which uses rtrace to shoot a |
115 |
series of rays against a surface under varying angles, in order |
116 |
to verify material behaviour under different parameters. Tests |
117 |
of this kind may require a custom script. |
118 |
|
119 |
Contributed tests can be of two kinds. In the simplest case, you |
120 |
can contribute a small(!) set of test data, the command line(s) |
121 |
used to run your tests on them, and a list of expected results. |
122 |
Result comparisons are typically done in text form (by line). |
123 |
If the result is a picture, we'll use ttyimage to pick out a few |
124 |
scan lines for comparison (the image dimensions must be less than |
125 |
128 pixels). Other binary data needs to be converted into a |
126 |
suitable text representation as well. If you're not sure what to |
127 |
use, the developers can help you about that point. They will then |
128 |
also wrap your test case into a Python module for integration |
129 |
with the framework. |
130 |
|
131 |
Contributors sufficiently familiar with the Python programming |
132 |
language and the PyUnit test framework can also submit complete |
133 |
test suites in Python. Please use the existing tests in the |
134 |
"py_tests" directory as a template, and check out the helper |
135 |
modules in "py_tests/unit_tools". |
136 |
|
137 |
In any case, please note that we can't use any shell scripts or |
138 |
similar tools in our tests. All tests should be able to run on |
139 |
all supported platforms, where your favourite shell may not be |
140 |
available. The Python programming language is available for |
141 |
pretty much any platform, so we decided to use only that. |
142 |
|
143 |
|
144 |
|