1 |
schorsch |
1.1 |
|
2 |
|
|
Radiance Testing Framework |
3 |
|
|
-------------------------- |
4 |
|
|
|
5 |
|
|
A toolkit to test all (eventually) components of the Radiance |
6 |
|
|
synthetic image generation system for conformance to their |
7 |
|
|
specification. |
8 |
|
|
|
9 |
|
|
|
10 |
|
|
Limitations |
11 |
|
|
|
12 |
schorsch |
1.4 |
We use the Python unittest module to run our tests. This means |
13 |
|
|
that we're currently restricted to test only complete programs, |
14 |
|
|
and not actual units (since unittest was designed to test Python |
15 |
|
|
units, not C). A C-level testing framework may be added later. |
16 |
schorsch |
1.1 |
|
17 |
schorsch |
1.2 |
There's no good way to automatically test GUI programs like |
18 |
|
|
rview. We have to rely on good human testers to check whether |
19 |
|
|
those work correctly or not. |
20 |
|
|
|
21 |
schorsch |
1.1 |
|
22 |
|
|
Requirements |
23 |
|
|
|
24 |
schorsch |
1.4 |
You need a working installation of Python 2.7 or 3.x on your |
25 |
|
|
system. Radiance must be either built with the executables still |
26 |
|
|
in the source tree (preferrable to test before installing), or as |
27 |
|
|
a live installation. |
28 |
schorsch |
1.1 |
|
29 |
|
|
|
30 |
|
|
How to run tests |
31 |
|
|
|
32 |
schorsch |
1.3 |
The simplest way to run tests is to use the SCons build system. |
33 |
|
|
The file ray/INSTALL.scons explains the requirements and details. |
34 |
|
|
Once you have SCons working, go to the ray directory and type |
35 |
|
|
|
36 |
schorsch |
1.4 |
$> scons build |
37 |
schorsch |
1.3 |
$> scons test |
38 |
|
|
|
39 |
schorsch |
1.4 |
The first command will build Radiance, and place the executables |
40 |
|
|
in a platform-specific directory below ray/scbuild/. |
41 |
|
|
The second command will automatically execute all available tests |
42 |
|
|
in the environment created by the build. |
43 |
|
|
|
44 |
|
|
Other build systems may chose to integrate the tests in a similar |
45 |
|
|
way. The file "run_tests.py" can either be invoked as a script or |
46 |
|
|
imported as a module. Note that in either case, you probably need |
47 |
|
|
to supply the correct paths to the Radiance binaries and library. |
48 |
|
|
|
49 |
|
|
As a script: |
50 |
|
|
|
51 |
|
|
usage: run_tests.py [-V] [-H] [-p bindir] [-l radlib] [-c cat] |
52 |
|
|
|
53 |
|
|
optional arguments: |
54 |
|
|
-V Verbose: Print all executed test cases to stderr |
55 |
|
|
-H Help: print this text to stderr and exit |
56 |
|
|
-p bindir Path to Radiance binaries |
57 |
|
|
-l radlib Path to Radiance library |
58 |
|
|
-c cat Category of tests to run (else all) |
59 |
|
|
|
60 |
|
|
As a module: |
61 |
|
|
|
62 |
|
|
Call the class run_tests.RadianceTests(...) with suitable arguments: |
63 |
|
|
bindir=[directory ...] - will be prepended to PATH during tests |
64 |
|
|
radlib=[directory ...] - will be prepended to RAYPATH during tests |
65 |
|
|
cat=[category ...] - only test those categories (else TESTCATS) |
66 |
|
|
V=False - if True, verbose listing of executed tests |
67 |
|
|
|
68 |
|
|
Both methods will run all the tests, or just the category given |
69 |
|
|
as the value of the "cat" argument. |
70 |
schorsch |
1.1 |
|
71 |
|
|
|
72 |
|
|
What gets tested |
73 |
|
|
|
74 |
schorsch |
1.4 |
There are several test categories, each containing a number of test |
75 |
|
|
suites, each containing one or more tests. When running tests, each |
76 |
|
|
test category will be printed to the console. Depending on the |
77 |
|
|
settings, the individual test cases may also be listed, or just |
78 |
|
|
indicated with a dot. And last the total results for each category are |
79 |
|
|
shown. |
80 |
schorsch |
1.1 |
|
81 |
|
|
If any test fails, there will be diagnostic output about the |
82 |
|
|
nature of the failure, but the remaining tests will continue to |
83 |
|
|
be executed. Note that several utility programs may be used to |
84 |
|
|
access the results of other calculations, so if eg. getinfo is |
85 |
|
|
broken, that may cause a number of seemingly unrelated tests to |
86 |
|
|
fail as well. |
87 |
|
|
|
88 |
|
|
|
89 |
|
|
How to report failures |
90 |
|
|
|
91 |
|
|
If any of the tests fail on your platform, please report your |
92 |
|
|
results (and as much ancilliary information about your system and |
93 |
schorsch |
1.2 |
Radiance version as possible) to the radiance code development |
94 |
|
|
mailing list on http://www.radiance-online.org/ |
95 |
schorsch |
1.1 |
The developers will then either try to fix the bug, or instruct |
96 |
|
|
you on how to refine your testing to get more information about |
97 |
|
|
what went wrong. |
98 |
|
|
|
99 |
|
|
|
100 |
|
|
How to contribute test cases |
101 |
|
|
|
102 |
schorsch |
1.4 |
The selection of tests to run is still very much incomplete, but |
103 |
|
|
will hopefully grow over time. You can contribute by creating |
104 |
|
|
tests too! Please ask on the code development mailing list first, |
105 |
|
|
so that we can avoid overlaps between the work of different |
106 |
|
|
contributors. |
107 |
schorsch |
1.1 |
|
108 |
|
|
There are two classes of tests to be considered: |
109 |
|
|
|
110 |
|
|
- Testing individual executables |
111 |
|
|
This means that an individual program like ev, xfom, or getinfo |
112 |
|
|
is tested with typical input data, and the output is compared |
113 |
|
|
against the expected result. |
114 |
|
|
|
115 |
|
|
- Testing specific calculations |
116 |
|
|
This will mainly affect the actual simulation programs rpict |
117 |
|
|
and rtrace. For example, there should be a test suite for every |
118 |
|
|
material (and modifier) type, which uses rtrace to shoot a |
119 |
|
|
series of rays against a surface under varying angles, in order |
120 |
|
|
to verify material behaviour under different parameters. Tests |
121 |
|
|
of this kind may require a custom script. |
122 |
|
|
|
123 |
|
|
Contributed tests can be of two kinds. In the simplest case, you |
124 |
|
|
can contribute a small(!) set of test data, the command line(s) |
125 |
|
|
used to run your tests on them, and a list of expected results. |
126 |
|
|
Result comparisons are typically done in text form (by line). |
127 |
|
|
If the result is a picture, we'll use ttyimage to pick out a few |
128 |
|
|
scan lines for comparison (the image dimensions must be less than |
129 |
|
|
128 pixels). Other binary data needs to be converted into a |
130 |
|
|
suitable text representation as well. If you're not sure what to |
131 |
schorsch |
1.4 |
use, the developers will be happy to assist you. They will then |
132 |
schorsch |
1.1 |
also wrap your test case into a Python module for integration |
133 |
|
|
with the framework. |
134 |
|
|
|
135 |
|
|
Contributors sufficiently familiar with the Python programming |
136 |
schorsch |
1.4 |
language and the unittest framework can also submit complete |
137 |
|
|
test suites in Python. Please use the existing tests below the |
138 |
|
|
"testcases" directory as a template, and check out the helper |
139 |
|
|
modules in ".../lib/pyradlib" (where ".../lib" is the location of |
140 |
|
|
the Radiance support library). |
141 |
|
|
|
142 |
|
|
Also note the pseudo-builtin module "testsupport" temporarily |
143 |
|
|
created by the RadianceTests() class (see the docstrings there), |
144 |
|
|
which provides information about the various required directory |
145 |
|
|
locations. |
146 |
|
|
|
147 |
|
|
And lastly you'll find that we have deliberately included a space |
148 |
|
|
character in the name of the "test data" directory, because it is |
149 |
|
|
a design requirement that all our executables can handle path |
150 |
|
|
names with spaces. |
151 |
schorsch |
1.1 |
|
152 |
schorsch |
1.4 |
In any case, remember that we can't use any shell scripts or |
153 |
schorsch |
1.1 |
similar tools in our tests. All tests should be able to run on |
154 |
|
|
all supported platforms, where your favourite shell may not be |
155 |
|
|
available. The Python programming language is available for |
156 |
|
|
pretty much any platform, so we decided to use only that. |
157 |
|
|
|
158 |
|
|
|
159 |
|
|
|