ViewVC Help
View File | Revision Log | Show Annotations | Download File | Root Listing
Revision: 1.3
Committed: Fri Dec 26 17:00:32 2008 UTC (15 years, 6 months ago) by schorsch
Content type: text/plain
Branch: MAIN
CVS Tags: rad4R2P2, rad5R0, rad4R2, rad4R1, rad4R0, rad4R2P1
Changes since 1.2: +14 -3 lines
Log Message:
Invoke test suite from SCons and installation instructions.

File Contents

# Content
2 Radiance Testing Framework
3 --------------------------
5 A toolkit to test all (eventually) components of the Radiance
6 synthetic image generation system for conformance to their
7 specification.
10 Limitations
12 For the moment, we use PyUnit to run our tests. This means that
13 we're restricted to test only complete programs, and not actual
14 units (since PyUnit was designed to test Python units, not C).
15 A C-level testing framework may be added later.
17 There's no good way to automatically test GUI programs like
18 rview. We have to rely on good human testers to check whether
19 those work correctly or not.
22 Requirements
24 You need a working installation of Python 2.1 (or newer) on your
25 system. The reason for this is that the PyUnit framework isn't
26 included with earlier versions. If you prefer to use an older
27 Python (back to 1.5.2), you can get PyUnit here, and install it
28 somewhere on your PYTHONPATH:
31 Our testing framework currently assumes that the Radiance files
32 reside in the following local file tree (seen from the "test/"
33 subdirectory where this file resides):
35 executables: ../bin/*[.exe]
36 support files: ../lib/*
37 data files: ./test data/*
39 This is the location where the SCons build system places
40 everything, which means we're testing the software after building
41 but before installing.
42 The space character in the name of the test data directory is
43 deliberate, because it is a design requirement that all our
44 executables can handle path names with spaces.
47 How to run tests
49 The simplest way to run tests is to use the SCons build system.
50 The file ray/INSTALL.scons explains the requirements and details.
51 Once you have SCons working, go to the ray directory and type
53 $> scons test
55 which will automatically execute all available tests in the
56 correct environment.
58 You can also run the tests manually:
60 On unix systems, just type "" in this directory to
61 test everything. If that file doesn't have execute rights, you
62 can supply it to the python interpreter as its single argument:
63 "python". You can also run individual test suites from
64 the "py_tests" directory directly: "python".
66 On Windows, this should usually work as well. As an alternative,
67 use the "winrun.bat" script. WARNING: You need to change the
68 paths in this script to match your Python installation first.
71 What gets tested
73 There are several test groups, each containing a number of test
74 suites, each containing one or more tests. When running tests,
75 the name of the test groups and test suites will get printed to
76 the console, the latter with an "ok" if they succeeded.
78 If any test fails, there will be diagnostic output about the
79 nature of the failure, but the remaining tests will continue to
80 be executed. Note that several utility programs may be used to
81 access the results of other calculations, so if eg. getinfo is
82 broken, that may cause a number of seemingly unrelated tests to
83 fail as well.
86 How to report failures
88 If any of the tests fail on your platform, please report your
89 results (and as much ancilliary information about your system and
90 Radiance version as possible) to the radiance code development
91 mailing list on
92 The developers will then either try to fix the bug, or instruct
93 you on how to refine your testing to get more information about
94 what went wrong.
97 How to contribute test cases
99 The list of tests run is still very much incomplete, but will
100 hopefully grow quickly. You can contribute by creating tests too!
101 Please ask on the code development mailing list first, so that we
102 can avoid overlaps between the work of different contributors.
104 There are two classes of tests to be considered:
106 - Testing individual executables
107 This means that an individual program like ev, xfom, or getinfo
108 is tested with typical input data, and the output is compared
109 against the expected result.
111 - Testing specific calculations
112 This will mainly affect the actual simulation programs rpict
113 and rtrace. For example, there should be a test suite for every
114 material (and modifier) type, which uses rtrace to shoot a
115 series of rays against a surface under varying angles, in order
116 to verify material behaviour under different parameters. Tests
117 of this kind may require a custom script.
119 Contributed tests can be of two kinds. In the simplest case, you
120 can contribute a small(!) set of test data, the command line(s)
121 used to run your tests on them, and a list of expected results.
122 Result comparisons are typically done in text form (by line).
123 If the result is a picture, we'll use ttyimage to pick out a few
124 scan lines for comparison (the image dimensions must be less than
125 128 pixels). Other binary data needs to be converted into a
126 suitable text representation as well. If you're not sure what to
127 use, the developers can help you about that point. They will then
128 also wrap your test case into a Python module for integration
129 with the framework.
131 Contributors sufficiently familiar with the Python programming
132 language and the PyUnit test framework can also submit complete
133 test suites in Python. Please use the existing tests in the
134 "py_tests" directory as a template, and check out the helper
135 modules in "py_tests/unit_tools".
137 In any case, please note that we can't use any shell scripts or
138 similar tools in our tests. All tests should be able to run on
139 all supported platforms, where your favourite shell may not be
140 available. The Python programming language is available for
141 pretty much any platform, so we decided to use only that.