[Radiance-general] Evaluation

[Alexa I. Ruppertsberg]

Hi Rob, Jack, Martin and Christoph,

thanks very much for your replies. Together with Christoph's survey I 
have now a better understanding of the usage. I would term this more 
like a 'relative' use: how do two (or more) design solutions compare to 
each other. A will give you more light on the desk than B, e.g.

The other emerging topic seems to me that of 'nice' pictures for the 
clients. Once you have a tool with which you can simulate the building 
and can produce a picture, there is the danger that clients 
'over'-interprete the image ('this is how it's going to look like'). Or 
put the other way, the image they can see either on a CRT or in a 
printed version does not reflect truely what the building is probably 
going to look because of the limited dynamic range of the medium (CRT or 
paper).

> Good question.  Radiance itself has been the subject of many validation
> studies, 
May I challenge this? Validation studies I am aware of are (if anyone 
knows of more then please tell me):

Grynberg, A.,“Validation of Radiance”, LBID 1575, LBL Technical 
Information Department, Lawrence Berkeley National Laboratory, Berkeley, 
California, July 1989.
This technical report proved to be inaccessible for me. If anyone has a 
copy, I would be more than happy to read it.

Khodulev, A. and E. Kopylov, “Physically accurate lighting simulation in 
computer graphics software”, 6. International conference on Computer 
Graphics and Visualization, St. Petersburg, Russia, July 1-5, 1996.
http://www.keldysh.ru/pages/cgraph/articles/pals/
This is a website and describes a white-box scenario, i.e a scenario for 
which you can actually calculate the solution analytically. But how much 
is this a valid scenario for complex illumination situations?
I have my own opinion about website-only references.

Houser, K.W., D.K. Tiller, and I.C. Pasini, “Toward the accuracy of 
lighting simulations in physically based computer graphics software”, 
Journal of the Illuminating Engineering Society, 28(1), Winter 1999, 
117-129.
the conclusion is that RADIANCE does not do what it says on the package 
(I have my own opinion about this paper)

Ubbelohde, M.S. and Humann, C. “Comparative evaluation of four 
daylighting software programs”, 1998 ACEEE Summer Study on Energy 
Efficiency in Buildings Proceedings, American Council for an 
Energy-Efficient Economy, 1998.
I asked the author to send me a copy of the paper about 18 months ago 
and I am still waiting.


Mardaljevic, J., “Validation of a lighting simulation program under real 
sky conditions”, Lighting research and Technology, 27(4), 1995, 181-188.
That is what I call validation: comparison of simulation output and 
measurements from the real environment.
In my personal opinion there is no validation study apart from John 
Mardaljevic's work.

Christoph, I saw your 2001 paper with Walkenhorst in the reference 
section of the survey. Is that a validation?

Rushmeier,Ward,Piatko,Sanders,Rust, 1995,Comparing Real and Synthetic 
Images: Some Ideas About Metrics, 6th Eurographics Workshop on Rendering.
this paper appears on the Radiance site and may I cite from the paper 
(p. 3, section 1.2): 'There are clearly very high levels of uncertainty 
in the measurements made in this experiment.' Now, that paper didn't set 
out as a validation for Radiance, but tried to do soemthing else, so in 
a way it's not an appropriate reference for a validation study, but it 
could have been, if measurements would have been taken more carefully 
and compared to the real room.

Can you please tell me whose work I'm missing here? Or do we mean 
different things when we talk about 'validaton'?
I hope I haven't upset anyone, because that's not what I intended to do. 
  I am just in the pursuit of evidence and if it is out there, then 
please tell me.

>>c) what magnitude of error is acceptable for your work?
I understand that the 'unpredictability' of the sky is the biggest 
problem basically.


Thanks again,

Alexa

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dr. Alexa I. Ruppertsberg
Department of Optometry
University of Bradford
Bradford
BD7 1DP
UK
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~