[Radiance-general] Evaluation

[Rob Guglielmetti]

Hi Alexa,

> triggered by Jan's and Richard's problem of evaluating glare, I was
> wondering how the architectual & lighting community (obviously I'm not a
> member of it) uses RADIANCE.

OK, so I guess I'll go first.  But I'm not standing out here all by
myself; I'd love to hear from the rest of you...

> a) how do you know that the results you come up with 'reflect the true
> values' (I assume 'true' is +/- an error)? I agree the simulation
> results are not completely out of order in terms of luminance, otherwise
> people wouldn't use RADIANCE.

Good question.  Radiance itself has been the subject of many validation
studies, and has been proven to be quite capable of coming up with the
"true values" for most scenes, assuming valid, high-quality input. 
There's the rub though; skies are variable, and every project brings with
it new materials -- often materials unavailable for accurate sampling at
the time of the simulation.  So, often I *don't* know I'm looking at "true
values", but I do know (hope) that the values are close enough with which
to make evaluations.  Many times, we are evaluating several different
schemes, and when they are all simulated in Radiance with the same kind of
what I call "accuracy settings" -- you know, the myriad values used for
rpict & rtrace -- I know for certain that I can say scheme-a is (insert
criteria here, brightness, uniformity, whathaveyou) than scheme-b.  Often
this is all that is needed, is for Radiance to guide us in a direction
that can be explored more fully, either with Radiance or with physical
mockups.

But yes, the temptation is there, to treat the numbers generated by
Radiance as THE numbers.  I have to fight it all the time; I submit a
report showing 290 Lux on a plan, and people go "oh, this doesn't work, we
can't have more than 270 Lux there."  That's my cue to ease into the
discussion of how a mathematical model of the sky's luminance distribution
is NOT /the sky's/ luminance distribution, etc.

> b) once a buliding has been built, has anyone gone back inside the
> office they simulated and obtained measurements to compare with their
> simulation results?

Many of the validation studies do just that.  My first big project
simulated with Radiance is still under construction, but we have done
similar tests with projects simulated with Lightscape and AGI and have
been generally pleased with the outcome.  Typically, the light levels are
not the same, but neither is the real space as compared to the simulation
model.  But the values are all in the ballpark and the clients have been
happy. Indeed, the last big museum project I did with Lightscape at my
previous firm was astonishingly accurate, I believe the light levels on
the day my boss measured them were within 5% of the caluclation.  But I
also know a thing or two about luck.  I don't tell clients to expect 5%
accuracy and neither should you.  Barring luck, the only way to get that
close is to do a simulation with measured sky data (and take readings of
the space under that same sky that you are measuring).  Right, John M.? 
This of course requires a finished building, which sorta misses the point
of the simulation!  But John's thesis work provides the basis for many of
us using Radiance to achieve real restful sleep at night. =8-)

> c) what magnitude of error is acceptable for your work?

Ian Ashdown says it better than I can, in his (excellent) "Thinking
Photometrically" coursenotes:

"As for daylighting calculations, it is likely that Jongewaard (1993) is
correct – the results are only as accurate as the accuracy of the input
data. Done with care, it should be possible to obtain ±20 percent accuracy
in the photometric predictions. However, this requires detailed knowledge
and accurate modeling of both the indoor and outdoor environments. If this
cannot be done, it may be advisable to walk softly and carry a calibrated
photometer."

> d) I've come across two opposing views on the accuracy of lumenaire
> descriptor files provided by manufacturers. One states that these can be
> off quite a bit (I think I read that in the 'Rendering with Radiance'
> book) and other authors strut how careful and accurate their simulation
> is by using manufacturer-provided lumenaire descriptors.

Photometry data from the manufacturers is a far better way to describe the
performance of a luminaire than most of the built-in tools in simulation
programs.  But yes, there are still problems.  Primarily, the issue of
far-field photometry.  Linear cove fixtures are trated as point sources
when photometred, and misuse of these IES files in a simulation can lead
to very inaccurate simulations.  Of course in Radiance you can increase
the -ds value to at least help the situation, by taking that "point"
distribution and sort-of arraying it along the fixture's axis.  As long as
the distribution is the same along the length of the luminaire, and your
-ds is suitably fine, you can get good results this way with
manufacturer-supplied data.  The other big lighting simulation packages
like AGI & Lumen Micro (and dear departed Lightscape) also allow you to do
this, in their own ways.

But sometimes the boast of accuracy simply because manufacturer-supplied
photometry is being used should be a warning sign...  I recently received
a mailer from one of the manufacturers of a popular lighting simulation
program, featuring a rendering on it that was supposed to impress upon me
how amazing and accurate the software is.  The thing is, the linear
uplight pendants in the image were casting this ridiculous round spot on
the ceiling, bearing no resemblance to the linear nature of the fixture --
in fact, it looked a heck of a lot like the operator knew nothing about
far-field photometry and the workarounds one must use when using
photometry files based on that method.  And this was the featured
rendering for the product's promotional literature -- worse, the rendering
was created by one of the company's in-house tech support/training people.
(!)

I think there is a big naivete in the industry -- when you get beyond this
group, who is obviously much more concerned with accuracy -- when it comes
to these photometry files, many designers just download the files and plug
them into their programs and hit the "do my job" button. In fact, these
files are really just ASCII dumps of a test report, a test that used a
certain lamp, with a certain lumen depreciation factor, which may be
different than the one in your spec; other light loss factors need to be
considered, the orientation may not even be what you expected.  So I guess
it just goes back to garbage in, garbage out.  Those manufacturer-supplied
files are only as good as the person integrating them into the simulation.

- Rob Guglielmetti
www.rumblestrip.org