[Radiance-general] Slow spawning of rtrace -n, mkillum -n with many light sources

[Greg Ward]

Hi Axel,

I'd be interested in looking at OBJ files where obj2mesh fails.

Cheers,
-Greg

> From: Axel Jacobs <jacobs.axel at gmail.com>
> Date: February 22, 2017 1:55:48 AM PST
> 
> Hi Greg,
> 
> thank you so much for solving this riddle.  I have to admit I have
> been blissfully unaware of the drawbacks of using RTMs. We use them as
> much as possible now, since I have come to appreciate the fast compile
> times and small octree sizes this can give us.  However, there have
> been quite a few projects where RTMs would take hours to compile,
> generating octrees many GB in size.  In such instances, we convert the
> OBJs into polygon, but without the normals.  Occasionally, we also
> need to go via obj2rad, simply because obj2mesh fails to generate the
> RTMs.
> 
> Looks as if we need to take a fresh look at our workflow.
> 
> Thank you for adding the limit on the source obstruction code.  I'll
> try this out in the next few days, and re-run the project
> 
> Thanks again
> 
> Axel
> 
> 
> On 20 February 2017 at 18:00, Greg Ward <gregoryjward at gmail.com> wrote:
>> Oh, I just remembered -- there's also the shadow cache, which gets initialized in marksources() as well.  This also ends up tracing some number of rays, about 400 per source, to look for near-source obstructions before the main calculation begins.  So, that could be part of your slow-down.  I guess 6,000 light sources would be about 2.5 million rays to trace, though I have trouble seeing how that would take 45 minutes even on one process.  It should only take a couple of minutes on a modern processor with enough memory to hold the scene.
>> 
>> You could try recompiling with -DSHADCACHE=0, but I don't think you would want to, as the shadow cache is the thing that really saves you with so many light sources.  I'd only do it to determine if that's what slows down the start-up.  (Although it would be an interesting test of the shadow cache under extreme conditions.)
>> 
>> I don't know how to sample running processes on your system, but it would be interesting to find out where rtrace is spending all its time during start-up....
>> 
>> Cheers,
>> -Greg
>> 
>> P.S.  More info on shadow cache in 2004 workshop program:       http://www.radiance-online.org/community/workshops/2004-fribourg/Ward_talk.pdf
>> 
>>> From: Axel Jacobs <jacobs.axel at gmail.com>
>>> Date: February 20, 2017 9:28:36 AM PST
>>> 
>>> Thanks for your answer, Greg.
>>> 
>>> There are no mirror or prism surfaces in this scene.  Just glass and
>>> plastic.  So this is weird, then.
>>> 
>>> I did notice this behaviour on previous projects where we had
>>> thousands of light sources, but didn't look into it back then.
>>> 
>>> Cheers
>>> 
>>> Axel
>>> 
>>> 
>>> On 20 February 2017 at 17:18, Greg Ward <gregoryjward at gmail.com> wrote:
>>>> Hi Axel,
>>>> 
>>>> There is quite a bit of initialization code, the goal of which is to get as much common data into shared memory as possible before calling fork().  This reduces the memory footprint of your processes, as well as avoiding redundant work that wouldn't make it go faster, anyway.
>>>> 
>>>> Included in this preamble are initializing the photon maps (if any), loading the octree, marking light sources (including virtual sources), and preloading the ambient cache (if one).  It also preloads all object data, including instanced octrees, meshes, pictures used in patterns, and so on.
>>>> 
>>>> Even with 6,000 light soruce, marking light sources shouldn't take all that long, *unless* you have "mirror" or "prism" surfaces in your scene.  These will create virtual light sources, multiplying the number of sources potentially by many times.  (Mirror surfaces that face each other are the worst case.)  The virtual light source preamble can take quite some time in such cases, as it tries to eliminate virtual source paths that would never pass light due to obstructions, etc.
>>>> 
>>>> If you don't have any mirror or prism surfaces, then I'm not sure why it would be taking so long.
>>>> 
>>>> Cheers,
>>>> -Greg
>>>> 
>>>>> From: Axel Jacobs <jacobs.axel at gmail.com>
>>>>> Date: February 20, 2017 4:01:33 AM PST
>>>>> 
>>>>> Dear list,
>>>>> 
>>>>> I'm running some rtrace -n xx calculations, and noticed that there is
>>>>> only one thread for 30 to 45 minutes, before the -n xx kicks in.  My
>>>>> scene contains some 6,000 artificial light sources.
>>>>> 
>>>>> The question I have is this:  Is there something within rtrace/mkillum
>>>>> that is not multi-threaded that is run before the actual ray tracing
>>>>> part starts (which does honour the -n option)?  I could think of some
>>>>> light source visibility or intensity test that need to be done before
>>>>> the actual ray tracing.
>>>>> 
>>>>> Many thanks for your thoughts
>>>>> 
>>>>> Best regards
>>>>> 
>>>>> Axel
>> 
>> _______________________________________________
>> Radiance-general mailing list
>> Radiance-general at radiance-online.org
>> http://www.radiance-online.org/mailman/listinfo/radiance-general
> 
> _______________________________________________
> Radiance-general mailing list
> Radiance-general at radiance-online.org
> http://www.radiance-online.org/mailman/listinfo/radiance-general