[Radiance-general] strange artifacts in rendering (and Yosemite bug)

[Gregory J. Ward]

As it turns out, this appears *not* to be a problem with Yosemite after all, but an issue with the last official release (4.2.2), whose accuracy isn't quite as good as the latest (HEAD or 5.0a) with respect to its Hessian interpolation method.  The artifacts MJ noticed were due to inaccuracies in the interreflection calculation, for which this scene is something of a worst case.  Decreasing the -aa parameter while increasing -ad seems to resolve these issues, especially with the latest code.

While playing around with Yosemite trying to reproduce these artifacts, I discovered that the problems I thought I was having with the system select() call were in fact due to a bug in the way Yosemite handles application-level file locking.  This was a complete surprise, but it turns out not to be so difficult to work around.

Without getting too much into the details, the stdio library has these lock/unlock calls to make the library thread-safe, which of course I don't care about because I don't use threads, but they severely impact performance even on unthreaded code.  In some cases, it requires a mutex lock/block/unlock procedure for every character that is input or output, completely defeating the purpose of i/o buffering that is the purpose of stdio in the first place.  Therefore, I had some workarounds in place to reduce or eliminate this overhead, basically locking the files once at the outset to avoid the per-call mutex overhead.

This all worked fine up until the latest release of Mac OS X (10.10), when Apple changed the behavior of fork() such that it preserves application-level i/o locking for no conceivable reason.  Since these locks are designed to negotiate between threads sharing memory (i/o buffers), and separate processes don't share memory, the locks should sever on fork().  Every Unix out there does this, except Mac OS X 10.10.x.  Therefore, when I made a call to flockfile() at the start of rvu or rtrace, then forked its children, each child would deadlock the instant it tried to use a shared file pointer (i.e., the ambient file).

You could say I was being cavalier about the use of flockfile(), but I was just trying to work around a horrible performance bottleneck introduced with automatic application-level locking in stdio.  It's one thing to introduce a new feature and suggest that people using multi-threading take advantage of it.  It's another thing to implement locking as the new default behavior, while decreasing i/o performance by about 90% (no exaggeration) even when it is not needed, then not even offer a means to disable it.

In any event, I checked in a change that I'm hoping fixes this issue under Yosemite, while not costing us too much on all the other Unix platforms that don't have this issue.

Cheers,
-Greg

> From: "Gregory J. Ward" <gregoryjward at gmail.com>
> Subject: Fwd: [Radiance-general] strange artifacts in rendering
> Date: August 20, 2015 9:14:23 AM PDT
> 
> Just forwarding this to the mailing list in case as an additional warning against installing Mac OS X Yosemite (10.10.x).  As if the multi-processing bug were not enough, there seem to be other problems with instances and/or textures (not sure) on that platform.
> 
> I may try investigating this on the weekend, but I don't hold out much hope for a resolution.
> 
> -Greg
> 
>> From: Minjung Kim <mkim85 at gmail.com>
>> Subject: Re: [Radiance-general] strange artifacts in rendering
>> Date: August 19, 2015 4:58:24 PM PDT
>> To: "Gregory J. Ward" <gregoryjward at gmail.com>
>> 
>> Hi Greg,
>> 
>> I think the problem might indeed be with OS 10.10. None of the 20 scenes that I rendered on OS 10.9 had any issues, whereas more than half of the 20 on OS 10.10 did. Please see attached files for examples. 
>> 
>> Thanks for your help!
>> 
>> MJ
>> 
>> On Wed, Aug 19, 2015 at 4:33 PM, Minjung Kim <mkim85 at gmail.com> wrote:
>> I'm going to render a bunch of other blobby objects on 10.9, and see if I get the problem here. I'll let you know what I find out. 
>> 
>> Thanks!
>> 
>> MJ
>> 
>> 
>> On Wed, Aug 19, 2015 at 4:30 PM, Gregory J. Ward <gregoryjward at gmail.com> wrote:
>> Thanks, Minjung.  
>> 
>> I guess we can add this to the list of bugs under Mac OS X 10.10, which to me has been a disastrous release.  I seriously doubt I will be able to fix problems on this platform, but I'll give it a try when I have some time.
>> 
>> Cheers,
>> -Greg
>> 
>>> From: Minjung Kim <mkim85 at gmail.com>
>>> Subject: Re: [Radiance-general] strange artifacts in rendering
>>> Date: August 19, 2015 4:16:50 PM EDT
>>> 
>>> Hi Greg,
>>> 
>>> Thanks for your help! I am still unable to get the artifacts to come back -- but only on my current machine (late-2013 iMac, 21.5 in., OS X 10.9). If I run the same code on a different machine (late-2013 iMac, 21.5 in., OS X 10.10), I reliably get the artifact. Isn't that strange?
>>> 
>>> Attached are the files that I used. To run, please use the following commands:
>>> 
>>> oconv square.rad > square.oct
>>> oconv scene.rad > scene.oct
>>> rpict -x 1024 -y 1024 -vh 12.018 -vv 12.018 -vp 0 -0.57 0 -vd 0 1 0 -vu 0 0 1 -av 0 0 0 -vtv -ab 1 scene.oct > scene.pic
>>> pfilt -1 -x 512 -y 512 -e 1 -r 0.6 scene.pic > scene.hdr
>>> 
>>> In scene.rad, the square is incorporated as an instance. I know that I don't really need to instance the square since I only have one, but the scene with the artifacts was using instances, so I thought it would be better to send you the code as is. 
>>> 
>>> If anything is unclear, please let me know. Thanks again!
>>> 
>>> Regards,
>>> 
>>> MJ
>>> 
>>> On Tue, Aug 18, 2015 at 7:00 PM, Minjung Kim <mkim85 at gmail.com> wrote:
>>> Hi Greg,
>>> 
>>> Thanks! I'm putting together the files, but I'm unable to replicate the problem using my latest code, though I don't think I've changed anything substantial. I'll get back to you once I can reliably replicate the artifacts.
>>> 
>>> MJ
>>> 
>>> On Mon, Aug 17, 2015 at 9:04 PM, Greg Ward <gregoryjward at gmail.com> wrote:
>>> Hi MJ,
>>> 
>>> Can you send me your model & rendering parameters off-list.  I won't know what's going on until I play with it.
>>> 
>>> Thanks,
>>> -Greg
>>> 
>>>> From: Minjung Kim <mkim85 at gmail.com>
>>>> Subject: [Radiance-general] strange artifacts in rendering
>>>> Date: August 17, 2015 5:30:37 PM EDT
>>>> 
>>>> Hi everyone,
>>>> 
>>>> I am getting strange artifacts in my renderings, and am wondering which parameters to tweak in order to get rid of them. The artifacts look like a white, or sometimes half-black and half-white, patch at random locations. See below for examples:
>>>> 
>>>> http://minjung.ca/tmp/artifacts1.png
>>>> http://minjung.ca/tmp/artifacts2.png
>>>> 
>>>> I've tried (1) setting ad = 4096, as = 1024, and (2) setting aa = 0, ar = 0, but neither of these two settings fixed the problem. Do you have any idea what might be happening? Here are some possibly relevant factors: 
>>>> 
>>>> - The object is completely Lambertian.
>>>> - The only light source in the scene is a glow source that is positioned behind the camera, and covers as much of the sky as possible. 
>>>> - I don't think I've ever encountered such artifacts with untextured stimuli.  I could be wrong, though.
>>>> 
>>>> Thanks in advance!
>>>> 
>>>> MJ
>> 
>> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.radiance-online.org/pipermail/radiance-general/attachments/20150821/2cbe2e4c/attachment.html>