[Radiance-general] parallel dctimestep

[Sarith Subramaniam]

Hi Santiago and Greg,


To add to my original reply, I ran my simulations using the standard 
version of dctimestep. I use a fairly souped-up Microsoft Azure 
cloud-based VM which has  250GB of RAM. If you are parallelizing 
image-based simulations, you'd want to explicitly specify the 
month-name, or number, in the string formatting option( something like 
-o results/Jan%03d.hdr). If not, there is a chance that the results from 
one instance of dctimestep might get overwritten by the other (I learnt 
this the hard way).

Sarith


On 9/22/2017 11:38 AM, Greg Ward wrote:
> Ah, that's a different problem!
>
> How much memory is on your machine?  If there is less RAM than the 
> combined size of the component images, then you won't see the benefit 
> of disk caching.  You can try decreasing the memory required for 
> caching by compacting your pictures (e.g., running "ra_rgbe -r" on 
> each of them).
>
> If you don't have as much free RAM as you have images and you're 
> running off a hard disk rather than an SSD, that is the worst case for 
> dctimestep and image processing.  Running in parallel may help you a 
> bit, but you still need to think about the i/o requirements.  Having a 
> bunch of processes fighting to access the same disk information might 
> not do that much.
>
> Cheers,
> -Greg
>
>> *From: *Santiago Torres <Santiago.Torres at arup.com 
>> <mailto:Santiago.Torres at arup.com>>
>>
>> *Date: *September 22, 2017 9:13:50 AM PDT
>>
>> *
>> *
>>
>> Dear Sarith, Greg,
>>
>> Thank you for your replies. In this case I am running the annual 
>> calculation of an image, and each time-step (daytime) is taking 3-4 
>> minutes to run, so the total calculation takes more than 9 days. And 
>> I would like to increase the resolution to reduce noise…
>>
>> So I thought if I run 10 dctimestep processes in parallel, it should 
>> take less than a day. I hope this makes sense. I will try it out and 
>> let you know if I find problems.
>>
>> Thank you,
>>
>> Santiago
>>
>> *From:*Sarith Subramaniam [mailto:sarith at psu.edu]
>> *Sent:* 22 September 2017 16:53*
>> *
>>
>> Hi Santiago,
>>
>> I pretty much did the exact same thing that you mentioned to run 
>> multiple instances of dctimestep with Python. The syntax is something 
>> like:
>>
>> /import multiprocessing as mp/
>>
>> /import os/
>>
>> /#Run dctimesteps, 16 at at time./
>>
>> /#runCommand is just a call to os.system/
>>
>> /#commands is a list of dctimestep commands like ["dctimestep dc.mtx 
>> sky0001.smx > res0001.smx",  "dctimestep dc.mtx sky0002.smx > 
>> res0002.smx" ...]/
>>
>> /pools = mp.Pool(processes=16)/
>>
>> /pools.map(runCommand, commands)/
>>
>>
>> For the above, you'd have to do some additional scripting to run 
>> rmtxop and compile all the results into a single file. There is also 
>> a method that was discussed by Wangda Zuo and Andy Mcneil in this 
>> paper: http://www.ibpsa.org/proceedings/BS2011/P_1155.pdf 
>> <http://secure-web.cisco.com/1Dg4sbIwWzfdNHSDFjoErkam7XpFZkZl_YajPwbdNSfN3qC1GSwtzOsoHjzUTCbiw63iGUUuE2gD4bllTAnDKmlmn_oe9aU9yRt8wzVUSrmsrE1PKj6ErqSMsFikzAe_P4HEM_ZQk3m52HImTpdUxnGXUzrzYTkz-kgXQZX-U9wtAWex8uSdvjmJXLQZIBdOdjRn7Um1GekbwQI--4karqReEQedYXrhJsex6RkYtNIiYFMCYiwet0zel4nY98Gol/http%3A%2F%2Fwww.ibpsa.org%2Fproceedings%2FBS2011%2FP_1155.pdf> 
>>
>>
>> Sarith
>>
>> On 9/22/2017 9:55 AM, Santiago Torres wrote:
>>
>>     Dear all,
>>
>>     This might be a silly question, but I couldn't find any reference. Is there a way to run dctimestep in parallel?
>>
>>     Or, would it be possible to create a weather tape for each month, create a matrix from each, and run each month separately?
>>
>>     Thanks,
>>
>>     Santiago
>>
>
>
> _______________________________________________
> Radiance-general mailing list
> Radiance-general at radiance-online.org
> https://www.radiance-online.org/mailman/listinfo/radiance-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.radiance-online.org/pipermail/radiance-general/attachments/20170922/b361b435/attachment-0001.html>