[Radiance-general] radiance + openmosix = :)

Sat Aug 28 00:43:30 CEST 2004

Hi all,

I had some time to do a few experiments with
openmosix and radiance during the past weeks.
And, luckily, I've been successful, or at least
now I know a little bit more about both systems ...

Maybe somebody can be interested ...

Probably many of you already know what openmosix is
(and of course _all_ of you should know about radiance,
even if day by day we discover something new about it ...):
as far as I've understood, openmosix is a linux kernel 
extension that provides single image clustering 
capabilities to a network of linux boxes.
In other words, instead of having a machine with a single
processor, it is possible to see as many processors
as all your networked linux boxes have.

The main advantages of the openmosix approach to do parallel
computation with radiance are:

1. no need to use nfs (it can really be painful to choose
   the right version and to set up the lockd daemon with
   linux)
2. no need to login to different machines to launch the rpiece
   processes (no rsh or ssh or telnet or whatever needed)
3. it works with rholo -n x, when x>1

What you need: a few computers laying around, a linux distro
(I prefer and suggest debian or knoppix/clusterknppix, maybe 
gentoo can also be practical), a newtork switch + cables, 
time, patience.

At work, I've been able to put over and under my desk (kidnapped) 
1, 2, 3, ..., 8 pc's, some old, some new (from 433Mhz PIII to 2.2Ghz PIV), 
and I started doing some tests in my very little spare time. 
It is a little bit hot now, and I get a headache after 5 minutes,
but ...

First I installed gnu/debian on each of them (well, I installed
it on one and then cloned it with rsync), and then recompiled
the linux kernel (2.4.26) with the unofficial openmosix patch
from http://openmosix.snarc.org/download: I used the unofficial
one because it is more updated, and I needed to have at least 
kernel 2.4.26 for a couple of hardware devices on my laptop to work:
it is very important for all the machines in the network to
have the same kernel and openmosix versions, otherwise they
won't communicate. There are no openmosix patches for 2.6.x kernels, 
yet.

After installing the kernel and compiling and installing radiance
(I used the optimisations for the slowest processors in the cluster,
PIII), I verified that everything was working: the new openmosix releases
include an autodiscovery daemon that takes care of automaticaly
adding new nodes to the cluster when they are available, and also use a 
special load balancing algorithm to automatically migrate the processes to 
the faster nodes. 
Debian has two openmosix packages, one for the patch (the patch is not very 
updated, 2.4.24, but also installs the daemon init scripts) and another for 
the userspace programs, to monitor the cluster and manage the migration
process. There also is an additional graphical interface, openmosixview,
quite helpful.

The first test I did was with rholo, and it worked pretty well with the
network of slowest computers: when adding the faster ones, some processes
were migrating to the fastest machines, other were spending a lot of time
in deciding what to do, adding overhead to the system and slowing down
the simulation. With rpiece it was even worse, and I was starting losing
my faith ...

As you may have noticed, my system is very asymmetrical, and this was
the cause of many failures.

After that, I've tried with the fastest ones (2.0 and 2.2GHz PIV), and
after some failures I realized that I had to launch the rpiece processes
in a different way, using some of the openmosix userspace tools.
Now I use "runhome oconv [options]" to compile the octree on the fastest
machine, and I launch a first rpiece process that stays on the first
machine ("runhome rpiece @args &") and several new rpiece processes that
are able to migrate to the other fast nodes with "mosrun -t 1 -d 700 rpiece 
@args &". I understood that it is better to turn off the slower pc's or to
keep them in a different network segment, otherwise they will make the
system slower and slower. So a symmetrical cluster is far better.

To make it possible to compare my results, I've benchmarked my 2-node cluster
with the bench2 package as taken from Mark Stock's wonderful site, and here 
are the results (radiance was compiled with PIV optimisations).

standard bench2 on the 2.0GHz PIV
21.92user 0.85system 0:23.45elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60294major+31417minor)pagefaults 0swaps
rpict: 0 rays, 0.00% after 0.004u 0.000s 0.004r hours on leviathan
rpict: 2740602 rays, 32.40% after 0.012u 0.000s 0.013r hours on leviathan
rpict: 5594096 rays, 42.60% after 0.020u 0.000s 0.021r hours on leviathan
rpict: 8454012 rays, 48.60% after 0.028u 0.000s 0.029r hours on leviathan
rpict: 11120053 rays, 52.20% after 0.036u 0.000s 0.038r hours on leviathan
rpict: 13856057 rays, 56.10% after 0.045u 0.000s 0.047r hours on leviathan
rpict: 16533987 rays, 60.60% after 0.054u 0.001s 0.055r hours on leviathan
rpict: 19224383 rays, 66.00% after 0.062u 0.001s 0.064r hours on leviathan
rpict: 21936654 rays, 71.10% after 0.071u 0.001s 0.072r hours on leviathan
rpict: 24532107 rays, 75.90% after 0.079u 0.001s 0.081r hours on leviathan
rpict: 27207234 rays, 81.30% after 0.087u 0.001s 0.089r hours on leviathan
rpict: 29704006 rays, 100.00% after 0.095u 0.001s 0.097r hours on leviathan
341.47user 2.70system 5:49.37elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60349major+31607minor)pagefaults 0swaps
0.46user 0.00system 0:00.52elapsed 87%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (484major+66minor)pagefaults 0swaps

So the rpict process has ended after 5:49.

By changing the doit script to this:
------------------------------------------------------------------------------
rm -f scene.oct scene.pic scene.amb syncfile args
/usr/bin/time runhome oconv scene.rad > scene.oct
echo 4 4 > syncfile
echo "-F syncfile -t 30 -vf v1.vf -af scene.amb -ab 1 -x 1000 -y 1000 \
 -aa 0.2  -ar 32 -ad 128 -as 0 -lr 6 -lw 0.005 -o scene.pic scene.oct" > args
/usr/bin/time runhome rpiece @args &
/usr/bin/time mosrun -t 1 -d 700 rpiece @args &
------------------------------------------------------------------------------

I got the following result:

21.85user 0.91system 0:22.94elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60294major+31417minor)pagefaults 0swaps
FRAME 1: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl 1.5
rpict: 0 rays, 0.00% after 0.004u 0.000s 0.008r hours on leviathan
rpict: 3987 rays, 100.00% after 0.004u 0.000s 0.008r hours on leviathan
FRAME 2: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl -0.5
rpict: 3987 rays, 0.00% after 0.004u 0.000s 0.008r hours on leviathan
FRAME 1: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl 0.5
rpict: 0 rays, 0.00% after 0.004u 0.000s 0.009r hours on leviathan
rpict: 4154 rays, 100.00% after 0.004u 0.000s 0.009r hours on leviathan
FRAME 2: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 1.5 -vl -1.5
rpict: 4154 rays, 0.00% after 0.004u 0.000s 0.009r hours on leviathan
rpict: 726110 rays, 100.00% after 0.006u 0.000s 0.012r hours on leviathan
FRAME 3: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl 1.5
rpict: 726110 rays, 0.00% after 0.006u 0.000s 0.012r hours on leviathan
rpict: 492809 rays, 100.00% after 0.005u 0.001s 0.011r hours on leviathan
FRAME 3: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl 0.5
rpict: 492809 rays, 0.00% after 0.005u 0.001s 0.011r hours on leviathan
rpict: 1324193 rays, 100.00% after 0.008u 0.001s 0.014r hours on leviathan
FRAME 4: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl -0.5
rpict: 1324193 rays, 0.00% after 0.008u 0.001s 0.014r hours on leviathan
rpict: 3678661 rays, 88.80% after 0.013u 0.001s 0.019r hours on leviathan
rpict: 3902848 rays, 100.00% after 0.014u 0.001s 0.020r hours on leviathan
FRAME 4: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs 0.5 -vl -1.5
rpict: 3902848 rays, 0.00% after 0.014u 0.001s 0.020r hours on leviathan
rpict: 3959908 rays, 39.60% after 0.016u 0.001s 0.022r hours on leviathan
rpict: 5912685 rays, 100.00% after 0.022u 0.001s 0.029r hours on leviathan
FRAME 5: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl 1.5
rpict: 5912685 rays, 0.00% after 0.022u 0.001s 0.029r hours on leviathan
rpict: 6612613 rays, 100.00% after 0.021u 0.001s 0.028r hours on leviathan
FRAME 5: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl 0.5
rpict: 6612613 rays, 0.00% after 0.021u 0.001s 0.028r hours on leviathan
rpict: 6599351 rays, 100.00% after 0.024u 0.001s 0.031r hours on leviathan
FRAME 6: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl -0.5
rpict: 6599351 rays, 0.00% after 0.024u 0.001s 0.031r hours on leviathan
rpict: 8845669 rays, 69.60% after 0.027u 0.001s 0.037r hours on leviathan
rpict: 8907044 rays, 18.00% after 0.031u 0.001s 0.039r hours on leviathan
rpict: 10518921 rays, 100.00% after 0.031u 0.001s 0.044r hours on leviathan
FRAME 6: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -0.5 -vl -1.5
rpict: 10518921 rays, 0.00% after 0.031u 0.001s 0.045r hours on leviathan
rpict: 11029597 rays, 40.80% after 0.038u 0.001s 0.048r hours on leviathan
rpict: 13086055 rays, 51.60% after 0.038u 0.001s 0.053r hours on leviathan
rpict: 13229033 rays, 100.00% after 0.039u 0.001s 0.053r hours on leviathan
FRAME 7: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl 1.5
rpict: 13229033 rays, 0.00% after 0.039u 0.001s 0.053r hours on leviathan
rpict: 13233114 rays, 100.00% after 0.039u 0.001s 0.053r hours on leviathan
FRAME 8: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl 0.5
rpict: 13233114 rays, 0.00% after 0.039u 0.001s 0.053r hours on leviathan
rpict: 13263283 rays, 100.00% after 0.039u 0.001s 0.054r hours on leviathan
FRAME 9: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl -0.5
rpict: 13263283 rays, 0.00% after 0.039u 0.001s 0.054r hours on leviathan
rpict: 13306793 rays, 75.60% after 0.046u 0.001s 0.056r hours on leviathan
rpict: 14642671 rays, 100.00% after 0.042u 0.001s 0.057r hours on leviathan
FRAME 10: -vtv -vp 23.6453 -93.6881 26.4175 -vd -0.227921 0.911685 -0.341882 
-vu 0 0 1 -vh 16.4264 -vv 16.4264 -vo 0 -va 0 -vs -1.5 -vl -1.5
rpict: 14642671 rays, 0.00% after 0.042u 0.001s 0.057r hours on leviathan
rpict: 14338103 rays, 100.00% after 0.049u 0.001s 0.059r hours on leviathan
177.45user 3.49system 3:33.27elapsed 84%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60493major+32009minor)pagefaults 0swaps
rpict: 15387552 rays, 100.00% after 0.044u 0.001s 0.059r hours on leviathan
159.67user 2.77system 3:40.42elapsed 73%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (60894major+33670minor)pagefaults 0swaps

So the rpiece processes ended after 3:40 (~63% of the previous time) 
-> very promising, isn't it?

You may have noticed that I've also shared the ambient file between the
two processes (I haven't cheated, it was erased at the beginning of
the simulation), and since the file was accessed on the same computer
there was no need to set up a network file system, and there was no need to 
think about file permissions and so on ...

Hope you may find this useful ...

Sorry for not writing very frequently.
And for writing this much this time (!).

I hope to see you all at the 2004 workshop
(any news?).

-- 
Francesco Anselmo
pisuke at blueyonder.co.uk