| Misc thoughts.... |
[Dec. 14th, 2004|11:34 pm]
Brad Fitzpatrick
|
I'm sure this has been done, in some form ... anybody got pointers?
Basically I'm back to thinking about "grid computing" and lazy evaluations and async stuff.
I want, in Perl:
-- a way to annotate functions as referentially transparent using perl attributes (ideally autodetect it walking the perl bytecode, but I think that's Hard, so I'll settle for manual attribution)
-- ability to run the same source code in "server mode" all over the network, and the clients automatically choose whether to run functions locally or on a remote node, depending on available local resources, ala OpenMosix or OpenSSI, except at a function level, instead of a process level. the Linda Tuple Space model has been working out well for us for a number of task distribution daemons we run ... i'd imagine doing the same for shipping out functions to be run, at least for an easy first implementation.
-- perhaps using ZeroConf (or whatever it's called nowadays) or mutlicast for servers to announce themselves to the clients.
-- automatically Memoize calls to functions tagged as Referentially transparent, caching globally across the farm in memcached. perhaps observe computation times vs. size to decide what's worth caching.
-- all referentially transparent functions run async, and immediately return to their caller a tied scalar/list that waits for its computation on first evaluation.
so:
$foo = long_operation();
$bar = other_long_operation();
$c = $foo +$bar;
The two long operations would run in parallel, since $foo and $bar are just tied objects which don't block until the addition and assignment on the 3rd line.
-- new parallel looping constructs like "parallel_foreach", simple to add with perl's "&" prototype.
The above would satistify me computation-wise.
Storage-wise, I want to learn GFS more, when you need POSIX semantics... keeping using MogileFS where it works.... use DRBD w/ InnoDB for a highly-available RDBMS, when that's needed....
Just rambling at this point.
Point me at fun projects, related or not. |
|
