Brad Fitzpatrick (brad) wrote,
Brad Fitzpatrick
brad

Misc thoughts....

I'm sure this has been done, in some form ... anybody got pointers?

Basically I'm back to thinking about "grid computing" and lazy evaluations and async stuff.

I want, in Perl:

-- a way to annotate functions as referentially transparent using perl attributes (ideally autodetect it walking the perl bytecode, but I think that's Hard, so I'll settle for manual attribution)

-- ability to run the same source code in "server mode" all over the network, and the clients automatically choose whether to run functions locally or on a remote node, depending on available local resources, ala OpenMosix or OpenSSI, except at a function level, instead of a process level. the Linda Tuple Space model has been working out well for us for a number of task distribution daemons we run ... i'd imagine doing the same for shipping out functions to be run, at least for an easy first implementation.

-- perhaps using ZeroConf (or whatever it's called nowadays) or mutlicast for servers to announce themselves to the clients.

-- automatically Memoize calls to functions tagged as Referentially transparent, caching globally across the farm in memcached. perhaps observe computation times vs. size to decide what's worth caching.

-- all referentially transparent functions run async, and immediately return to their caller a tied scalar/list that waits for its computation on first evaluation.

so:

$foo = long_operation();
$bar = other_long_operation();
$c = $foo +$bar;

The two long operations would run in parallel, since $foo and $bar are just tied objects which don't block until the addition and assignment on the 3rd line.

-- new parallel looping constructs like "parallel_foreach", simple to add with perl's "&" prototype.

The above would satistify me computation-wise.

Storage-wise, I want to learn GFS more, when you need POSIX semantics... keeping using MogileFS where it works.... use DRBD w/ InnoDB for a highly-available RDBMS, when that's needed....

Just rambling at this point.

Point me at fun projects, related or not.
Tags: perl, tech
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 6 comments