?

Log in

No account? Create an account
Misc thoughts.... - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Misc thoughts.... [Dec. 14th, 2004|11:34 pm]
Brad Fitzpatrick
[Tags|, ]

I'm sure this has been done, in some form ... anybody got pointers?

Basically I'm back to thinking about "grid computing" and lazy evaluations and async stuff.

I want, in Perl:

-- a way to annotate functions as referentially transparent using perl attributes (ideally autodetect it walking the perl bytecode, but I think that's Hard, so I'll settle for manual attribution)

-- ability to run the same source code in "server mode" all over the network, and the clients automatically choose whether to run functions locally or on a remote node, depending on available local resources, ala OpenMosix or OpenSSI, except at a function level, instead of a process level. the Linda Tuple Space model has been working out well for us for a number of task distribution daemons we run ... i'd imagine doing the same for shipping out functions to be run, at least for an easy first implementation.

-- perhaps using ZeroConf (or whatever it's called nowadays) or mutlicast for servers to announce themselves to the clients.

-- automatically Memoize calls to functions tagged as Referentially transparent, caching globally across the farm in memcached. perhaps observe computation times vs. size to decide what's worth caching.

-- all referentially transparent functions run async, and immediately return to their caller a tied scalar/list that waits for its computation on first evaluation.

so:

$foo = long_operation();
$bar = other_long_operation();
$c = $foo +$bar;

The two long operations would run in parallel, since $foo and $bar are just tied objects which don't block until the addition and assignment on the 3rd line.

-- new parallel looping constructs like "parallel_foreach", simple to add with perl's "&" prototype.

The above would satistify me computation-wise.

Storage-wise, I want to learn GFS more, when you need POSIX semantics... keeping using MogileFS where it works.... use DRBD w/ InnoDB for a highly-available RDBMS, when that's needed....

Just rambling at this point.

Point me at fun projects, related or not.
LinkReply

Comments:
From: pos_le_terrible
2004-12-15 10:37 am (UTC)
>> i'd imagine doing the same for shipping out functions to be
>> run, at least for an easy first implementation.

you can use Storable to freeze/thaw code references:
Storable#CODE_REFERENCES


the async $foo $bar thing is just awesome! $c=$foo+$bar could even result in a tied variable which would in turn lazily evaluate when it really has to (when $c is printed or serialized for example).

Another link (this time I'm almost sure you already know this one) :
Coro (Marc Lehmann again!)
(Reply) (Thread)
[User Picture]From: brad
2004-12-15 05:42 pm (UTC)
As much as I use Storable, I've never done code freezing. I wonder if this means I could run dummy servers that don't know the code and learn it over time... using AUTOLOAD to catch unknown functions and ask the clients for the latest version.

I know about libcoro, but not Coro.pm! Thanks for the link!
(Reply) (Parent) (Thread)
[User Picture]From: ckd
2004-12-15 06:19 pm (UTC)
ZeroConf is a neat idea. Howl seems to be the most common implementation people are using. There is also Net::Rendezvous for client side (looking up services) in Perl; announcements will probably need you to use the Howl API.

Googling found some references to Net::Zeroconf::Backend::Howl but the page that claims to be its home 404s; it's now here as Net::Rendezvous::Publish::Backend::Howl. The same author has some other stuff; start here--I'm not sure what the distinctions are between the "Native" and "Publish-Backend" bits are.
(Reply) (Thread)
[User Picture]From: bryan
2004-12-15 09:52 pm (UTC)

Geekiest Brad in-joke ever

I plan on creating my own file system. It'll make you THINK it's saving a file, when really it does nothing.

I'll call it FitwelFS.

I need more sleep.
(Reply) (Thread)
[User Picture]From: ch
2004-12-16 06:49 am (UTC)
as for analyzing the functions automagically, i don't belive you can do it in all generality (halting problem). certainly there is an (interesting) subset of functions that you can detect automagically.

as far as distributing the work, you can't ignore the network. rpc introduces latencies and failure modes that (local) function call doesn't have. see http://research.sun.com/techrep/1994/smli_tr-94-29.pdf for an introduction to the issues.
(Reply) (Thread)
[User Picture]From: taral
2004-12-16 08:09 am (UTC)
Urg. More perl retrofit? Why not use Haskell?
(Reply) (Thread)