...A most interesting idea, John!
memcached + zeroconf-based formation/management features would be pretty interesting: It would be nice if the clients didn't have to hard-code the addresses of the cache nodes.
But the more I think about it, nodes flickering on/off would cause buckets to change, killing cache hit rates. Or worse: netsplits, where you store a val1 in key foo, then store val2 in foo (now on a new server), that server goes down, then you retrieve foo and get val1, on the old server. Handling that would kinda suck.
I guess you could get away from the hash of hashes model (where nodes aren't buckets), and actually replicate content around, but I think that's the wrong way to go... LJ hits over 50% cache hit rate after being empty in like seconds. So maybe we shouldn't care about cache hit rates going to zero when servers go on/off-line and just worry about the netsplit problem?
My god... this is my first hard-core blogging, where I quote/link to other people. I feel so dirty.