I need to go on a spending spree and play with some of this cool hardware. Some of the cool stuff costs around $65k+, though.
I bought the new book Linux Kernel Development by Robert Love.
I've been searching for a reason to try my hand at kernel development (yeah, laugh) and have been reading the LKML, lwn.net kernel news, and various bits of the source for ages in preparation.
I'm thinking of writing a stacked block device driver that puts a multi-memcached-based caching layer on top of another block device. memcached would be extended to support a faster protocol, so memcached wouldn't have to do parsing and constructing internal hashing values. I'd make a parameter of the block device be a uniq ID, which would be sent along to the memcached servers, so multiple machines with independent block devices could share some subset of the same set of memcached servers.
The kernel driver would only give the remote memcached servers a few milliseconds to respond before considering them loaded or down and falling back to the lower block device.
The driver wouldn't do re-hashing, the avoid getting stale data after a machine disappears and reappears on the network, with its cache intact. (I suspected we might run into that problem on LiveJournal with memcached, and we have a bit lately... I'm working on a solution which I'll write about later.)
Anyway, I don't think would be too difficult. I'd be able to leverage the md driver (multi-disk, for software RAID/etc) in the kernel for the block device stacking, and I'd be able to leverage the nbd driver (network block device) for learning how to do network I/O within kernel space.
Alternatively, I could probably find a way to do this all in userspace. I know I could in FreeBSD (Jeff was just writing about GEOM_DISK the other day), and I know there are filesystems implemented in userland on Linux, so a userland block device isn't unreasonable to assume existing. But kernel would be more fun, for learning.
And don't worry--- it would be years until I put something like this into use on LiveJournal, if ever. Your data is safe. Don't fear my non-existent skills in kernel space.
(somewhat related: Brian Aker and I have been discussing extending MySQL's buffer/row caches system with memcached support... or making a new table handler completely. That might be an easier place to start, but block-device-level would be so much more cool.)