Log in

No account? Create an account
storage, layer block device for memcached - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

storage, layer block device for memcached [Oct. 29th, 2003|08:39 pm]
Brad Fitzpatrick
[Tags|, ]

I've been busy learning all about SANs, solid state disks (battery backed SDRAM vs. Flash), Fibre Channel and topologies, global filesystems, the Linux block device layer, etc...

I need to go on a spending spree and play with some of this cool hardware. Some of the cool stuff costs around $65k+, though.

I bought the new book Linux Kernel Development by Robert Love.

I've been searching for a reason to try my hand at kernel development (yeah, laugh) and have been reading the LKML, lwn.net kernel news, and various bits of the source for ages in preparation.

I'm thinking of writing a stacked block device driver that puts a multi-memcached-based caching layer on top of another block device. memcached would be extended to support a faster protocol, so memcached wouldn't have to do parsing and constructing internal hashing values. I'd make a parameter of the block device be a uniq ID, which would be sent along to the memcached servers, so multiple machines with independent block devices could share some subset of the same set of memcached servers.

The kernel driver would only give the remote memcached servers a few milliseconds to respond before considering them loaded or down and falling back to the lower block device.

The driver wouldn't do re-hashing, the avoid getting stale data after a machine disappears and reappears on the network, with its cache intact. (I suspected we might run into that problem on LiveJournal with memcached, and we have a bit lately... I'm working on a solution which I'll write about later.)

Anyway, I don't think would be too difficult. I'd be able to leverage the md driver (multi-disk, for software RAID/etc) in the kernel for the block device stacking, and I'd be able to leverage the nbd driver (network block device) for learning how to do network I/O within kernel space.

Alternatively, I could probably find a way to do this all in userspace. I know I could in FreeBSD (Jeff was just writing about GEOM_DISK the other day), and I know there are filesystems implemented in userland on Linux, so a userland block device isn't unreasonable to assume existing. But kernel would be more fun, for learning.

And don't worry--- it would be years until I put something like this into use on LiveJournal, if ever. Your data is safe. Don't fear my non-existent skills in kernel space.

(somewhat related: Brian Aker and I have been discussing extending MySQL's buffer/row caches system with memcached support... or making a new table handler completely. That might be an easier place to start, but block-device-level would be so much more cool.)

From: jon
2003-10-29 09:55 pm (UTC)
Unrelated to memcached, but related to clusters ...

I attend Matt Dillon's talk about DragonFly BSD over at Berkeley last week. He spent some time talking about ways in which DragonFly could operate well in a clustered environment (going after the Single System Image goal), and he spoke specifically about dealing with stale data in local caches on machines in the cluster. He had some good information in there. You should check out his slides (OpenOffice format).
(Reply) (Thread)
[User Picture]From: brad
2003-10-29 10:12 pm (UTC)
Thanks for the link. Skimmed it... I'll read it more later. (girlfriend wants to watch TV)

I've seen some impressive stuff lately wrt SSI. I've been meaning to play with Linux OpenSSI and GFS.
(Reply) (Parent) (Thread)
[User Picture]From: mulix
2003-10-30 03:42 am (UTC)
You wrote: "
I'm thinking of writing a stacked block device driver that puts a multi-memcached-based caching layer on top of another block device. "

How is this better than the usual caching the kernel already does? (e.g. page cache, buffer cache). If I understand correctly, you mean to have *distributed* caching? if you are indeed, this is very interesting, but not trivial. Could you elaborate?

Kernel hacking is fun, by the way. Welcome aboard :-)
(Reply) (Thread)
[User Picture]From: brad
2003-10-30 10:18 am (UTC)
Yeah, we already do distributed caching for LiveJournal on the application layer. Why not push it down a bit?

I'm not aboard yet, but thanks. :-)
(Reply) (Parent) (Thread)
From: jeffr
2003-11-01 10:37 pm (UTC)
There is a fellow at the university of washington that has done a distributed page cache with FreeBSD. I could look it up if you're interested.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2003-11-02 02:11 am (UTC)
Yeah, that'd be cool.
(Reply) (Parent) (Thread)