brad's life - weather, running, distributed cache daemon [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

weather, running, distributed cache daemon [May. 22nd, 2003|05:05 am]
Previous Entry Share Next Entry
[Tags|, , ]

I can’t sleep, so I’ll post.

The weather lately is so wonderful. Every few minutes today I just opened the door and walked outside for a bit and walked back in. Had all the windows open... gentle breeze... birds chirping.

I went for a bike ride & a run, then another run in the evening.

This morning (and when I say morning, I mean noon) I had a great idea in the shower:

A distributed memory cache daemon with a bounded memory limit. The LiveJournal server config file will just have an array of IPs and ports running cache daemons, and then the LJ API will be able intelligently hash a get/set request onto the right process.

And our idle machine with 12 GB? No problem... we just run 4 memcached processes with 2.5GB limits and put them all in the config. Any machine that has extra memory, we use it up, running a memcached.

Any process can go up or down at any time. The LJ API will consistently rehash the key until it hits a memcache process that’s alive. (and keep track of down servers, so it doesn’t keep trying to connect to them)

And if a process goes down and comes back up later? So now it’s empty, but gets refilled, and the stuff that got put on the other cache processes will quickly pass out of the cache without ever being promoted by an access. (I'm also considering making the client API know the uptime of the cache server, and if it's too low, assume it was recently dead and move onto the next hashed process if it didn't get what it wanted from the real one. If it ends up getting something from the second, put it in the first and delete it from the second.)

The data structure is just a dictionary. By default the hashing is over the key, but callers can optionally provide an alternate hashing value (say, username), you’ll get all their content together on the same cache process, making lookups take less connects later. (not that local connects are expensive... *shrug*)

Cache items can also have expiration times. The cache is LRU and everything is O(1). Accessed items get promoted to the top of the list. Neglected items fall of the bottom when the system needs to purge.

So, that’s what I built today.

The memcached process also provides commands to view stats: cache size in items/bytes, cache items, ages, hit rates.

The server is about 100 lines (thanks to the beauty of POE) and the client API is about 80.

Will it work? The question is: how many disk seeks can we avoid? I guess we’ll find that out tomorrow or so. I need to go use the API now to cache all the expensive/seeky DB calls.

Or sleep. I think it's getting light out, though... that makes it harder to sleep.
LinkReply

Comments:
[User Picture]From: supersat
2003-05-22 05:14 am (UTC)

(Link)

Those smart quotes are screwing up your links, yo.
[User Picture]From: brad
2003-05-22 05:15 am (UTC)

(Link)

[User Picture]From: elfy
2003-05-22 11:10 am (UTC)

(Link)

Uhm... You didn't got something from me, or? A package? A friend of mine visited NY a few weeks (3? 4?) ago and took it with her, so the stamps would be cheaper...
I hope I still used the right address, so that it didn't change in between... but you never mentioned it or said something, so... ^^;; I just hope it didn't get lost.
[User Picture]From: scsi
2003-05-22 06:31 pm (UTC)

(Link)

I'm assuming that this memcached is going to be optional.. :) For those of us without 12 gigs of ram (just 1/4th of that).. :)