||[Oct. 15th, 2004|12:31 am]
Work's been interesting lately. Dozen things going on in all directions.
I'll try to list some of them:
-- Mahlon's investigating using MySQL Cluster. He has 4 nodes up (2 sets of 2 redundant nodes) so we can learn how to admin it, monitor it, benchmark it, understand its limitations, etc. Fun stuff. This is the future.
-- We've been running an increasing number of async jobs distributed. The problem with that is how do they get their work? They can all independently query what needs to be done, randomize the list, and iterate over it with proper locking on each work item, but it gets to be a waste of time. We're moving all to a new model (called a "message board" or "Linda tuple space") where a job server is the hub between workers and populators, all working in parallel, and the job server atomically hands out jobs ("grab" operation). This is speeding up async work a lot, and we're only just starting to use it in more places.
-- We've moved all userpics and phoneposts to MogileFS, stressing it in a way I didn't anticipate: ephemeral port exhaustion. 65k should be enough for anybody! (well, more like half that) Anyway, came up with some fixes for that, some done, some pending. (also got a bunch new hardware tonight that'll help quite a bit... we'd been running pretty low)
-- Ephemeral port exhaustion led us to work on persistent connections for Perlbal from the client side. Previously it only did persistent connections between perlbal and backends. But because we can do that funky reproxy (internal redirect) thing, those reproxied connections weren't cached and were wasting ephemeral ports pretty quickly. Junior to made the webserver (mogstored, which is perlbal's libraries) to persistent connections, then we need to make the internal redirect HTTP client stay persistent.
-- Minor perl profiling / optimization work with Perlbal. You could waste days and weeks on this... best to limit yourself and not get carried away. Makes for good work when everything else is frustrating you though.
-- Been getting lots of new hardware in and setup. Ordered 8 new dual proc 3.0 Ghz Xeon 1MBs. Also 2 dual 246 Opterons w/ 8 GB of memory. Fought some NUMA bugs where Linux was unreliable in certain topologies. lkml people blamed the motherboard for even allowing it, or being unstable with it as it was. Got two new Itanium machines coming soon.
-- FotoBilder's new protocol is coming along. Some more of us need to fully review it and officially publish it to spur some client developent.
-- Been doing a lot of late-night database maintenance, so I sit here while operations are pending, writing long-winded posts.