We do this every time we start to run out of CPU. We also ordered 8 new Dual Xeon 3.06 Ghz 1MB machines w/ 2GB of memory. This time instead of diskless 1Us we're getting 1Us with two SATA hot-swap on front, but without disks. But when we fill up our 8TB of disk with FotoBilder images, we'll add 4TB of 250GB disks to these 8 new machines.
Speaking of which, I need to finish MogileFS. A PHP/Python(?) user wants to use it too, so I'm moving more from the library into the daemon, making the library really light-weight so bindings are simple. Fortunately I'll be able to reuse large parts of Perlbal (my load balancer) inside the MogileFS TCP listener.
The MogileFS daemon will run and maintain at least 3 forked processes:
-- TCP listener (async, event-based, answering requests for clients)
-- Replicator
-- Lazy deleter
-- ...
And the daemon will run on multiple machines, so there are no points of failure. The client will be configured to know about all of them, and just pick one at random until one responds quickly.
I would've worked on the above today, but I got stuck doing the profiling. Damn sysadmin stuff. :-)