Brad Fitzpatrick (brad) wrote,
Brad Fitzpatrick
brad

Work

Whitaker and I profiled the hot code paths on the site today, starting with some simple SQL commands. Since we log the codepath name, CPU consumption, and wall time of each request, we can just ask the database, "What codepaths take the most CPU, on average?" Then we make a test wrapper to hit that codepath hundreds of time and profile. In the end we made some of our most popular codepaths about 25% faster.

We do this every time we start to run out of CPU. We also ordered 8 new Dual Xeon 3.06 Ghz 1MB machines w/ 2GB of memory. This time instead of diskless 1Us we're getting 1Us with two SATA hot-swap on front, but without disks. But when we fill up our 8TB of disk with FotoBilder images, we'll add 4TB of 250GB disks to these 8 new machines.

Speaking of which, I need to finish MogileFS. A PHP/Python(?) user wants to use it too, so I'm moving more from the library into the daemon, making the library really light-weight so bindings are simple. Fortunately I'll be able to reuse large parts of Perlbal (my load balancer) inside the MogileFS TCP listener.

The MogileFS daemon will run and maintain at least 3 forked processes:

-- TCP listener (async, event-based, answering requests for clients)
-- Replicator
-- Lazy deleter
-- ...

And the daemon will run on multiple machines, so there are no points of failure. The client will be configured to know about all of them, and just pick one at random until one responds quickly.

I would've worked on the above today, but I got stuck doing the profiling. Damn sysadmin stuff. :-)
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 6 comments