Brad Fitzpatrick (brad) wrote,
Brad Fitzpatrick

M:N DBI proxy, x86, InnoDB...

I want a M:N DBI- or MySQL-level proxy, where M max threads from client processes are serialized over N max real db connections. (especially useful for the master db in a tree of replicated slaves, where the master db connections are largely idle, the slaves doing all the selecting....) Or maybe even a thread pool on the client machines themselves. I wander if anybody's done that with mod_perl2 and Apache2 yet. (is the DBI library threadsafe?)

x86 sucks. (well, for accessing lots of memory) There's a patch (in -aa) to change the default kernel/user split to 0.5:3.5 GB, but the InnoDB manual states glibc is still buggy and the stack and heap can collide if you allocate more than 2GB. But perhaps the glibc they statically link is aware of mapping changes with the -aa patch? I'd hope.

So say I have a machine with 4GB. I can safely have the database address 2GB. Factor into 500+ connections, taking 2.25 MB each. Remove 20MB for Inno's dictionary pool and 30 MB for misc MyISAM table indices which can't run on Inno... Down to 738 MB for InnoDB's data/key cache. Out of 4GB. Pathetic. See why I want the DBI proxy and more address space?

Sure, the rest of the memory will be used somewhat-usefully by the kernel's buffer cache, but not ideally. The database could utilize it better.

Unfortunately I don't have a spare machine with 4GB of memory to tinker with. I have three with 2 GB in the closet, but they all have motherboards with 2 GB limits.

Actually, I guess I do have a spare machine, since LJ just got two new databases... Waiting for lisa to get them online, then I can play.
Tags: mysql, tech
  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.