Brad Fitzpatrick (brad) wrote,
Brad Fitzpatrick
brad

fun hack

When you shutdown a MySQL server that's been running for weeks with --delayed-key-writes on (obviously a for-speed instance only, since that's so unreliable!), the shutdown process takes forever as MySQL goes to rewrite all its index files... sometimes (in LJ's case) up to a few gig.

So you run /etc/init.d/mysql stop and it just seems to hang.

Back when I was less experienced, I'd kill -9 it so I could start shit back up. Didn't take me more than about 2 times doing that to realize I was just totally fucking shit up and I'd have to resync from a master. Not bad... but a waste of time.

What kills me isn't the waiting... but not knowing the status.

But I have a way now!

* find the child-most mysqld pid
* strace -p <pid>
* watch the pwrite calls... take note of 1st argument (the fd) and last arguemnt (the offset)
* look at /proc/<pid>/fd/<fd> and see where that's symlinked to
* now we know the file it's writing to, and the total size

So now I'm not so impatient. But... I want a pretty interface to poll the status every 10 seconds or so (don't want to keep strace attached the whole time) and show the current index file and the percentage done.

So I'm thinking of writing a script to do all this, then see if I can't make a debian package out of it. But that's the tricky part. It'd depend on strace, sure, but then how to integrate it into the /etc/init.d script? No hooks are in there. It might just need to include directions on how to add it in there.

Fun fun.
Tags: mysql, tech
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 5 comments