DBs, memcaches, CPU, network... all seem fine.
So I have no clue why web processes are stacking up. Straced a few... see nothing odd.
What I'd like to do is measure the total wall time per file descriptor that the process spends blocked on. (which would require the tool knowing the different syscalls, and keeping track of what fds were which, or just looking it up as it goes....)
Can oprofile produce reports for that? I don't want to measure where CPU is used... I want to figure out where blocking is happening.