Log in

No account? Create an account
Bugs, Meetings, Testing - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Bugs, Meetings, Testing [Jul. 10th, 2005|11:46 pm]
Brad Fitzpatrick
[Tags|, , , , ]

So the combination of like four of us teamed up to contribute seemingly benign patches which in combination produced a bug that was caught and fixed within hours of release.

It was suggested that we have a post-mortem to discuss it, but I, being stubborn, said I wouldn't be attending, since we'd already had a suitable (I thought) post-mortem with the developers involved the night of the bug, and we'd recently had a long post-mortem for another unrelated issue that ended up covering tons of stuff, so I thought nothing new could come of this 3rd post-mortem that wasn't already covered in the first two, short of playing the blame game or something.

Ah, but I was wrong.

While I wanted to avoid a meeting and perhaps hack, it appears my time savings argument was fruitless as I've likely spent more time reading/writing emails about the meeting and the issue than the proposed meeting would've taken had I just attended.

So I admit defeat, and in addition to having spent time and my precious wrists/fingers writing emails, I will also be attending the meeting, if only to cut my losses and not type anymore.

But it might be fun as I'd love to discuss writing a test suite to cover the entirety of LiveJournal. Historically I've shunned tests, mostly because anything non-trivial I work with is distributed on lots of machines, deals with timing, and is just generally a bitch to test accurately. Lately, however, I've had success writing test suites of pretty complicated things, like LWPx::ParanoidAgent, OpenID, and just this weekend with Ben, Gearman (which Ben pretty much did).

So I'm warming up to automated testing, especially considering it'd be something the sysadmins could run first to feel better about code being pushed, and there'd be proof in the code repo that a test was or was not written. (which there would then be policy to include)

In conclusion: fun, fun.

[User Picture]From: substitute
2005-07-11 06:54 am (UTC)

CSI: Production Website Failure

I hate post mortems, and particularly post mortems about the failure of a public product or service. I've yet to attend one that didn't become the blame game, no matter what the good intentions were of the people who called the meeting.

There are sometimes good results (changes in process that prevent further outages, etc.) but I can't escape the feeling each time that those results could have been accomplished without the Ceremonial Humilitiation in Conference Room 3.

For me it always boiled down to the need for better QA. YMMV, but in the case of my own job we didn't have these meetings so much after we got a committed and meticulous QA team.
(Reply) (Thread)