Heh. Welcome back to the land of the living. I've spent some time babysitting servers before, and although it's nothing like what you guys are dealing with, I feel at least a bit of your pain.
Get some sleep. It'll do you good.
Happy new year!!!!!! ;-)
Good to see everything's back. Damn internap or whatever the hell it is. (Im obviously not a self made computer millionaire)
thank you and all your colleagues for the hard work in bringing everything back up and I commend you for being so cautious and ensuring things were the best they could be before coming back online.
I hope you can enjoy some of the holiday weekend. I'm grateful for the efforts!
2005-01-16 01:06 am (UTC)
ohhh livejournal is back!
Fuckin' cheesy that happened again dude. Sorry I was a pain about not wanting some rack batteries in there.
Both times were freak accidents. Internap's one of the most carefully wired datacenters I know.
Thanks for your job!
It would be really interesting to read the story.
Greatly appreciate all the efforts (and lost sleep) of you and the rest of the LJ SysAdmin crew. On a weekend no less.
Having experienced 3 datacenter power "events" in the space of 10 hours once ( conveniently just after we'd recovered from the previous incident o_O ), I understand some small part of your pain, and really appreciate the caution you've taken in bringing the site online.
Cheers, and if you ever make it to Australia, I owe you a beer.
2005-01-16 02:26 am (UTC)
Thanks for all your hard work!
Thanks Brad. I'll bet that was a bit of work. I hope you didn't have to drive or even walk anywhere today in this ice. Yikes what a weekend. Thanks to you and your cohorts for your wonderful site.
2005-01-16 02:29 am (UTC)
I'm eager to hear about what amusing failure befell Internap this time!
I think is just cause for a new BradIsExhauste lj userpic. Or perhaps BradSaysFuckInternap or something.
Although I'm sure you'll hear this from a lot of users, I want to thank you and your team for the large amount of work that you put into getting the system back in such short order. Take care of yourself now. You're more important than the writeup (although it will probably make an interesting, perhaps publishable, case study).
Thank's for your hard work getting the site back up. Are your servers located in the Westin Building? Oddly, the datacenter my friend works at in West Seattle blew a breaker, taking down half their servers (including the ones that send out alerts about outages), around noon yesterday.
Was entranced by /powerloss.
Kept on checking it as furiously as I've ever checked my friends list.
Sounded like a fun time.
Thanks for everything you do man.
I just went through a scheduled proper power down of a server room and it was a major bitch. I know how badly your night had to have sucked.
Keep up the good work!