Now, driving to Seattle because something's "too slow" by 30-40 milliseconds sounds ridiculous, but to put it in perspective, the latency should be between 0.01 and 0.3 milliseconds.
I hoped to get up here and find some stupid misconfiguration, but the problem was a couple of our switches (one in particular) were maxed out, doing more pps than they could handle.
Put in a desperate call to Semaphore and four of their top network engineering guys rushed over and went crazy on our network, analyzing and rearranging. Very impressive. We're considering moving all our stuff over to their facility, and with their help today, and seeing them in action.... very convincing.
Anyway, our internal network is good for now. Long-term it's still not ideal, but much better than before. Unfortunately, now we're pushing 98.7 Mbps to the Internet, over fast ethernet (100 mbps limit). The Semaphore guys were hooting and hollering after seeing that, as I guess they'd only seen 97.0 mbps before on FastEthernet, and only during an attack. yay us? But now we need to upgrade our connection to the net, for realz. (which is why we were aware of Semaphore in the first place.... we were talking to them about GigE connection(s) to the net, knowing we were soon pushing our transit limits....)
I still have a lot of other stuff to do on LJ, but it's stuff I know now, so I'm relaxed more.
Oh yeah, today was
brett's first day of work. He knows more networking than me and zipped through those IOS and Catylst-OS screens faster than I could, but once the Semaphore guys came we just kinda stood in the corner. We'll try to get him badged tomorrow, and start training him on the LJ network.Afterwards, beer with
erik, henry, brett, jenn, Dana, Quentin, and some other girl whose name I forget. Have I ever mentioned how much I suck at names? :-(So, back to work.