?

Log in

No account? Create an account
Seattle - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Seattle [Jun. 24th, 2004|11:42 pm]
Brad Fitzpatrick
Drove up to Seattle this morning because LJ's doomed-from-day-one network architecture finally bit it, with half the network effectively segmented from the other by way of moderate packet loss and 30-40 ms latency.

Now, driving to Seattle because something's "too slow" by 30-40 milliseconds sounds ridiculous, but to put it in perspective, the latency should be between 0.01 and 0.3 milliseconds.

I hoped to get up here and find some stupid misconfiguration, but the problem was a couple of our switches (one in particular) were maxed out, doing more pps than they could handle.

Put in a desperate call to Semaphore and four of their top network engineering guys rushed over and went crazy on our network, analyzing and rearranging. Very impressive. We're considering moving all our stuff over to their facility, and with their help today, and seeing them in action.... very convincing.

Anyway, our internal network is good for now. Long-term it's still not ideal, but much better than before. Unfortunately, now we're pushing 98.7 Mbps to the Internet, over fast ethernet (100 mbps limit). The Semaphore guys were hooting and hollering after seeing that, as I guess they'd only seen 97.0 mbps before on FastEthernet, and only during an attack. yay us? But now we need to upgrade our connection to the net, for realz. (which is why we were aware of Semaphore in the first place.... we were talking to them about GigE connection(s) to the net, knowing we were soon pushing our transit limits....)

I still have a lot of other stuff to do on LJ, but it's stuff I know now, so I'm relaxed more.

Oh yeah, today was brett's first day of work. He knows more networking than me and zipped through those IOS and Catylst-OS screens faster than I could, but once the Semaphore guys came we just kinda stood in the corner. We'll try to get him badged tomorrow, and start training him on the LJ network.

Afterwards, beer with erik, henry, brett, jenn, Dana, Quentin, and some other girl whose name I forget. Have I ever mentioned how much I suck at names? :-(

So, back to work.
LinkReply

Comments:
[User Picture]From: xaosenkosmos
2004-06-24 11:50 pm (UTC)
How many machines do you guys push that much traffic out of? That's pretty fast for being mostly MySQL (we only do about 250Mbps at ibiblio, off about 8 machines and probably 80+% static content).

If you're like me, you can't remember names, but you can remember email addresses like nobody's business. I really like people who take their name or initials as their email address...
(Reply) (Thread)
[User Picture]From: brad
2004-06-24 11:59 pm (UTC)
80-some machines.

Almost all that traffic is compressed text, dynamically generated. Images are Akamai, for now, until our contract's up.
(Reply) (Parent) (Thread)
[User Picture]From: supersat
2004-06-24 11:51 pm (UTC)
Wow, 98.7 Mbps? Impressive! Now imagine what it'd be like if you were still serving images locally and not compressing. ;)
(Reply) (Thread)
[User Picture]From: chuck
2004-06-24 11:52 pm (UTC)
If you're still around seattle around 7 you can stop by for some beers. I'll email my number and address to your danga account.
(Reply) (Thread)
[User Picture]From: brad
2004-06-25 12:01 am (UTC)
Cool.
(Reply) (Parent) (Thread)
[User Picture]From: scsi
2004-06-24 11:52 pm (UTC)
First day of work eh? Talk about trial by fire. I hope he comes into work tomorrow.. :)
(Reply) (Thread)
[User Picture]From: blueroo
2004-06-24 11:59 pm (UTC)
Dylan and the rest of the folks at Semaphore are all super smart and very talented folks. I hope they've impressed you enough to tip the scales and bring LJ on to their network. That would be tres' cool. :)
(Reply) (Thread)
[User Picture]From: brad
2004-06-25 12:00 am (UTC)
In any case, I don't look forward to moving 4 full, dense racks of servers halfway across town.
(Reply) (Parent) (Thread)
[User Picture]From: blueroo
2004-06-25 12:17 am (UTC)
How would the users take to a weekend downtime? You don't have to answer that. Stupid questions don't require answers. :) I assume if you did this, you probably wouldn't move the entire thing at once but break it up in to bits and move them over the span of a week or so. Probably not a difficult task really, considering that you guys seem to have most resources broken up in to individual clusters already.

The biggest pains of relocating networks in my experience has been (in order of least to greatest tendancy to cause violence):

Moving hardware
Recabling, sorting electrical load issues, and the documentation of the same
Reconfiguring switches, routers, and servers

Big strong people at $15/hr make the first one easy. The second one is a good way to get exercise for a few hours. The last takes forever, every step is accompanied with a change to documentation, and it's easy to futz up or fat finger. Only pains-taking planning, as much automation as possible, and caffeine make it bearable.

But I'm sure you have better things to do than listen to me babble about obvious stuff. :P
(Reply) (Parent) (Thread)
[User Picture]From: brad
2004-06-25 12:43 am (UTC)
We have about two of everything, almost. I think step 1 is convert the few older user clusters from master-slave to master-master, then we'll have two of everything, and we SHOULD be able to move half of it, switch DNS, and move the other half.

We'll see.
(Reply) (Parent) (Thread)
[User Picture]From: edm
2004-06-25 05:02 pm (UTC)
Doing that (going master-master, moving half of it, switching DNS, moving rest) or moving one cluster at a time (by building onto new machines at the new location) would be how I'd handle it. Given your internal cluster architecture I suspect having clusters in different locations would just up your bandwidth requirements something awful during the transition (inter-cluster queries).

Certainly moving all the machines at once is a recipe for lots of downtime and a horribly stressful day. The production-network-migration I'm involved with at present (about 50 machines in production, fortunately only pushing about 10Mbps) is being done on a few-machines-at-time basis with appropriate IP redirection magic for this reason. Probably over the course of a couple of weeks.

BTW, I'm mighty impressed that you're managing to push 98.7 Mbps through a 100Mbps link (even with one system at each end, full duplex) -- typically I'd have expected it to be melting over about 90 Mbps sustained.

Ewen



(Reply) (Parent) (Thread)
[User Picture]From: cjcollier
2004-06-25 12:25 am (UTC)
Where'd you get your experience with Cisco networking hardware? I'm just amazed by your talent, Brad. :) You're an amazing programmer, a great businessman and you know more about networking than I expect I'll ever know.
(Reply) (Thread)
[User Picture]From: brad
2004-06-25 12:45 am (UTC)
I don't know shit about Cisco, really. I can run some diagnostic commands and change interface parameters, but that's pushing it for me.

I can tell you all about different protocols in theory and specification, but I can't setup a damn thing when it comes to real life networking hardware. I just plug switches into each other and hope for the best.
(Reply) (Parent) (Thread)
From: (Anonymous)
2004-06-25 09:03 am (UTC)
? is all you need to know to configure a cisco. Oh, and switchport is how you configure vlans. I'm sure I could pass the CCNA simply by using ? alone.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2004-06-25 09:21 am (UTC)
Hah. ? is about all I know!
(Reply) (Parent) (Thread)
[User Picture]From: octal
2004-06-25 12:27 am (UTC)
Semaphore is pretty awesome.

Why couldn't they take over your contract at the colo and provide services/support/bandwidth, and just put the NEW machines at a new site, eventually migrating entirely through normal replacement?
(Reply) (Thread)
[User Picture]From: brad
2004-06-25 12:46 am (UTC)
Our contract's almost up, so no big loss abandoning our contract. What we pay Internap is mostly overage charges. Our minimum is low.
(Reply) (Parent) (Thread)
[User Picture]From: chris
2004-06-25 12:34 am (UTC)
wow. I'm pretty sure fast ethernet effectively saturates at around 87%. Anything above that and people at my work start going into panic mode.

I know a couple of people from semaphore, and they seem to be good people.
(Reply) (Thread)
[User Picture]From: brad
2004-06-25 12:47 am (UTC)
My guess:

87% or so when you have separate hosts doing it.

98.7+% is possible when only a single machine is sending packets, timed correctly.

(Reply) (Parent) (Thread)
[User Picture]From: scosol
2004-06-25 12:39 am (UTC)
98.7?!?!?!?!

good god damn- i'll believe it when I see it myself :p

but yeah- with the shitload of machines that you have, that's the only way I could ever imagine getting that much utilization-

if you can imagine that a single machine has a certain amount of wash bw that collides, the only way to push above that threshold is to have many many machines, each adding that little bit of bw
but yeah- that's impressive- but at the same time it says "give me my gig-e!" :P
(Reply) (Thread)
[User Picture]From: brad
2004-06-25 12:48 am (UTC)
A single machine (one BIG-IP) was doing that 98.7, so no collisions.

The internal network is much different, but we're no longer congested there.
(Reply) (Parent) (Thread)
[User Picture]From: dakus
2004-06-25 12:59 am (UTC)
You GOTTA strap a machine to a goat and walk the goat to the new place...

%)

(Reply) (Thread)
[User Picture]From: erik
2004-06-25 09:29 am (UTC)
Cara (carebearc) was the other girl's name.
(Reply) (Thread)
[User Picture]From: brad
2004-06-25 09:46 am (UTC)
Ah, yeah, she looked familiar. I recognize her from all Blythe's posts ages back.
(Reply) (Parent) (Thread)
[User Picture]From: llarian
2004-06-25 11:08 am (UTC)
In all fairness, you got a sales guy who was along for the ride to watch us work, our company's president, the resident "Mr. Fix Anything", and the Sr. Developer who deies he knows anything about networking anymore but gets forced to do it anyways sometimes (Myself) =)

Honestly, I just came by accident. Garth was leaving for Livejournal just as I happened to be dropping off an expense report with him, so I decided to tag along. I've been wanting to see your setup for a while anyways.

Glad we could help somewhat!
(Reply) (Thread)
[User Picture]From: brad
2004-06-26 04:02 pm (UTC)
You all seemed to have a clue and work well together, though, so I was happy. :-)
(Reply) (Parent) (Thread)
[User Picture]From: muerte
2004-06-25 12:53 pm (UTC)
Is the next step in bandwidth gigabit ethernet? Just purely for cost, it may be cheaper to get a second 100mbit ethernet link and do Cisco EtherChannel either to your Cisco switch or to a Linux box (I believe it's in the Kernel).

Of course that obviously all depends on your network design. But it's an option. I'd love to play with EtherChannel stuff I just don't have enough bandwidth requirements to even need it yet.
(Reply) (Thread)
[User Picture]From: brad
2004-06-26 04:01 pm (UTC)
That's our band-aid fix, yeah.
(Reply) (Parent) (Thread)