Log in

No account? Create an account
marklar on the move - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

marklar on the move [Nov. 30th, 2005|02:12 pm]
Brad Fitzpatrick
[Tags|, , ]

danga:~# uptime
 14:05:15 up 286 days,  1:48, 21 users,  load average: 0.00, 0.00, 0.00
danga:~# halt

The ancient and dependable box, Marklar, long-time home of danga.com, fitzpat.com, bradfitz.com, updates.sixapart.com, cvs.livejornal.org, cvs.danga.com, and until earlier this year, also pics.livejournal.com and picpix.com, is now being halted and shipped to San Francisco.

I've been telling people it's turning out to be easier to move 150 boxes doing 1 function than 1 box doing 150 functions. Actually I guess neither was that hard, but both are boring.

So I've spent the past several days moving dozens of services off Marklar to other places, hence flaky mail/cvs/web/etc.

I'm getting it split into a bunch of small, easily-manageable Xen instances, each doing specific roles:

-- personal/family/friends websites
-- mail, webmail, anti-spam, imapd, etc
-- cvs/svn, and viewcvs
-- rt/zilla
-- danga's websites (and updates.sixapart.com)

So much easier working with Xen than one monolithic box, especially being able to realloc memory and disk, resizing filesystems and such.

Taking the opportunity to switch from uw-imapd crap to Dovecot and Maildir.

Still a lot not set up, though, so my boredom continues for another few days, I'm sure......


From: jmason
2005-11-30 11:03 pm (UTC)

more details!

I'm (still) curious to hear what you did with the filesystem layout for those instances. Entirely separate copies? NFS sharing of bits of /usr? Copy-on-write hairiness?

Also have you kept one aside as a "base", easily clonable for new instances?

So many questions...
(Reply) (Thread)
[User Picture]From: brad
2005-11-30 11:50 pm (UTC)

Re: more details!

Separate block devices, on LVM2, no NFS, using XFS as filesystem because it can be grown so easily. debootstrap makes a new debian image in no time.
(Reply) (Parent) (Thread)
[User Picture]From: jwz
2005-11-30 11:14 pm (UTC)
Wow, you're using Xen?

I think this is the first time I've heard anyone speak of it in anything but the hushed and reverent tones usually reserved for anticipation of The Hurd!
(Reply) (Thread)
[User Picture]From: brad
2005-11-30 11:49 pm (UTC)
I'm using Xen 2 (stable). Seems all the building hype is about Xen 3.
(Reply) (Parent) (Thread)
[User Picture]From: crucially
2005-12-01 12:39 am (UTC)
I have used for a while, the migration stuff is way cool if you got a shared block level device or NFS.

I know Bytemark are looking to move their entire user mode linux setup to xen.
(Reply) (Parent) (Thread)
[User Picture]From: erik
2005-11-30 11:28 pm (UTC)
Make sure they pack that thing carefully... it'd be tragic if anything happened to www.bradfitz.com/lisa/.
(Reply) (Thread)
[User Picture]From: ioerror
2005-12-01 12:43 am (UTC)
That's funny. Marklar was the name of my old linux server far far before OS X was revealed to be on X86.

I'm glad I'm not the only dork that did that.
(Reply) (Thread)
[User Picture]From: muerte
2005-12-01 03:35 am (UTC)
Moving to Dovecot to Maildir was pretty painless, and ultimately totally worth it. I highly recommend it for a small mail server.
(Reply) (Thread)
From: evan
2005-12-01 03:38 am (UTC)

(Reply) (Thread)
[User Picture]From: iamo
2005-12-01 03:43 am (UTC)
Hm, I've never heard of dovecot before. I use courier, but don't like it very much (too much crap comes with it). What in particular did you hear/do you know about dovecot that makes you want to switch?
(Reply) (Thread)
[User Picture]From: avatraxiom
2005-12-01 11:22 am (UTC)
Dovecot is great. It's simple, straightforward, actively developed, secure, and Just Works. It doesn't yet do ACLs or Sieve, but that's about it, pretty much. I've been using it for a year or so, now, with not a single problem. In fact, I think I haven't had to touch the config file since I first set it up.

(Reply) (Parent) (Thread)
[User Picture]From: whitaker
2005-12-01 04:07 am (UTC)
"Taking the opportunity to switch from uw-imapd crap to Dovecot and Maildir."

Omfgz I was praying. I've been trying to get my mail situation under control but uw-imapd (or something) has been a real bitch lately. I called a truce the other day hoping that something new using maildir would be set up on the new box. Thanks!
(Reply) (Thread)
[User Picture]From: scosol
2005-12-01 08:09 am (UTC)
*sigh* yeah- i've actually done the "pull the plug halfway out and jumper to a ups" thing before just to preserve my pristine uptime
it's an OCD thing

and hmmm- i seem to remember rejecting dovecot for some reason... weird security shit?
does djb not like it?
i don't remember-
i ended up just doing courier on localhost and mandating ssh tunnel access-
(Reply) (Thread)
[User Picture]From: iamo
2005-12-01 08:22 am (UTC)
If dovecot has weird security shit, then courier has jabberwocky security shit. I just switched my courier install over to it and it seems to run under daemontools well, if that's what you meant. Just pass it -F.
(Reply) (Parent) (Thread)
[User Picture]From: scosol
2005-12-01 06:08 pm (UTC)
ahh i remember now- (this was a year or so ago) i tried it and immediately had some weird behavior, so i then looked at the peculiar release naming/schedule/status and went with something else-
i guess it's gained some stability these days :)
(Reply) (Parent) (Thread)
From: funzoneq
2005-12-01 09:27 am (UTC)


$ w
10:27AM up 221 days, 12:53, 1 user, load averages: 0.00, 0.05, 0.07
funzoneq p0 10:27AM - w
(Reply) (Thread)
[User Picture]From: tijuanacartel
2005-12-04 02:48 am (UTC)
Whoa, you're using Xen and Debian? I'm assuming Xen 2.0.x and Sarge here, but feel free to correct me.

Xen 2.0.6 was a bit of a clusterfuck owing to the lockout of 2.6.11 kernel-source from the debian tree.

I do lots (and I mean lots) of work with Xen during my job, and we use Debian too. I've meant to write more docs on it but I havent had much time. (points at the last one)

So, curious. Did you roll your own debs, or use the ones from Adam Heath, or another source?
(Reply) (Thread)
[User Picture]From: brad
2005-12-04 03:45 am (UTC)
Xen 2.0.x and Sarge, yes.

No .debs... just installed new kernels (dom0 and domU) and installed the xend/xm tools from source.
(Reply) (Parent) (Thread)
[User Picture]From: tijuanacartel
2005-12-04 03:51 am (UTC)
I take it then that Xen is being used on just this box, and hasnt taken over the rest of Livejournal infrastructure, yet.

is Debian pretty much the main distro running there? Or do you also have rhel?
(Reply) (Parent) (Thread)
[User Picture]From: brad
2005-12-04 04:27 am (UTC)
We're not using Xen everywhere yet. Just starting. We have logical places for it just about everywhere, though, so it's only a matter of time.

No RHEL. Only Debian (sarge nowadays), everywhere.
(Reply) (Parent) (Thread)
[User Picture]From: tijuanacartel
2005-12-04 07:23 am (UTC)
Good man.

Debian and Xen powering the worlds biggest MySQL cluster. Now if only all the glue code in Python you'd probably be my ideal workplace ;)

(Shameless plug. If you know anyone looking for an experienced Debian/Xen hacker with a lean towards high availibility/clustering, please let me know, or please let them know I exist. My talents are being wasted here. Thanks.)
(Reply) (Parent) (Thread)
[User Picture]From: brad
2005-12-04 07:29 pm (UTC)
Apply! We have several openings that you'd probably fit into:


Copy me (brad@danga.com) on your resume too, so I'll know your real name and can put in a good word.
(Reply) (Parent) (Thread)
[User Picture]From: tijuanacartel
2005-12-04 10:46 pm (UTC)
You're right! I think i'll apply for the hell of it. But at the moment I'm in New Zealand, so getting over there for the interview might be a little difficult ;)
(Reply) (Parent) (Thread)
[User Picture]From: brad
2005-12-04 03:48 am (UTC)
And nice post to the xensource list! I love DRBD and Xen ... I've yet to combine them, but your guide will save me a lot of time. Thanks!
(Reply) (Parent) (Thread)