Log in

No account? Create an account
djabberd: c10k? hah! - brad's life [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

djabberd: c10k? hah! [Jun. 26th, 2006|10:09 pm]
Brad Fitzpatrick
[Tags|, , , ]

DJabberd just did 25,200 (fully setup) connections with 97 MB of RAM before my Xen instance ran out of memory. It's now 3.4kB of overhead per connection (contrast to 30kB this morning) but there's still obvious ways to trim it down. Should be able to get it down to 2kB. The big win was when I implemented a [forget design pattern name] system where libxml parsers are shared, returned, kept on a freelist, etc.

From what Artur and I can tell, this is better than most/all the other jabber servers out there.

It means with 1GB of ram we can do 300k connections per process. (8GB of RAM boxes, 2x 2x core)

<3 epoll.

[User Picture]From: dormando
2006-06-27 05:13 am (UTC)
How are you doing the memory profiling? That's incredible :)
(Reply) (Thread)
[User Picture]From: brad
2006-06-27 05:15 am (UTC)
Gtop for these measurements, but as part of the slimming process: a lot of Devel::Size, Devel::Peek, Data::Dumper and (as of today) Artur's new Devel::Gladiator.
(Reply) (Parent) (Thread)
[User Picture]From: muerte
2006-06-27 03:02 pm (UTC)
Devel::Gladiator hahaha that rocks. What does it do?
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-06-27 04:49 pm (UTC)
Walks the arena.
(Reply) (Parent) (Thread)
[User Picture]From: awwaiid
2006-06-27 11:45 pm (UTC)
That one publically available somewhere? Don't notice it on CPAN.
(Reply) (Parent) (Thread)
[User Picture]From: veroz
2006-06-27 05:24 am (UTC)
Holy shit.
(Reply) (Thread)
[User Picture]From: xb95
2006-06-27 06:41 am (UTC)
You'll have to explain this to me - how the hell can you get 300,000 connections on a single machine? Won't it run out of fds?

Or can you set the max to something higher in the kernel? But then don't you run out of kernel fd memory?
(Reply) (Thread)
From: (Anonymous)
2006-06-27 06:51 am (UTC)
Dude, autotuning of available fd's have been in linux since 2.6.
(Reply) (Parent) (Thread)
From: divelog
2006-06-27 06:50 am (UTC)

# XXX - Hack

Hmm so did you fix the fugly exec call to ldapsearch I imagine :)
(Reply) (Thread)
[User Picture]From: brad
2006-06-27 03:44 pm (UTC)

Re: # XXX - Hack

Nope. That's just in the SixApart office subclass of DJabberd. I did change it to cache the company directory for an hour though, figuring we're hiring at less than one new employee per hour. :-)
(Reply) (Parent) (Thread)
[User Picture]From: mrgatsby
2006-06-27 06:55 am (UTC)
Wow! Can I get just the vanilla c25k server framework, without jabberd?
(Reply) (Thread)
[User Picture]From: brad
2006-06-27 04:50 pm (UTC)
That's just epoll. If you want to make event-based programs easily check out Danga::Socket or POE on CPAN.
(Reply) (Parent) (Thread)
[User Picture]From: robbat2
2006-06-27 08:13 am (UTC)
unless there's some info I'm missing (like how many IPs the box has), you'll run out of ports before the 300k connections. IPv4 TCP is a bitch, but we gotta live with it until IPv6 TCP6 is a lot more mature (read: north american residental ISPs start offering IPv6).
(Reply) (Thread)
[User Picture]From: brad
2006-06-27 03:42 pm (UTC)
Well, I'm not opening ephemeral ports. Port exhaustion is the client's problem.
(Reply) (Parent) (Thread)
[User Picture]From: dossy
2006-06-27 02:14 pm (UTC)
Is DJabberd threaded (can Perl do threads?) or all entirely event-driven?
(Reply) (Thread)
[User Picture]From: brad
2006-06-27 03:43 pm (UTC)
event-driven. Although child processes speaking over socketpairs (or processes elsewhere on the network speaking over tcp) can do slow blocking operations on behalf of the non-blocking event loop. Like database lookups, etc.
(Reply) (Parent) (Thread)
[User Picture]From: dossy
2006-06-27 03:54 pm (UTC)
Cool, so it's multi-process with some method of inter-process communication. Very sexy.

LJ needs to open a New Jersey office, so I can apply to work there. For reals. :-)
(Reply) (Parent) (Thread)
From: jzedd
2006-06-27 03:28 pm (UTC)
thats...awesome. wow.
(Reply) (Thread)
[User Picture]From: markpasc
2006-06-27 10:28 pm (UTC)
You made me try to think of the twelve letter word for scalability that starts with c and ends with k, before I looked it up.
(Reply) (Thread)
[User Picture]From: scosol
2006-06-28 12:43 am (UTC)
> [forget design pattern name]

yeah- i don't know if it's been formalized-
i call it a "handler pool"-
in the (best known in apache as the threadpool) model of spawning an anticipated number of X-handlers and then using them as needed-
of course, it's the event-driven interface behind them that enables the scalability, but it saves cycles and memory to have a semi-fixed number of objects there at all times, and sacrifice ultimate responsiveness for scalability-

i'm currently adding such functionality to Io's socket interface :)
(Reply) (Thread)
[User Picture]From: visions
2006-06-28 05:26 am (UTC)
sounds like things are coming together. has anything panned out of your work with jive? (we use their open source server at work for company communications)
(Reply) (Thread)