| LJ Talk activity (or, ejabberd vs djabberd) |
[Oct. 16th, 2006|12:40 pm] |
I've been watching our LJ Talk ganglia stats and also comparing them to the Jabber.org status (which runs ejabberd).
Our memory usage, even with a known memory leak, is way better. ejabberd seems to take 184 kB/connection, while djabberd is currently using 34 kB/connection. (which includes leaked data .... when it starts it's closer to 5 kB/connection)
In a week or two it looks like our connected clients will overtake jabber.org's too, at least with our current rate of growth. They currently peak at ~10,000 users. Our peak, currently at 4,000 users, keeps climbing each day, from a peak of just 1,000 a few days ago.
At least it's really easy now to track down memory leaks, using Devel::Gladiator and $^P |= 0x200, and Devel::Peek::CvGV .....
All objects in memory ...
( Large dump.... )
So that'll give me something to do on the plane, too. |
|
|
| SixApart joins the Jabber world |
[Jun. 28th, 2006|07:22 pm] |
The SixApart employee Jabber server now has server-to-server enabled, so all @sixapart.com email addresses are also valid Jabber JIDs and publicly-reachable from GTalk, jabber.org, etc. And vice versa.
I have no hope for AIM or MSN speaking Jabber (or any open federation protocol) anytime soon, but what about Yahoo? They're trying really hard lately to be open. Zawodny? |
|
|
| Djabberd connections, continued... |
[Jun. 27th, 2006|02:30 pm] |
Just did 174,700 connections with 606 MB of memory. That's 3.5kB/connection, inclusive the initial 14 MB startup size. |
|
|
| djabberd: c10k? hah! |
[Jun. 26th, 2006|10:09 pm] |
DJabberd just did 25,200 (fully setup) connections with 97 MB of RAM before my Xen instance ran out of memory. It's now 3.4kB of overhead per connection (contrast to 30kB this morning) but there's still obvious ways to trim it down. Should be able to get it down to 2kB. The big win was when I implemented a [forget design pattern name] system where libxml parsers are shared, returned, kept on a freelist, etc.
From what Artur and I can tell, this is better than most/all the other jabber servers out there.
It means with 1GB of ram we can do 300k connections per process. (8GB of RAM boxes, 2x 2x core)
<3 epoll. |
|
|
| My new JID, or how to run DJabberd on Debian |
[May. 14th, 2006|11:48 am] |
Now that s2s is secure and memory leaks are fixed, I'm now running djabberd on my personal server/domain.
My new canonical instant messaging JID is now brad@fitzpat.com . Chat me up. (if you're using something which uses Jabber for IM interop, which includes Google Talk and I think Gizmo too....)
If you want to run this yourself on Debian, here's a short guide:
$ wget http://danga.com/debs/libdanga-socket-perl_1.51-1_all.deb
$ wget http://danga.com/debs/libsys-syscall-perl_0.1-1_all.deb
$ sudo dpkg -i libsys*.deb libdanga*.deb
$ sudo apt-get install openssl libdbd-sqlite3-perl \
libnet-ssleay-perl libnet-dns-perl libdigest-sha1-perl \
libxml-sax-perl libxml-libxml-perl liblog-log4perl-perl
subversion libdigest-hmac-perl
$ svn co http://code.sixapart.com/svn/djabberd/trunk/ djabberd
$ cd djabberd
$ openssl req -x509 -newkey rsa:1024 -keyout server-key.pem \
-out server-cert.pem -days 365 -nodes
$ htdigest -c djabberd.users djabberd [your-username-without-at-sign](enter password twice)
Make a conf file. mine is:$ cat fitzpat.conf
OldSSL enable
<VHost fitzpat.com>
S2S enable
RequireSSL yes
<Plugin DJabberd::Authen::HTDigest>
Realm djabberd
HtDigest /home/bradfitz/djabberd/djabberd.users
</Plugin>
<Plugin DJabberd::RosterStorage::SQLite>
Database /home/bradfitz/djabberd/fitzjabber.sqlite
</Plugin>
</VHost>Then run it:
$ ./djabberd --conf=fitzpat.conf
Or daemonize it with --daemon|-d.
Enjoy. |
|
|
| DJabberd update |
[May. 13th, 2006|10:53 pm] |
Artur and I have been hacking all weekend on DJabberd. Major accomplishments:
-- switched from expat to libxml
-- fixed a ton of memory leaks
-- fixed a ton of TODOs/FIXMEs, much more spec compliance (we've been graphing this, it's quite fun)
-- DJabberd::Authen::HTDigest -- use your existing apache tools to 'provision' accounts.
-- a dope ass test suite with lots of helpers to make writing jabber-specific tests easy
-- lots of tests
-- lots of bugs fixed as a result of tests
-- lots of progress made quickly because of tests finding regression
-- iChat weirdness magically went away as a result of more spec compliance
Here's the graph of FIXME/TODO over time (well, revisions) so far.... we started this weekend at revision 212:
( Graph... ) |
|
|
| DJabberd, PAM, simple config... |
[May. 6th, 2006|10:57 pm] |
With PAM support and automatic plugins when you don't specify plugins for a given phase, a simple DJabberd server config is down to:OldSSL enable
<VHost jabber.bradfitz.com>
S2S enable
RequireSSL yes
<Plugin DJabberd::Authen::PAM />
<Plugin DJabberd::RosterStorage::SQLite>
Database roster.sqlite
</Plugin>
</VHost> |
|
|
| DJabberd config file |
[May. 6th, 2006|09:28 pm] |
Ahhh yeah, Artur and I just added Apache-style conf to DJabberd so you don't have to write a Perl file with a bunch of server/vhost/plugin constructors by hand. Now it looks like:
$ cat djabberd.conf
OldSSL enable
# defaults:
ClientPort 5222
ServerPort 5269
#PerlModule DJabberd::SixApart
<VHost jabber.bradfitz.com>
S2S enable
RequireSSL no
<Plugin DJabberd::Authen::AllowedUsers>
Policy accept
AllowedUsers brad crucially test
</Plugin>
<Plugin DJabberd::Authen::StaticPassword>
Password password
</Plugin>
<Plugin DJabberd::PresenceChecker::Local />
<Plugin DJabberd::Delivery::Local />
<Plugin DJabberd::Delivery::S2S />
<Plugin DJabberd::RosterStorage::SQLite>
Database roster.sqlite
</Plugin>
</VHost>
$ ./djabberd --conf=djabberd.conf
Next step is removing the need for the must-have plugins and making them automatic. That means adding plugin-declared before/after dependencies so DJabberd can do the hook ordering. Should be easy. Then DJabberd should be getting ready for usability by the masses. :-)
Update: the as_bool function was fun:sub as_bool {
my $val = shift;
return 1 if $val =~ /^1|yes|true|t|on|enabled?$/;
return 0 if $val =~ /^0|no|false|f|off|disabled?$/;
die "Can't determine booleanness of '$val'\n";
} |
|
|
| DJabberd: vhosts, s2s improvements... |
[May. 4th, 2006|12:37 am] |
Artur and I made some nice progress in DJabberd tonight. We split the server class into server and vhost classes, so a server can now have multiple vhosts. We were going to write an inter-server delivery plugin for messages sent between vhosts (and probably still will) because the local-node delivery plugin is actually local-vhost-node only, but the normal s2s delivery plugin worked... it just connected to itself. Cute.
Also removed a number of FIXME/TODOs, which is always nice. I need to graph counts of FIXMEs and TODOs as a function of time and/or svn revision number. :-)
A notable fix was removing the hard-coded server-secret used for dialback verifications. It now, in the single-node case, can auto-generate a server secret but it's setup in such a way that we can easily add in a new plugin hook phase for a cluster-wide server secret with minimal work. That way we can hook into LJ's get_secret stuff.
Also did some stuff that wasn't in the spec as far as we can see, about sending gratuitous directed presence after an inbound subscribe when the server is supposed to automatically reply with "subscribed" without bugging the user. In that case (servers getting out of sync wrt rosters), you also need to let the other party know the presence of the user, at least it feels that way.
Blahblahblah....
Point is: almost time for a CPAN release and getting more people using it.
It's pretty fun using it at work, hooked up to LDAP. Don't have to use AIM anymore. |
|
|
| DJabberd authentication system |
[Apr. 13th, 2006|12:02 am] |
With ![[info]](http://l-stat.livejournal.com/img/userinfo.gif) hachi's LDAP wizardry and some ghetto Perl, I got the SixApart chat server up today. I subclassed the SQLite RosterStorage plugin and made it so when any employee logs in, it adds to their roster any employee that they're missing. Further nickname changes and group modifications are respected (that falls through to the SQLite layer), as well as the addition of non-SixApart JIDs, but the roster must always contain at least all employees, and automatically in the two-way approved state for presence. No more asking the new employee what his/her AIM name is, then going through the approval "add me to your whitelist!" crap.
But currently everybody's password is "password" because the authentication system isn't built out nicely yet.
I've been contemplating how to do it generically, so plugins can just declare what sort of authentication world they're living in, and DJabberd can then advertise on to the client the right auth modes which'll work with the underlying authn plugin.
In summary, I think the major worlds are:
1) The easiest, whereby client provides plaintext password (over SSL), and plugin can check it. A lot of times the client can provide the cleartext but the server doesn't know the cleartext because it's protected by, say, LDAP, or stored as only a digest on the server. Disadvantage is this requires SSL, but this is where a lot of auth plugins would have to go. And once client provides plaintext, DJabberd can then do whatever auth foo is required to test it.
2) But if the server knows the cleartext and can declare that, and provide the cleartext to Djabberd on request, a lot more auth options are available. (challenge-response stuff)
3) Finally, if user won't provide cleartext (no SSL) and server can't fetch the cleartext, then you hope there's an upstream auth provider that speaks SASL, and you just proxy SASL challenge/responses back and forth. A lot of LDAP servers let you speak SASL. I think ours at work will, but it's disabled? So I think I'll need to make our authn plugin do option 1) above, where we get plaintext passwords over SSL and then try to bind to the LDAP server. And that's what's inspired this post: DJabberd currently doesn't do SASL, and doesn't ask for cleartext passwords ever (when it advertises auth methods).
But too tired. Just dumping thoughts.
What other auth types am I missing that don't fit into those 3 categories? |
|
|
| DJabberd Status Update |
[Apr. 9th, 2006|11:54 pm] |
Tons of work on roster and presence in DJabberd tonight. This is where it's getting really fun. Roster stuff was easy/tedious, but required. Also lots of boring refactoring.
But the end result is that I can subscribe/ack presence requests now, have my roster update (SQLite plugin), and see my account on jabber.org change status now from my other Jabber client connected to DJabberd, subscribed to the jabber.org presence. Fun fun.
BTW, this will all run on POE once hachi bridges the Danga::Socket and POE worlds. So DJabberd plugins (which are all async by design) can use POE components. |
|
|
| DJabberd Status: bug fixes, and roster flesh-out |
[Apr. 8th, 2006|07:27 pm] |
Lest you thought I'd forgotten about it, I'm back to hacking on DJabberd. But without Artur today.
Today:
-- SSL fixed. I'd broken it during earlier refactoring.
-- iChat support fixed. it sends whitespace where other clients don't, and I was using the wrong method which wasn't skipping whitespace.
-- Roster Add(/Update)/Delete -- but only the place where I call the hooks. haven't updated the SQLite plugin to respond yet. also haven't done global roster pushes yet, which requires a hook phase for inter-node communication. should write a reference implementation of that using Spread.
Actually, that's not much. I'd hoped to finish more, but kept getting distracted. There was also some re-reading of the specs and code to catch back up.
Plan:
-- finish roster hooks/APIs to the point where I can finish presence, which is largely untouched, but some of which kinda works already because we route messages around between users, even if we can't understand their magic.
-- get it all documented
-- make sure it works on a single node easily
-- run it on my own server, start testing it
-- use it in the office, with LDAP auth and roster support. (all of the company is on roster by default)
-- finish SASL
-- work more on LJ-specific internode communication and offline storage (not using SQLite! :))
-- ESN delivery to Jabber.
-- Jabber/LJ integration.
-- World Domination. |
|
|
| fakesms server |
[Apr. 5th, 2006|06:37 pm] |
The cool thing about having an extensible Jabber server/baseclass is that I can crank out crazy Jabber servers in minutes.
For testing LJ's upcoming SMS integration, we needed a way for developers to pretend they were sending and getting SMSes from their developer LJ installs. Also, non-technical employees needed to test, so command-line clients are out. Web stuff is too painful async. Working with carriers/SMS aggregators is 10x more painful than all that combined.
Solution? Jabber server, of course.
I forgot about this until I got a Google News Alert about it:
http://cvs.livejournal.org/browse.cgi/livejournal/bin/fakesms-djabberd
Users log in with username of 5551212 (or whatever phonenumber) and a password of "smstest" (arbitrary), then it's like they have that cellphone, from LJ's point of view.
Here's the server:
http://cvs.livejournal.org/browse.cgi/livejournal/bin/fakesms-djabberd?rev=1.3
The only new code are the two FakeSMS modules:
http://cvs.livejournal.org/browse.cgi/livejournal/cgi-bin/DJabberd/Delivery/FakeSMS.pm?rev=1.1
http://cvs.livejournal.org/browse.cgi/livejournal/cgi-bin/DJabberd/RosterStorage/FakeSMS.pm?rev=1.1
First one delivers messages into LJ (misc/fakesms.bml endpoints, only enabled for developers), and the other one just puts the sms@$SERVER item into users' rosters, to ease testing.
(SMS from LJ to Jabber is handled by the LJ code when $IS_DEV_SERVER) |
|
|
| DJabberd Status: Rosters |
[Mar. 19th, 2006|12:23 am] |
Artur and I worked on rosters tonight. Nice, clean hooks and abstractions now....
Data structures:
DJabberd::Roster
DJabberd::RosterItem
Plugins:
DJabberd::RosterStorage -- abstract base class
DJabberd::RosterStorage::SQLite -- functional
DJabberd::RosterStorage::Dummy -- example
DJabberd::RosterStorage::LiveJournal -- LJ integration
All in cvs. Check it out.
Lot more to do yet:
-- finish Roster API (adding/removing items)
-- do roster pushes to all connected clients (easily locally, more fun with a cluster)
-- presence (big one, but roster pushes will get it a lot of the way there)
-- easy way for plugin authors to note certain functions as blocking and run them in child thread that won't stall the event loop (Jonathan Steinert's working on some Gearman modifications to make this transparent and easy, whether it's in a local process or anywhere else on the network...)
-- finish LJ integration |
|
|
| DJabberd Status Update |
[Mar. 17th, 2006|12:11 am] |
With this commit to DJabberd, Artur and I finished s2s support, both in/out, with all the hook chains, filtering, switching, delivery, etc.
Artur was on my server, I was on jabber.org, and we chatted. OMG LOL.
Next up:
-- fix some lingering SSL bugs for some clients. easy, but boring.
-- presence. not done at all.
-- cleanup roster, roster pushes, etc, etc. very basic support now.
-- clean up auth. add SASL.
Fun.
Anybody care to start hacking on this with us, now that it's starting to become useful? mart? |
|
|
| DJabberd Status Update |
[Mar. 16th, 2006|09:48 am] |
Artur and I worked on DJabberd last night after work. We:
-- designed the flow of client-incoming and server-incoming messages:
-- filtering, switching (deciding local processing vs. delivery), and then processing or delivery
-- built out a ton of hookchains for said phases
-- implemented default fallback hooks for said phases, adhering to specs
-- annotated more of the specs, so our must-implement-specs test case has more tests
-- documented another ambiguity in the spec
-- started separting out the "Server" object from the "Vhost" object. (a server HAS 1+ vhosts)
Then we built DJabberd::Delivery::Local
Which just does local delivery. (currently just local vhost, but in future local vhost first, then local server, but injecting the message back through the "almost beginning" of the other vhost's filtering hookchain, just bypassing all XML crap, so that vhost's filtering/routing hooks still run all normally.)
The big accomplishment is that message delivery then works. Both client-incoming and server-incoming messages eventually end up in either server-process or deliver, or client-process or deliver. (so the deliver is shared, once filtering/switching is done)
So we were IMing each other from same node, and from GTalk, Jabber.org, etc.
Now that we have the delivery framework, we need to build DJabberd::Delivery::S2S so we send outgoing messages back to GTalk/Jabberg.org/etc. (we'll also be building DJabberd::Delivery::LocalCluster after that to do inter-domain delivery for a load-balanced/HA jabber installation.....) Outgoing S2S will be easy considering we already do outgoing connections for Dialback and such. Just had to build out the delivery framework first before we could even really test it. |
|
|
| DJabberd status update |
[Mar. 9th, 2006|10:10 pm] |
Got a bit more Djabberd work done today. It's coming along nicely. One of the big s2s hurdles, not working with GTalk, is now fixed, due to a quickly-mentioned part of the spec that I overlooked. (and which is a MUST which depends on a SHOULD ... spec bug?)
At this point I need to finish outgoing s2s (should be damn easy given all the infrastructure which has been built out), fix one notable SSL bug (should be straight-forward, but boring, now that I've read more about OpenSSL programming), and then finish up the routing/delivery framework which I spec'ed up today and mostly finished.
But tomorrow is Seattle, then Portland, so I'm not sure how much productive hacking I'll get done without consistent Internet access.
Also: we now have our SMS gateway starting to come online. We're linked up with Sprint now, waiting for the other carriers. Then LJ's SMS features will start to appear. |
|
|