| Fedora Core, Perlbal, ... |
[Mar. 8th, 2008|08:37 pm] |
A new Fedora Core release is coming out soonish, which includes Perl 5.10. The last remaining package in Fedora Core which didn't work with Perl 5.10 was Perlbal. Whoops. Turned out I did some fancy things with reblessing fields objects. That doesn't quite work in 5.10 (arguably their fault).
Fortunately the Perl community had a patch for me and I just had to make a new release tonight, and ended up going through my inbox (and making a gazillion new gmail filters) and applying a ton of patches.
Yay getting caught back up on open source projects.
Time to get my drink on.
I'm going to attempt going to a party I tried to go last night, only to find out I had the wrong day and the party's actually tonight. Yay me. |
|
|
| readahead / blocking sendfile |
[Jun. 5th, 2006|11:34 pm] |
I've had a known inefficiency in Perlbal for ages now and finally broke down and fixed it. The inefficiency is that sendfile can block, even if the destination fd is a non-blocking socket, because the source fd (a disk-based file), can force a disk read if it's not already in pagecache.
FreeBSD has a fancy sendfile that lets you request it not block, but Linux doesn't.
The solution on Linux is to do a readahead() call first in another thread, or just sendfile() in another thread, either of which IO::AIO can do. I wanted to test the theory changing as little code as possible, so I went with the async readahead.
Before I did that, though, I wrote a test case.
The test case runs two processes in parallel: one fetching 3 small hot files over and over again, measuring the mean speed of 100 requests. The other process is there just to mess with the first one: it doesn't actually output anything. The second process either fetches the same 3 small files, or with the "big" parameter, fetches seven 100MB in a loop, more than this xen instances's 512 MB of memory. The idea is see if the disk reads serving the big files stall the event loop and decrease turn-around time.
Yup:
lj@LJ_web:~$ ./parallel.pl small; ./parallel.pl small; ./parallel.pl small; ./parallel.pl big;./parallel.pl big;./parallel.pl big;
mean: 0.287987213134766, stddev: 0.0829109309255669
mean: 0.279777903556824, stddev: 0.0957734761804354
mean: 0.238886480331421, stddev: 0.0949280425469577
mean: 0.351436612606049, stddev: 0.0791952577383974
mean: 0.361295075416565, stddev: 0.0863025646086743
mean: 0.3904807305336, stddev: 0.173639453608837
The first set of three lines is the time to serve small files with other small files being served in the background. The second send of three is serving small files with big files being served in the background.
Adding in the async readahead() call, doing the sendfile in the callback (once the data to be sendfile'd is in the pagecache), and the results even out a bunch:
lj@LJ_web:~$ ./parallel.pl small; ./parallel.pl small; ./parallel.pl small; ./parallel.pl big;./parallel.pl big;./parallel.pl big;
mean: 0.296060967445374, stddev: 0.0586433388625736
mean: 0.262518639564514, stddev: 0.0726501212827927
mean: 0.285000162124634, stddev: 0.0321991597111094
mean: 0.302280473709106, stddev: 0.0811435349061447
mean: 0.303003549575806, stddev: 0.0787071540895621
mean: 0.298841729164124, stddev: 0.0953137343692458
Probably some more work to be done, but promising. |
|
|
| splice() |
[Mar. 30th, 2006|10:38 am] |
Is anybody else excited about the in-development splice() system call?
I've wanted this for, like, ever.
I need to add support to Sys::Syscall so Perlbal can use it, avoiding copies to/from userspace to/from sockets. |
|
|
| sockets and close/shutdown/RST packets |
[Sep. 13th, 2005|02:30 pm] |
Dear Lazyweb,
Network programming question:
I have a listening socket. Client connects to it, and I write a bunch of data to it, in chunks as it becomes writable. After I'm done writing and want to properly close the connection, what do I have to do?
a) just close the socket?
b) shutdown and close the socket?
c) wait for socket to become writable again, and (a) or (b)?
I'd thought it was just (a), but Perlbal is sending RST packets like crazy when I do that. If I put in arbitrary delays before the close, the RST packets go away, but that's really hacky and lame.
We'd never noticed this on LiveJournal.com before, because the BIG-IP sanitizes the situation to some degree, but RST packets still reach end users, and it's technically wrong. Also, other sites (like discogs.com) which are behind Alteons don't handle the RST packets well.
*braced for cluestick beating*
Educated me. Thanks! |
|
|
| splicer? |
[Aug. 24th, 2005|01:28 pm] |
Follow-up on this post, what about the name splicer for Perlbal?
Advantages:
-- no longer a perl built-in function
-- doesn't conflict with splice() system call
-- 10x less google hits than "splice"
BTW, anything with "web" in it is out, because Perlbal will do non-web load balancing soon enough. "Net" might be okay.... NetSplice? NetSplicer? Not a huge fan.
Dina randomly suggested "Mango" because it was pretty sounding/tasting.
I'm down for any ideas. |
|
|
| Perlbal name change imminent |
[Aug. 23rd, 2005|09:02 pm] |
A lot of people have been telling me lately what I've long suspected: The name "perlbal" is fucking terrible. I suppose this is good, though... it means people are using it, even if they hate the name.
But I think I should rename it sooner than later. People hate the name because:
-- it has Perl in it, so people think it only works with Perl. (docs.php.net is behind Perlbal, incidentally)
-- it has Perl in it, so people think it's ugly/slow/____ (insert Perl predisposition)
-- the "bal" part is kinda useless
And I hate it because it was a working name that never changed.
After lots of thinking, I'm leaning towards the new name splice, but I'd like your thoughts.
Rationale on splice:
-- would get good Google loving. hell, the simple word "pound" is google #1 hit.
-- technically the correct word: we're splicing together connections
-- nice ring for non-tech people
-- one day we might use TCPSP for TCP splicing, which advertises it'd be good for "layer-7 switching, web proxy and application firewall running in the user space".... which is us.
-- one day we might use Larry McVoy's long-proposed splice() system call, now being worked on again after many years by Linus Torvals.
Pros, cons? |
|
|
| mod_proxy_balancer |
[Aug. 17th, 2005|02:19 pm] |
Just yesterday I was bitching about how bad mod_proxy was, but one of my biggest complaints is being addressed in the upcoming Apache 2.2:
http://httpd.apache.org/docs/2.1/mod/mod_proxy_balancer.html
Still looks like it's lacking, but at least it's becoming a load balancer now, without having to use mod_rewrite and an external rewrite program. |
|
|
| Perbal speaks SSL |
[Aug. 16th, 2005|05:02 pm] |
Perlbal can now do SSL like pound, speaking SSL to clients, but plain HTTP to the backends.
And it was only like almost no work thanks to IO::Socket::SSL.
I'm sure there's some more work to do here and there, but I didn't expect it to just work after so frickin' little effort. |
|
|
| Never ending feed of Atom feeds |
[Aug. 16th, 2005|12:58 pm] |
An increasing number of companies (large and small) are really insistent that we ping them with all blog updates, for reasons I won't rant about.
Just to prove a point, I flooded a couple of them and found that sure enough, nobody can really keep up. It's even more annoying when they don't even support persistent HTTP connections.
So --- I decided to turn things on their head and make them get data from us. If they can't keep up, it's their loss.
Prototype: (not its final home)
$ telnet danga.com 8081
GET /atom-stream.xml HTTP/1.0<enter>
<enter>
And enjoy the never ending XML stream of Atom feeds, each containing one entry. And if you get more than 256k behind (not including your TCP window size), then we start dropping entries to you and you see:
<sorryTooSlow youMissed="23" />
I think soon we'll get TypePad and perhaps MovableType blogs all being sent through this. The final home will probably be on a subdomain of sixapart.com somewhere, including documentation better than this blog entry.
And yes, I'm sure my Atom syntax is bogus or something. I spent a good 2 minutes on that part of it. |
|
|
| Perlbal release |
[Jul. 26th, 2005|01:29 pm] |
New Perlbal release (announcement, which I'm incredibly proud of from a software engineering point-of-view. Code's cleaned up a lot where it had grown unwieldy, lot of new features, old features removed, it has a nice big test suite, docs of options/defaults/types are integrated into the code where they're required to be maintained because they're used for boths docs and normal command processing, it's very portable, etc., etc...
http://www.danga.com/dist/Perlbal/Perlbal-1.3.tar.gz
Give it a whirl for all your load balancing and/or web serving needs! |
|
|
| monobal 0.1; more C# questions |
[Dec. 23rd, 2003|12:37 am] |
I've named my C# reverse proxy load balancer "monobal", sounding like "cannibal". (stress on first syllable, not 1st and 3rd) I named it that mostly because I was afraid of any legal implications of putting "sharp" or "dot" or "dot net" in the name, but also to promote mono, and because all those "SharpFoo" projects are getting lame, like KFoo and GFoo in KDE and GNOME lands, respectively.
Anyway, it kinda works now. Shitloads of FIXMEs and TODOs, but it's working some of the time now, on some types of requests. :-)
Instead of forking off a thread per client (which would've been easy, but I'm a sucker for pain), I did all async IO with callbacks. And because .NET's async IO is done with threads (thread pools), the callback can be done in another thread, I get to deal with synchronization issues as well! More fun.
At this point you're probably thinking: uh, if async IO is done with threads, why didn't you just use threads in the first place with blocking IO? I wondered the same after discovering that, and still am a little, but I observed that it's really efficient with its thread pools. There isn't a thread for each connection... it just uses a pooled thread to do IO whenever necessary. And mono is even using Linux 2.6 syscalls... saw it doing futex(). Go mono!
Um, C# wizards.... anybody know the interactions between doing async IO (BeginWrite) on a BufferedStream wrapping a NetworkStream? The callback never seems to be run after myBufferedStream.BeginWrite(...). I'd expect the BufferedSteam to at least invoke my callback saying, "Yo, I added your crap to my buffer, even if I'm not telling you whether or not I'm pushing it out to the NetworkStream yet." And I don't even get the callback(s) after doing myBufferedStream.Flush(). I mean, the whole program runs and I get the data in my test HTTP client, but I never got the callbacks to all my writes to the client. Mysterious.
Eh... I can probably just not use the BufferedStream class since I'm pretty much doing its job already, but I want to understand it.
Oh, and as an update to my earlier rant about working with strings without encodings, and doing regular expressions on buffers, ignoring but preserving 8 bit data: The Encoding class only has static instances of Encoding.ASCII, .UTF8, .UTF7 and some other Unicode-ish ones, so I didn't think to just do: Encoding.GetEncoding("ISO-8859-1"). Or any other encoding with 256 unique byte/char mappings. So I can take Mr. Anonymous Unknown8bit buffer, assume it's Latin-1, fiddle with it as a String ignoring high bytes, then go back to a byte buffer using the same encoding, even if it was actually Russian or something.
After learning C#'s networking and IO classes a bit more, next step is abstracting out the backend node selection class. (round robin vs. random vs. custom)
Fun fun. |
|
|
excited