April 19th, 2004


today rocks

Good things about today that aren't computer-related:

-- Music (Modest Mouse and Aesop Rock)
-- Weather (pouring rain)
-- Bike (picked up my Trek Fuel 90 2004 w/ disc brakes)

But the main cool thing: I worked from 8:30am - 6:30pm on my web server/proxy and the time just flew. In summary, it is/has:

-- written in Perl, but using Linux epoll and sendfile (IO::Epoll, IO::SendFile)
-- all async I/O
-- all event-driven (no need for synchronization! so easy to program!)
-- beautiful code & class hierachy which the event framework calls into
-- class for parsing/manipulating HTTP headers, both response and request, including merging headers, but not Set-Cookie headers in HTTP responses.
-- beautiful infrastructure for read/write buffering
-- proxy can read-ahead the backend node's response into a configurably-sized buffer (default 250k) and close down the backend's connection while the buffer is fed to the client. if page is 280k, 250k will be buffered, and once client has 30k, the 250k buffer will now contain the end, and the backend will shut down.
-- supports reproxying: the backend web node (say, a huge mod_perl process) can return the HTTP response headers (content-length, mime-type, etc) but tell the proxy to return the file body from somewhere else (say, some NFS path), and that's what uses sendfile (which means I not only avoid having to do async IO to files (which you can't do easily(?) in Linux yet), but I also avoid copies to/from userspace). this is what we'll need for photo hosting on LJ, so modem users don't tie up mod_perls while they download many-MB files.

I love Perl and how easy it is to bang stuff like the above out in record time (12 hours so far). Admittedly I've given this all a lot of thought, but it's still cool. During the process I compared parts to the C# equivalent and C# was incredibly verbose. I don't think I can productively use a language without built-in complex datatypes, or at least type inference. (Although a few times today I got bitten by misnamed hash keys as class attributes.... :-/ I used accessor (getter/setter) functions between classes, but raw hash keys within a class implementation....)

sendfile documentation sucks?

$ man 2 sendfile (Debian; manpages-dev 1.66-1)

Things to note:

-- it doesn't say offset can be NULL
-- it says "Sendfile does not modify the current file pointer of in_fd, but does for out_fd."

Well, turns out offset can be NULL (Linux 2.6.5, fs/read_write.c:638, sys_sendfile) and the in_fd is moved (*ppos = ... ; Linux 2.6.5, mm/filemap.c:725).

So on one hand I'm disappointed in crappy docs, but at least the source is available. *shrug?*