Brad Fitzpatrick (brad) wrote,
Brad Fitzpatrick

Perlbal and persistent connections

I'm adding persistent connection support to Perlbal, both to clients and to backend nodes.

Some fun problems on both sides.

The client part is pretty easy. The main rule is that that HTTP/1.0 clients don't understand chunked encoding, so if a 1.0 client hits a proxy (Perlbal) which does a 1.1 request to the backend, the backend might reply with chunked encoding and the client would get confused.

There are two choices here:

1) When proxy gets a 1.0 request, send a 1.0 request to the backend, so the backend replies without chunked encoding (And thus a Connection: close).

2) Make the proxy convert the unchunk the backend's reply.

Number 2 would work, but would be more CPU heavy.

The problem with #1 is that I think I might have to make a new HTTP/1.0 connection to the backend. Ideally I'd be able to re-use an idle HTTP/1.1 connection and send a 1.0 request to it w/ an explicit keep-alive. If I'm lucky, the backend replies with with an explcit Content-Length, honors the HTTP/1.0 Keep-Alive token, and my connection is still good, and I can keep pushing 1.1 or 1.0 requests at it. If i'm not lucky, the backend sends a response without a content-length header (and without chunked-encoding, since it's 1.0) and closes the connection at the end, at which case I lose a transport and just reconnect.

I just need to investigate/read whether that's supported (mixing HTTP/1.0 requests in with HTTP/1.1 requests over the same connection). I'm guessing it's not in the spec, but the way Apache works, I bet it works.

The other fun problem for proxies is that ideally you want a bunch of indepedent connections to clients and connections to backends. Clients aren't tied to backends. So you get a stream of requests over the client connections, and you mix 'em up however and send them to available backend connections. But backends usually assume all the requests in one connection are from the same client.

This is why the BIG-IP's default behavior is to only aggregate connections to backends from clients with the same IP address, so it doesn't confuse the backend's logs. The BIG-IP actually fakes the IP headers/routing to make it look like they're coming from the clients directly, so this is more of a problem. Perlbal doesn't do that, instead just scrubbing/inserting an X-Forwarded-For header to let the backend know to fix up its idea of who the client is. So for Perlbal, nothing much needs to change, since the backends must already be prepared to deal with adjusting their backend.

So here's the fun part:

I want persistent connections on the BIG-IP, on Perlbal, and on backends. And I want the BIG-IP's connection aggregation (to Perlbal) happening all the time, regardless of source IP.

So on the BIG-IP I do "b global web aggregate all" or whatever, but then Perlbal gets the wrong IPs of users, and passes those wrong IPs to mod_perl.

So instead we make the BIG-IP scrub/insert a "X-BIGIP-Client:" header. We then configure Perlbal to trust/recognize the "X-BIGIP-Client" header as the real client header, and use that instead of the socket source address when it tells ITS backend the client (in the "X-Forwarded-For") header.

Anyway, I'm not done yet, but I have a plan, so I'm hyper. Unfortunately none of this is important for what I need to do in the next few days, so I should probably switch to another project.

Obligatory preemptive answer to common question (at least question I get over email)
Why two layers of proxies!? Go read prior posts. In short: Perlbal is either a webserver, reverse-proxy, or hybrid. (in reverse proxy mode, backends can tell Perlbal to kick into webserver mode). Perlbal buffers requests too, spoon-feeding modem users. The BIG-IP doesn't. The BIG-IP can do a lot of stuff Perlbal can't. Together they are awesome.

Update, 7:36pm: I'm on crack. I need to dechunk it anyway to find the end of the stream.

  • Ukraine

    Nobody reads my LiveJournal anymore, but thank you to everybody in Russia protesting Putin's insane war against Ukraine. (I know it's risky…

  • Happy Birthday!

    Happy 20th Birthday, LiveJournal! 🐐🎂🎉

  • hi

    Posting from the iPhone app. Maybe I'm unblocked now.

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.