A few thoughts about this approach. The first is that if you return a relative URL in a redirect many servers in many situations will do a local redirect and fetch the resource themselves. I've not tried with Apache/mod_proxy, but at very least it'd be worth investigating; if it doesn't do it already it may be a quick point to hack something in.

Secondly when I last needed to do something like this (Apache/mod_perl site, serve large files to slow (modem) clients -- in this case mp3 files on a music sale site (I don't know if they still use the same technique)), what I did was have the Apache/mod_perl site generate a URL for the file on a thin HTTP server which:

• Specified what file
• Included some salt (eg, random text)
• Included a crypto checksum of the file to retrieve, the salt, a shared secret, and some time/date stuff ,etc

Then on the thin HTTP server there was a tiny extension (CGI in this case, but you could probably hack it into the source) which validated the URL was a "good" redirect (ie, recalculated the crypto checksum), and if it was spooled the file out.

That kept the load on the Apache/mod_perl site down to creating the validation token, and allowed the thin HTTP server to validate access with a simple calculation (eg, no database access required).

IIRC we did client side redirects for this, but because there was a thin access control done on the thin HTTP server, it didn't matter -- the redirect URL was only good for the time that we allowed (a few minutes; you could probably make it shorter). Bypassing the access controls would require brute forcing the authentication calculation algorithm being used, which is one reason for including the salt to make it harder.

If I were doing it again now and really cared about the content getting out I'd probably consider using public key crypto instead of a shared secret; four years ago (when I did this) it wasn't as widely available/easily usable, and it didn't seem worth the extra CPU time.

Ewen

One more thing I should have mentioned. If you send the right type of redirect code, even a client side redirect won't result in a changed URL displayed by the client. A 302 (temporary) or 303 (get content here) redirect would probably be suitable for this purpose. (If you're embedding the large files in a page it probably doesn't even matter.) You'll want a non-cached redirect in this situation anyway, as you'll need to regenerate the authentication URL after a few minutes.

Also where I say "the checksum of the file to retrieve,..." I of course mean of the filename (combined with the other stuff). The idea being that it should be trivial to validate, but difficult to generate without knowing some secret information. (Kind of like a NP-hard problem.)

Ewen

Is there any way you could replace the heavy-weight mod_perl+Apache that you're using for auth[z], with a simple C module? Then have separate frontends just for doing this simple task, and leave full mod_perl ones for more important tasks?

Also I don't get what you mean about ProxyPassReverse not being good enough, we're using it to front some JSP servers without native Apache connectors, and it doesn't disclose the URLs - infact the JSP servers do not have public routable addresses.

It'd be too much work to do the authz in C. Very perl library dependent.

I know ProxyPass[Reverse] work, but I don't think you understood my post. I want the backend to tell mod_proxy to go to another host entirely, without exposing that host or HTTP redirect to the end client.

Just invent your own wacky HTTP response code (say, 309 Wacky LJ Redirect) and then hack mod_proxy! You can probably also make it reject the “magic” URLs unless they came back in a Wacky LJ Redirect, thus avoiding the chance that anyone will be able to bypass authentication by accidental exposing of the URL.

I realise “just hack mod_proxy” is not as easy as saying it, but I can't imagine it'd be incredibly hard since it'd just involve creating a subrequest just like any Apache internal redirect, I assume.

Would it be practical to have the proxy do any authentication? I was thinking the client could be given some kind of cookie, and when it tries to grab files from the stripped down fast servers, the proxy between it and them could do some cryptographic challenge stuff with the client and ask mod_perl again if that cookie was allowed to get the path it wanted. Or maybe it could take care of authenticating the request itself if it was a smart enough proxy.

Also, squid does rewrites as a reverse proxy.

Squid doesn't seem to do what I want. Those docs you linked are the same as Apache's ProxyPassReverse.

Cryptographic hashes in ugly, temporary URLs aren't good. Kills client-side caching.