Brad Fitzpatrick (brad) wrote,
Brad Fitzpatrick


This afternoon/evening I learned LWP inside and out, in order to make LWPx::ParanoidAgent, a subclass of LWP::UserAgent that protects you from evil. In particular:

-- won't connect to private, loopback, or multicast IPs. including on redirects

-- configurable blacklist of hostnames or hostname regexps

-- avoids a malicious/accidental tarpitting webserver, sending 1 byte per 9 seconds, to avoid LWP::UserAgent's timeout parameter if you set it at, say, 10 seconds. (my ParanoidAgent has a global timeout, including across all redirects)

I had to end up forking LWP::Protocol::http since there's pretty much just one huge function in it, and I had to change it. So LWPx::ParanoidAgent is subclassed, and it allows only two schemes, http and https, which map to protocols LWPx::Protocol::http_timelimit and https_timelimit. (https code is like 20 lines, just calling the http code after a different socket is made) Proxy support is explicity removed, telling you to do your paranoia on your proxy if you want to use a proxy.

This will be released on CPAN at about the same time as OpenID::Consumer, which lets you specify your own LWP::UserAgent subclass. Currently LiveJournal uses a version of this called, but it has-a LWP::UserAgent in it, and it's not a sub-class, so it's annoying to use since it doesn't always work everywhere. And it uses alarm(), which just always sucks, and isn't portable. ParanoidAgent just tracks the time remaining and sets the select timeouts accordingly.

I had to end up writing a little httpd (just run from xinetd to make it easy) that lets me specify redirects and timeouts from the URL.

The test suite was fun:

use strict;
use LWPx::ParanoidAgent;
use Time::HiRes qw(time);
use Test::More 'no_plan';

my ($t1, $td);
my $delta = sub { printf " %.03f secs\n", $td; };

my $ua = LWPx::ParanoidAgent->new;
ok((ref $ua) =~ /LWPx::ParanoidAgent/);


my $res;

# black-listed via blocked_hosts
$res = $ua->get("http://brad.lj/");
print $res->status_line, "\n";
ok(! $res->is_success);

# checking that port isn't affected
$res = $ua->get("http://brad.lj:80/");
print $res->status_line, "\n";
ok(! $res->is_success);

# this domain is okay. isn't blocked
$res = $ua->get("");
print $res->status_line, "\n";
ok( $res->is_success);

# internal. bad. blocked by default by module.
$res = $ua->get("");
print $res->status_line, "\n";
ok(! $res->is_success);

# okay
$res = $ua->get("");
print $res->status_line, "\n";
ok( $res->is_success);

# localhost is blocked, case insensitive
$res = $ua->get("http://LOCALhost/temp/");
print $res->status_line, "\n";
ok(! $res->is_success);

# redirecting to invalid host
$res = $ua->get("$HELPER_SERVER/redir/");
print $res->status_line, "\n";
ok(! $res->is_success);

# redirect with tarpitting
print "4 second redirect tarpit (tolerance 2)...\n";
$res = $ua->get("$HELPER_SERVER/redir-4/");
ok(! $res->is_success);

# lots of slow redirects adding up to a lot of time
print "Three 1-second redirect tarpits (tolerance 2)...\n";
$t1 = time();
$res = $ua->get("$HELPER_SERVER/redir-1/$HELPER_SERVER/redir-1/$HELPER_SERVER/redir-1/");
$td = time() - $t1;
ok($td < 2.5);
ok(! $res->is_success);

# redirecting a bunch and getting the final good host
$res = $ua->get("$HELPER_SERVER/redir/$HELPER_SERVER/redir/$HELPER_SERVER/redir/");
ok( $res->is_success && $res->request->uri->host eq "");

# dying in a tarpit
print "5 second tarpit (tolerance 2)...\n";
$res = $ua->get("$HELPER_SERVER/1.5");
ok(! $res->is_success);

# making it out of a tarpit.
print "3 second tarpit (tolerance 4)...\n";
$res = $ua->get("$HELPER_SERVER/1.3");
ok( $res->is_success);
Tags: openid, perl, tech

  • Happy Birthday!

    Happy 20th Birthday, LiveJournal! 🐐🎂🎉

  • hi

    Posting from the iPhone app. Maybe I'm unblocked now.

  • Why, hello...

    Long time no see. How's my baby doing?

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.