Perl on App Engine

Fellow Perl hackers,

I'm happy to announce that the Google App Engine team has given me permission to talk about a 20% project inside Google to to add Perl support to App Engine.  To be clear:  I'm not a member of the App Engine team and the App Engine team is not promising to add Perl support.  They're just saying that I (along with other Perl hackers here at Google) are now allowed to work on this 20% project of ours out in the open where other Perl hackers can help us out, should you be so inclined.

As background, I've been writing Perl code for almost 15 years now and quite fond of the language.  (I'm "bradfitz" on CPAN.)  Here at Google, though, it's not one of our big languages so I don't get to write as much Perl as I used to.  I'd still like to run my personal web apps on App Engine, though, and I'd like to write them in Perl.  And I'm definitely not alone, looking at how many people have starred the wishlist bug.  Some of you have already started talking about it.  We'd like to join the discussion, and start hacking out in the public.

In the process we can build the start of an open source App Engine server clone that's suitable for many purposes:  initially just for regression testing & local development (like the "dev_appserver" that comes with the App Engine Python SDK), but perhaps in the future (once Hypertable/Hbase/etc are ready) a full stack to give to ISPs to let them run App Engine apps on their own.

Before I get into my proposed roadmap, let me describe what's publicly known about the App Engine architecture.  In a nutshell, it looks like this:


The App runs in a multi-layer hardened environment, one layer of which will need to be a hardened Perl interpreter.

Basically, we need a hardened Perl runtime which can:

• open & read files
• NOT write files
• NOT open sockets
• NOT fork
• NOT do any other system functionality that's not strictly needed for a web app

Basically we need a Perl interpreter that's very tame and isn't allowed to do anything other than read web requests and write out responses.  Any privileged operations (like Datastore access, fetching URLs, etc) need to be done via a trusted XS Perl module (the "apiproxy") that takes a service request parameter and returns a service response.  The request and response are both encoded as Protocol Buffers, which were recently open sourced by Google.

Perl on App Engine then would involve the following steps (in no particular order):

• Hardened Perl Interpreter:  basically, we'll be statically linking in a hardened, customized libperl to a C++ application, disabling all Perl dynamic loading.  Only vetted, security-audited XS modules will be allowed.  Only safe Perl opcodes will be allowed.  (No sockets, no ioctl, no fork, etc, etc.)  To get a preview for what this'll feel like restriction-wise, check out the newly written Sys::Protect which Artur and I wrote this evening and will be continuing to develop for people's dev environments (not production).
• Protocol Buffers for Perl:  we need support for Protocol Buffers for Perl.  I've started on this project internally and will open source the code soon, once I have a few free minutes.
• Server:  we need to write an App Engine server for testing, local development, and potentially production deployment.  (Replace Bigtable with MySQL, Hypertable, Hbase, Couch DB, etc.)
• Libraries:  Perl client libraries for Datastore, URLFetch, etc services.  Including docs.

Not included is the Google-internal side of things, gluing the hardened Perl interpreter into the GAE world.  That needs to be done by a Googler and not open source. 

If you'd like to discuss this and/or help out, join the perl-appengine mailing list.  We'll be submitting code to the appengine-perl project on Google Code hosting.  For more information about this, see the Perl-on-AppEngine FAQ.

Brad & the other Perl Googlers