Log in

No account? Create an account
Perl on App Engine - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Perl on App Engine [Jul. 22nd, 2008|08:38 pm]
Brad Fitzpatrick
[Tags|, , , ]

Fellow Perl hackers,

I'm happy to announce that the Google App Engine team has given me permission to talk about a 20% project inside Google to to add Perl support to App Engine.  To be clear:  I'm not a member of the App Engine team and the App Engine team is not promising to add Perl support.  They're just saying that I (along with other Perl hackers here at Google) are now allowed to work on this 20% project of ours out in the open where other Perl hackers can help us out, should you be so inclined.

As background, I've been writing Perl code for almost 15 years now and quite fond of the language.  (I'm "bradfitz" on CPAN.)  Here at Google, though, it's not one of our big languages so I don't get to write as much Perl as I used to.  I'd still like to run my personal web apps on App Engine, though, and I'd like to write them in Perl.  And I'm definitely not alone, looking at how many people have starred the wishlist bug.  Some of you have already started talking about it.  We'd like to join the discussion, and start hacking out in the public.

In the process we can build the start of an open source App Engine server clone that's suitable for many purposes:  initially just for regression testing & local development (like the "dev_appserver" that comes with the App Engine Python SDK), but perhaps in the future (once Hypertable/Hbase/etc are ready) a full stack to give to ISPs to let them run App Engine apps on their own.

Before I get into my proposed roadmap, let me describe what's publicly known about the App Engine architecture.  In a nutshell, it looks like this:

The App runs in a multi-layer hardened environment, one layer of which will need to be a hardened Perl interpreter.

Basically, we need a hardened Perl runtime which can:
  • open & read files
  • NOT write files
  • NOT open sockets
  • NOT fork
  • NOT do any other system functionality that's not strictly needed for a web app
Basically we need a Perl interpreter that's very tame and isn't allowed to do anything other than read web requests and write out responses.  Any privileged operations (like Datastore access, fetching URLs, etc) need to be done via a trusted XS Perl module (the "apiproxy") that takes a service request parameter and returns a service response.  The request and response are both encoded as Protocol Buffers, which were recently open sourced by Google.

Perl on App Engine then would involve the following steps (in no particular order):
  • Hardened Perl Interpreter:  basically, we'll be statically linking in a hardened, customized libperl to a C++ application, disabling all Perl dynamic loading.  Only vetted, security-audited XS modules will be allowed.  Only safe Perl opcodes will be allowed.  (No sockets, no ioctl, no fork, etc, etc.)  To get a preview for what this'll feel like restriction-wise, check out the newly written Sys::Protect which Artur and I wrote this evening and will be continuing to develop for people's dev environments (not production).
  • Protocol Buffers for Perl:  we need support for Protocol Buffers for Perl.  I've started on this project internally and will open source the code soon, once I have a few free minutes.
  • Server:  we need to write an App Engine server for testing, local development, and potentially production deployment.  (Replace Bigtable with MySQL, Hypertable, Hbase, Couch DB, etc.)
  • Libraries:  Perl client libraries for Datastore, URLFetch, etc services.  Including docs.
Not included is the Google-internal side of things, gluing the hardened Perl interpreter into the GAE world.  That needs to be done by a Googler and not open source. 

If you'd like to discuss this and/or help out, join the perl-appengine mailing list.  We'll be submitting code to the appengine-perl project on Google Code hosting.  For more information about this, see the Perl-on-AppEngine FAQ.

Brad & the other Perl Googlers

[User Picture]From: loganb
2008-07-23 04:13 am (UTC)
For many years PostgreSQL has included a hardened version of Perl (Pl/Perl). I have no idea how it works, but it may be worth looking at as a starting point for securing the interpreter.
(Reply) (Thread)
[User Picture]From: loic
2008-07-23 05:34 am (UTC)
While I actively dislike perl this seems like a really fun project.

Does your hardened perl interpreter need to provide a way for the hosting environment to limit the time a script can execute for, or provide a way to kill running scripts? Or can that just be done with unix process control syscalls (kill, wait, etc).
(Reply) (Thread)
[User Picture]From: brad
2008-07-23 07:30 am (UTC)
App Engine already provides that sort of stuff, but we'll probably provide some really basic implementation of it in our dev appserver, if it's easy.
(Reply) (Parent) (Thread)
From: (Anonymous)
2008-07-24 12:58 am (UTC)

Actively disliking

When you say you actively dislike Perl, do you mean you actually sat around a lot, never programming in Perl, but bashing it on the intertubes?
(Reply) (Parent) (Thread)
[User Picture]From: volk007
2008-07-23 06:36 am (UTC)
And I, for one, hoped for Ruby.
(Reply) (Thread)
[User Picture]From: brad
2008-07-23 07:29 am (UTC)
They're not mutually exclusive.

Or you hoped that I would work on Ruby? That'd be a bad wish, because I don't know Ruby internals.
(Reply) (Parent) (Thread)
From: (Anonymous)
2008-07-23 12:00 pm (UTC)
Nono i think he meant Ruby powered Apps, like perl now. (on the wishlist)
Not you working on them! :)

It is clear that python is the winner for google.

For me, however, I have been writing since 5 years in Ruby and it is my favourite. My level of C knowledge though is very limited (which is why I tell EEEEEEVERY programmer to master C first. Ruby goes so easy on your brain, but C is significantly harder than Ruby. It requires more discipline to make things work)
(Reply) (Parent) (Thread)
[User Picture]From: volk007
2008-07-24 01:47 am (UTC)
Well, between rewriting my existing (Asp.Net) app to Ruby, Python and Perl I personally would first choose Ruby, then Python, but not Perl. Of course this is my personal unscientific-not-clearly-motivated choice.

With current state of things I'm going to rewrite it to Python.

It's exciting to see such announcement for Perl, since it shows that there is hope for other languages. :)
(Reply) (Parent) (Thread)
From: ext_60419
2008-07-23 06:55 am (UTC)


The Safe.pm module is rarely mentioned at all, but it should do most of what you need already.

But probably a more drastic (aka, delete offending parts) approach is preferred here.
(Reply) (Thread)
[User Picture]From: brad
2008-07-23 07:28 am (UTC)

Re: Safe.pm

Safe.pm isn't actually all that safe, from what I hear. Historically at least it hasn't.

The more drastic approach (removing the implementation of all the bad opcodes from libperl itself) is what we'll do for a real implementation. The Sys::Protect module that Artur and I released tonight could be done with Safe.pm, but we have more ambitions with it than Safe.pm does, but in the end Sys::Protect is just for testing on developers' local machines anyway, so Safe.pm could've worked there, were it not for other sandbox restrictions we need to do in the XS code.
(Reply) (Parent) (Thread)
(Deleted comment)
[User Picture]From: brad
2008-07-23 07:52 am (UTC)
You're right. Google's big languages are C++, Java, Python, and JavaScript.
(Reply) (Parent) (Thread)
[User Picture]From: ghewgill
2008-07-23 08:22 am (UTC)
That's very cool.

Another approach might be to offer a hardened JVM or CLR runtime. That way, you can just drop compiled code from whatever language onto the appengine and make it go. All the hardening of the libraries and VM would only have to be done once.
(Reply) (Thread)
[User Picture]From: d_ilmari
2008-07-23 03:07 pm (UTC)
Or a hardened Parrot runtime! ;-)
(Reply) (Parent) (Thread)
[User Picture]From: publius_ovidius
2008-07-23 03:12 pm (UTC)
You're correct, but it doesn't get us Perl :)

Once Parrot is ready, that will open up a whole host of new approaches, but we're not quite there yet.
(Reply) (Parent) (Thread)
[User Picture]From: crucially
2008-07-23 04:01 pm (UTC)
Not quite there yet?

Are we lost in the Potemkin villages?
(Reply) (Parent) (Thread)
From: (Anonymous)
2008-07-23 10:16 am (UTC)

dia clouds

I see you're using those "funky" dia clouds :)

I created the following in inkscape, which dia
can open, and can then be copied and pasted as required:
(Reply) (Thread)
From: (Anonymous)
2008-07-23 11:09 am (UTC)

Very nice!

Of cause I like the idea of a hardened VM too. How about parrot? That would give Perl 6 a real kick start.
(Reply) (Thread)
From: (Anonymous)
2008-07-23 11:36 am (UTC)


"Only vetted, security-audited XS modules will be allowed."

Does this mean all XS modules need to be vetted - or all modules must be both XS and vetted?

I'm worried if it's the second; what would be the point of perl without CPAN?
(Reply) (Thread)
[User Picture]From: brad
2008-07-23 04:34 pm (UTC)


Any pure-Perl would be allowed. So most of CPAN is still available... just not XS code on CPAN.
(Reply) (Parent) (Thread)
[User Picture]From: dossy
2008-07-23 03:03 pm (UTC)
Are there any Tcl enthusiasts at Google who would want to add Tcl support to the App Engine list? Tcl already has a "safe interp" functionality implemented - it would be a piece of cake to add Tcl support to App Engine.

Sigh. The greatest scripting language gets left by the wayside, again.
(Reply) (Thread)
[User Picture]From: publius_ovidius
2008-07-23 03:12 pm (UTC)
I'm very happy to hear about this. Thanks for giving it a shot!
(Reply) (Thread)
From: ext_112136
2008-07-23 04:02 pm (UTC)


I am so happy to hear this. I love Perl and I can't stand Python. The language itself is a joke. What kind of programming langauge relies on tabbing. I can't believe you took it seriously.
(Reply) (Thread)
From: (Anonymous)
2008-07-23 06:57 pm (UTC)


Taking it seriously generally involves paying attention to things a little more significant than white space. Apparently you can't look past that minor detail.

FWIW I dislike Python and generally use Perl, but claiming that you can't stand it and that it is a joke and then not citing anything significant is just trolling.
(Reply) (Parent) (Thread)
From: ext_112331
2008-07-24 04:15 pm (UTC)


I resisted Python for a long time for that very reason (whitespace). Truth be told, once you spend some time in the language, and get your vim settings right, it's not that big a deal. Python is far from being "a joke".

BTW, don't use tabs for formatting in Perl or Python--it makes baby Buddha cry.
(Reply) (Parent) (Thread)
[User Picture]From: xb95
2008-07-23 06:12 pm (UTC)
This could be rad.
(Reply) (Thread)
[User Picture]From: schernyshev
2008-07-24 02:07 pm (UTC)
Just recently I was thinking that the reason for many of my app ideas not developed is that I only feel in the right mood in my favorite Perl and that I don't want to care for infrastructure anymore - too tired of it at work.

Google Apps was part of the answer and I even considered to start torturing myself into Python on a daily basis so I don't get that tired of it, but it didn't work for about 5-6 times so Perl support might do that for me ;)

I wonder how much of LWP you're planning to bring over - will it be possible to have granular HTTP support of modules like http://search.cpan.org/~sergeyche/LWP-ConnCache-Resolving/ (I wrote it to make single connection to LiveJournal servers when fetching FOAF files so there is only one connection to Perlbal).
(Reply) (Thread)
[User Picture]From: markemer
2008-07-25 05:51 pm (UTC)

I agree wholeheartedly

When ever I get a yen to do something cool, I always seem to do it in Perl. I just can't get in the right mood using python. I have tried quite a few times to get into it, but I just can't. I for one am willing to help as much as I can manage to find time.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2008-07-27 12:55 am (UTC)
LWP's so common that I imagine I'll provide a fake module which quacks like a duck, so must be LWP. (but actually does the URLFetch service transparently)
(Reply) (Parent) (Thread)
From: atl
2008-07-25 10:46 am (UTC)
Although this writeup has a lot of negative commentary, the hardened Perl interpreter does sound similar to what's going on at the BBC/Siemens. With Google's clout, sounds like an interesting place to reach out for expertise.
(Reply) (Thread)