Log in

A proposal: email to URL mapping - brad's life [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

A proposal: email to URL mapping [Feb. 3rd, 2008|01:43 pm]
Brad Fitzpatrick
[Tags|, ]

People have different identifiers, of different security, that they give out depending on how much they trust you. Examples might include:
  • Homepage URL (very public)
  • Email address (little bit more secret)
  • Mobile phone number (perhaps pretty secretive)
As has been shown with OpenID, XFN, etc... URLs are people too. You can do a lot of things with a URL: give out information, point to other identifiers, do Yadis service discovery on it (to find, say, an OpenID server, calendar server, friend/contact server, etc...)

It's also possible to do a <a href="mailto:me@example.com" rel="me"> to an email address, making a one-way claim that you own an email address. But how do you make a rel="me" back from the email address to a URL, completing the cycle?

Another problem people have been bringing up regularly is how to use an email address as an OpenID identifier. For this to work, you need to do service discovery on it to find out the O.

If you could map from email address to URL (going from a private identifier to a more public identifier), both problems are solved... the mapping from email to URL is the rel="me" link, and the pointed-to-URL can then be used for any URL-like purpose:
  • Being an OpenID identifier
  • hosting an hCard
  • Pointing to another Yadis service type (OAuth-protected friends/contact server)


How to map from an email to a URL?
I propose:

Given, say, bradfitz@my-email-service.com, you do Yadis capabilty discovery on my-email-service.com, looking in the resultant XRDS service document for a capability of type, say, "http://schemas.net/2008/email-to-url/", and the resultant endpoint which speaks that capability protocol. Here's an example document (retrieved via Yadis, which means sending HTTP Accept: header of right mime type and getting it immediately, or looking at link from <head>):

<?xml version="1.0" encoding="UTF-8"?>
<!-- Sample YADIS XRDS file -->

    <Service priority="0">


The 2008/email-to-url capability endpoint (email2url_mapper.cgi, in this example), then speaks this "protocol":
GET /email2url_mapper.cgi?email=bradfitz@my-email-service.com HTTP/1.1
Host: apis.my-email-service.com

HTTP/1.1 302 Found
Location: http://bradfitz.com/
That's about it.


Why the Yadis indirection?
That's what Yadis is for. Discovery capabilities of an endpoint. This is exactly how OpenID works. There are libraries for it. Yadis discovery is cached. In practice, this step won't cost.

Privacy! Stealing my email addresses!
No, you start with the email address. You already have it. It's up to the user to determine if they want a public URL (presumably more public than their email address) attached to their email address.

Why not use $X?
What's X? I'm not aware of anything else. (Except for something I saw recently which was tied to OpenID and was pattern-based)

Why not pattern-based?
I want to tell, say, hotmail.com that my URL is http://bradfitz.com/, not MSN Spaces, or whatever hotmail.com might choose for a static username-to-URL mapping. It needs to be a dynamic lookup, not a published pattern.

Why not tie this to OpenID?
Layering violation.

The 302 could include an expires header.

But only the dorks would support this.
Maybe, but that's how it always starts. Maybe we could get some big email providers to do this too. Imagine a tab in your favorite Big3/Big4's email options which says:
Your public URL: [___________________________]
(This is the web URL that will be given out to anybody with your email address.)

The end

(Deleted comment)
(Deleted comment)
[User Picture]From: mart
2008-02-03 11:32 pm (UTC)

Is this like "Google employees with a good pagerank shouldn't confuse the search engine by linking to spammy sites?" ;)

(Reply) (Parent) (Thread)
From: evan
2008-02-03 11:42 pm (UTC)
You know, you're totally right. He can link to whatever he wants.
(Reply) (Parent) (Thread)
[User Picture]From: avatraxiom
2008-02-03 10:47 pm (UTC)
This would be beautiful and would definitely be the solution I've been looking for, for email-based SSO across all websites. :-) OpenID is great, but it's not currently so good if you absolutely require an email address in order to interact with a system (because that system is either super-tied to the concept of email, or it needs to send you lots of emails).

I'd definitely want to implement this in Bugzilla ASAP after it was standardized. I think this could finally eliminate the need for any local password storage in systems other than email systems, if it spread far enough.

My only thought on it is that from a technical standpoint, not all mail servers have a web server on the same domain, though they all have DNS. If there was some convenient way to make Yadis discovery happen in DNS instead of HTTP, that would probably make life easier for administrators, though that problem might not matter enough to block adoption.

(Reply) (Thread)
[User Picture]From: brad
2008-02-04 04:27 am (UTC)
DNS is the technically correct solution, but I think people needing to modify their DNS would make it a non-starter.

I'm for pragmatism... Yadis is simple.
(Reply) (Parent) (Thread)
[User Picture]From: tomercohen.com
2008-02-03 11:03 pm (UTC)

No Spam

How can you make sure it is spammers-safe?
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 04:28 am (UTC)

Re: No Spam

I addressed that in my post.

The person querying it already has the email address. It doesn't give you email addresses.
(Reply) (Parent) (Thread)
[User Picture]From: pne
2008-02-04 08:03 am (UTC)

Re: No Spam

Though I suppose once you know an email address on a given domain, you could dictionary-attack the email-to-url mapper and see what it gives out.

It would seem to be fairly simple to make the mapper dictionary-attach resistent (e.g. teergrubing or other timeouts if too many requests occur in too short a period of time, especially if they're unsuccessful), but you'd have to think of it; for smtpd's, I think it's already common knowledge.
(Reply) (Parent) (Thread)
[User Picture]From: mart
2008-02-03 11:30 pm (UTC)

This seems like a similar proposal someone made which included this information in the DNS. The main differences are:

  • This uses HTTP rather than DNS NAPTR records, and is therefore more accessible from the get-go, though will likely encounter a lot more friction from people who fuss about this sort of thing. (e.g. why should an email provider have to have a website?)
  • You support programmatic mapping, rather than regex, as you mention in your post.

Now I just need to figure out whether I'm a person who fusses about this sort of thing or whether I'm a person who says "hell, this seems like it should probably work". I'm currently leaning towards the latter.

(Reply) (Thread)
From: evan
2008-02-03 11:42 pm (UTC)
Good luck on getting anyone to configure their DNS. ;)
(Reply) (Parent) (Thread)
From: jmason
2008-02-04 10:13 am (UTC)
is it intended that email administrators publish this stuff? because email guys are *much* more comfortable with twiddling DNS than with twiddling Apache, I can tell you.
(Reply) (Parent) (Thread)
[User Picture]From: andrewducker
2008-02-03 11:45 pm (UTC)
Seems like the obvious way to do it. You only have the domain name and email address to work with - so using the former to find a translation point to parse the latter is the limit of your ability. Anything simpler would tie you down far too much.
(Reply) (Thread)
From: ext_83303
2008-02-04 12:01 am (UTC)

obligatory spam/dumb(?) questions for faq

Would we see mass registration from email addresses that translate to a spamvertised url?

What party carries the load of resource utilization in the mapping process?
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 04:38 am (UTC)

Re: obligatory spam/dumb(?) questions for faq

The email domain's webserver hosts the Yadis service document, and that points to the mapper endpoint. The mapper will most likely be run by the same people as the mail provider.

Mass registration just to point to spamvertised URLs? Probably, because they'll try anything, but how's that hurt? You afraid that we just moved domain registration to mail providers and people will now use, say, Gmail as their DynDNS provider to their IP address of their home linux box? It won't work in a browser, so I doubt it. What's your concern?
(Reply) (Parent) (Thread)
[User Picture]From: dossy
2008-02-04 12:07 am (UTC)
This smells like a brutal case of overengineering.

Remember, way back in the early days of the web, where a username was mapped to webspace using "~username"? What's wrong with that?


dossy@panoptic.com -> http://panoptic.com/~dossy/

That URL can call out my OpenID server, etc. in the <head> metadata of the response. This eliminates extending DNS, requiring special server-side software, requiring dynamic requests to be fulfilled (the response for http://host.dom/~username can be a static document and/or cached response).

Am I missing something, here?

Edited at 2008-02-04 12:09 am (UTC)
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 04:29 am (UTC)
I addresses this in my post.

You want users to choose their URL. Why force mail providers to provide web hosting?
(Reply) (Parent) (Thread)
[User Picture]From: dossy
2008-02-04 04:59 am (UTC)
Do you know of a mail provider today that doesn't provide web hosting?

Even if they don't want to offer ad-hoc web hosting where the email account holder has access to the web content, if there's value-add to offer at least OpenID off the email address, setting up the service to map user@domain.com to http://domain.com/~user would be easy to do.

Even your scheme implies mail providers having to set up something that responds to HTTP requests. All I'm saying is to use http://domain.com/~user as the Well Known Location of the service.
(Reply) (Parent) (Thread)
[User Picture]From: andrewducker
2008-02-04 01:30 pm (UTC)
Do you know of a mail provider today that doesn't provide web hosting?

And certainly neither of those are going to provide hosting on their web domains...
(Reply) (Parent) (Thread)
[User Picture]From: mart
2008-02-04 07:04 pm (UTC)

Depending on the tech in use at the root of the site, it might well be easier to add a static Yadis reference to the home page which points at some ugly URL somewhere else in the URLspace than it is to make /~user dispatch correctly.

(Reply) (Parent) (Thread)
[User Picture]From: scosol
2008-02-04 06:25 am (UTC)
i'm with you-
DNS has been able to provide this for years-
the issue of non-usage is not one of "broken protocol"-
it's one of adoption and standardization, where the introduction of a new protocol doesn't change anything
(Reply) (Parent) (Thread)
[User Picture]From: avva
2008-02-04 12:11 am (UTC)
What's "Yadis capability discovery on my-email-service.com"? I read the Yadis PDF and its protocol starts with a URL. If what you mean is an HTTP request to my-email-service.com, path /, accept-header such and such, then it's a bad idea.

Generally, your proposal is still about the URL. People don't care about their own URLs. Most people don't even have any URL that they control and would like to use as their identity, but they do have email addresses. If you get big providers on board, you can make a fake URL out of an email address in a patterned way, but it's at most a hack that you don't want to show your users - they won't understand it or care about it.

Maybe instead of 302 to the "public URL", your endpoint could give me
the Yadis document with the OpenID details inside?
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 04:34 am (UTC)
I don't want this tied to OpenID.

I want a general way to get a URL from an email address. I don't want a patterned, ugly URL that users won't care about. I want it to return their blog, or myspace profile, etc. The one they do know about and care for and associate as theirs.

(Reply) (Parent) (Thread)
[User Picture]From: avva
2008-02-04 06:55 pm (UTC)
I don't want this tied to OpenID.

That's why I said return a Yadis doc, not an OpenID provider URL outright. Stick whatever you want in the Yadis. Let your Big3 mail provider give you a way to configure your custom stuff in the Yadis it spits out.

I want it to return their blog, or myspace profile, etc. The one they do know about and care for and associate as theirs.

But they don't associate it as their id-defining URL, or at least many of them don't. Your proposal is not working for two huge groups of users (maybe they're the majority of users when put together): those without a convenient URL to structure their identity around, and those with too many such URLs.

I don't have a URL. I have a Russian journal that I don't want to send random English-speaking people to, lest they get confused, and a rarely-updated English blog that I don't take good care of. I log in to facebook maybe once a month and don't use myspace. I sure wish I could ID myself with my handy primary email address though! But your solution isn't doing anything for me, it wants to take me away from my email to invent some URL.

Someone else has all that stuff - facebook, myspace, personal blog, professional blog - and will have a problem choosing one as the defining URL. Why should they? They typically have one email address though, why not let them use it for ID.

What about someone with no facebook, no myspace, no blog (or no blog they want to advertise), just want to leave authenticated comments at blogs?

Having to own a URL, or remembering that you own that particular URL as your main id, is just another thing to remember to take care of, just another nuisance. It's like another site that forces me to invent a username, I don't even use sites anymore that won't take email for my ID.
(Reply) (Parent) (Thread)
[User Picture]From: mart
2008-02-04 07:07 pm (UTC)


Why not make the endpoint be like an XRI resolver! Then you can do all that neato stuff that XRI does like brad@example.com(+contact) and brad@example.com(+blog) and have multiple exposed services per email address!

(I'm joking. Mostly. ;))

(Reply) (Parent) (Thread)
[User Picture]From: mrgatsby
2008-02-04 12:31 am (UTC)
Back in the old good days it was mere VRFY cmd for MX record.
Today it's:
vrfy bradfitz
252 Just try sending a mail and we'll see how it turns out ...

Damn you spammers!
(Reply) (Thread)
[User Picture]From: nothings
2008-02-04 02:51 am (UTC)
This maybe falls under "Privacy! Stealing my email addresses!", but this provides another mechanism for spammers to guess-and-test existence of email addresses, which means implementers will probably want/need to put some mechanisms in to avoid bulk banging on it from the getgo.
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 04:32 am (UTC)
True, but half the SMTP servers out there already reply in the negative to a bogus RCPT TO, so this is no different.

Also, in the use case I'm most interested in, the email address is already on the internet with a rel="me" to it, so the email's already public... you just want to complete the verification loop.

Maybe I'm spoiled by gmail, but I don't get any spam... not really worried about that.

In any case, it's up to users to choose to publish a URL for an email.
(Reply) (Parent) (Thread)
From: ext_83321
2008-02-04 05:00 am (UTC)

Yadis vs NAPTR

Given current tools, I lean towards a (well specified!) Yadis, if we can get one. And it works for a provider with 30m email accounts but naptr would probably fall over.

What about multiple urls per email?
(Reply) (Thread)
[User Picture]From: ch
2008-02-04 06:05 am (UTC)
do the look up on SHA1($email_addr).

solves the issue of giving up $email_addr.
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 07:05 am (UTC)
I assume your email provider already has your email address.
(Reply) (Parent) (Thread)
From: ext_4309
2008-02-04 06:55 am (UTC)

What about the other way round

You propose to map all users' e-mail addresses to the same Yadis file, and then you "personalize", to each user, the result by going through the CGI indirection.

What about the other way around: you allow a potentially different Yadis file for every user. That way, each user could express user-specific information in that file, instead of having to be put into the same straightjacket as all other million users. Also, it would allow to return multiple values, which in turn could be annotated with things like priority or whatever annotations make sense.(E.g. multiple "me" relationships with a qualifier what kind of "me" it is)

foo@example.com -> Yadis discovery on http://example.com/~foo -> contains the actual endpoint, not the CGI script. Of course, sites could still decide to map all accounts to the same Yadis file, but architecturally it wouldn't be a requirement.
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 07:06 am (UTC)

Re: What about the other way round

You can do Yadis discovery (or hCard parsing, etc) on the resultant URL.

So Yadis on gmail.com returns bradfitz.com. Yadis on that returns my primary and secondary openid servers, optionally using delegation.
(Reply) (Parent) (Thread)
[User Picture]From: avatraxiom
2008-02-04 06:56 am (UTC)
I think it might be important in whatever standard is written to explicitly state that non-existent or unregistered email addresses do not return a "success" message, which would make it a lot easier to use the system as a simple email verification system without having to go any further.

Also, some explicit description of what the URL is (possibly by some other extension header) would be extremely handy, so that we know whether or not the URL can be used as a further Yadis service. I suppose we could always query the URL with Yadis discovery, but this is a lot of web requests already, from the viewpoint of an authenticating application.

(Reply) (Thread)
From: ext_83337
2008-02-04 07:45 am (UTC)

You nailed it

After all the proposals for email address mapping, this is the one that finally does it right.
(Reply) (Thread)
[User Picture]From: hacker.klever.net
2008-02-04 09:42 am (UTC)
I wonder what about those who are paranoid enough to want strictly SSL (not exactly my own concern, but I can foresee that)? That is, not a single http request? With OpenID you at least have this possibility to enter https explicitly.
(Reply) (Thread)
[User Picture]From: brad
2008-02-04 06:20 pm (UTC)
You can do the Yadis discovery over HTTPS.
(Reply) (Parent) (Thread)
[User Picture]From: mart
2008-02-04 07:14 pm (UTC)

I think the concern here was that you've "hard-coded" the mapping between email address and initial discovery URL to point at http://. Given blah@example.com, you're going to do Yadis discovery on http://example.com/, not https://example.com/. The former could redirect to the latter, but then you're still vulnerable to attacks on the http URL. (OpenID gets around this because you use the redirect result as the identifier; that's not true here.)

Perhaps the half-way house would be to default to the above but allow it to be overridden somehow. I hear they have this thing called the Domain Name System that allows you to publish this sort of thing... (lawlz!)

(Reply) (Parent) (Thread)
From: jsmarr
2008-02-04 04:19 pm (UTC)
I love it. Let's get some code running and get it built into the SG API (or at least the nodemapper lib) so we can query on e-mail addresses as identifiers and find additional sites. ;) Know any large email providers who enjoy being thought-leaders and want to be early adopters here? Should be a trivial engineering task, and like you said the adding of a URL can be opt-in. js
(Reply) (Thread)
[User Picture]From: andrewducker
2009-11-22 11:36 am (UTC)
Did anything ever come of this?
(Reply) (Thread)
[User Picture]From: brad
2009-11-22 03:54 pm (UTC)
(Reply) (Parent) (Thread)