When you start to build something new ...

When you start to build something new, think about the what could be, the what may be, and the what will be. Don't settle, don't give up, don't get stuck in a box built by other people's misguided interaction paradigms. The internet is open and free, and that means there are no rules.

-- Dustin Curtis, The Manifesto

reMAP Meetup at SXSW?

Joshua Baer (founder of email startup OtherInbox) and me are thinking about putting together a meetup for people interested in replacing IMAP.

We would meet sometime around the SXSW Interactive festival in Austin, Texas (March 12-16). Exact time is still unclear, but Joshua has kindly offered up his company's offices, which are just a few blocks from the event.

Who's interested?

The IMAP Compatibility Matrix

There's a great discussion going on in the comments of my post about replacing IMAP, and also on YCombinator's Hacker News.

One frequent point made about my proposal is that IMAP already has a number of extensions that do things I suggested, such as the THREADS, SEARCH, and IDLE commands for conversations, search, and long polling ("push").

Commenter Bron (who works on a popular IMAP server) writes:

Also, IMAP plus a random mix of extentions becomes a HUGE compatibility matrix, and you have to offer fallback from all the nice stuff to really shitty workarounds just in case the server doesn't support a feature that makes your life bearable.

So - I'd love to see a decent competitor to IMAP as a protocol. If nothing else because it would reduce the compatibility matrix. Alternatively, I'd be happy for an IMAP5 to be defined which included all the good extentions - so anything which advertised support for IMAP5 had to support them.

Defining an IMAP5 standard might be a good alternative than something like reMAP, and adoption might be much faster. However, a potential IMAP5 would still remain a protocol inaccessible to many developers so it might not unleash the storm of creativity I'm hoping for.

Keep those comments coming.

How to Replace IMAP

IMAP is a complex and difficult protocol. Its original design, RFC 1730, dates back to 1994. Here's a sample IMAP session that shows you all of IMAP's quirks: It's a stateful protocol where you login and select a mailbox to operate on, it has an unconventional parenthesis-heavy representation for structured data, and fetching messages is a multi-step operation.

Design Outline

One of these days, I'll write up a spec for the mail access protocol of my dreams. I'm planning to call it the "reimagined Mail Access Protocol" (short: reMAP) which is handy because reMail already owns the domain name :-)

I would aim for a RESTful design with the following properties:

1. All communication over HTTP / HTTPS: Pure TCP connections are great, but for transferring large amounts of email, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved. There is no reason to solve them again.


2. Stateless: There's no reason to introduce state like IMAP does with its selected mailboxes. All you need is a HTTP session cookie used for authentication purposes. Session cookie also allow for things like OAuth. OAuth would let third parties get your permission to access your email without having to give them your username and password.


3. JSON and UTF-8: All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. UTF-8 would be the only encoding allowed, since it is able to represent any character in the Unicode standard.


4. Conversations as first-order objects: Gmail, the iPhone SMS app, and Facebook's messaging system have shown the value of viewing messages not individually but in the context of conversations. In reMAP, the server would be responsible for grouping together messages. While you could still access individual emails, the first-order unit of data would be a conversation.


5. Labels, not folders: Labels are much for flexible than folders. Each conversation should have multiple labels, and the labels would be included when you request the message, rather than having to scan all folders for the message via IMAP.
   
6. Stable and unique IDs: IMAP has a UID for each message, but it changes the moment you move the message into a different folder. An IMAP server can also declare all UIDs to be invalid at any moment throughout the session. No more! reMAP would have stable and unique IDs for all conversations, emails, and attachments.


7. The beginning of the end for MIME: Yes, you could still get the MIME representation of each message that is sent. But MIME is a messy and complex beast. Instead of requesting the MIME-encoded message parts, you could just ask the server to give you the message as represented in plain text or HTML. Attachments can be downloaded in separate HTTP calls.


8. Push built in: The two prevailing methods for implementing push email are the IMAP IDLE command (not widely available in IMAP servers) and Microsoft's ActiveSync, which requires developers to purchase a license from Microsoft. In reMap, clients could just call an HTTP endpoint on the server which returns as soon as new messages are arrive.


9. Full-text index on the server: reMAP servers would need to maintain a full-text index of the contents of all messages. There's no reason clients should be required to download and index everything in order to do an exhaustive full-text search of your email.

Enormous Potential

I believe that one of the reasons for the lack of innovation in the email space is the lack of a simple yet powerful email access protocol. Every developer that wants to try something with email needs to first jump through the hoops of IMAP and MIME, or worse, the Outlook Object Model and MAPI. A new protocol like reMAP would lift this burden off their shoulders. We've seen what open, simple standards can do for innovation with Twitter's and Flickr's API. Now imagine unleashing the same sort of creativity to the vast ocean of data that is email.

Let me know what you think. My goal with this post was to encourage a discussion about this topic, and your comments are much appreciated.

If you liked this, consider following me on Twitter where I often tweet about email-related topics!

"A good email client, please."

My friend Patrick Collison writes on Facebook:

"Gmail proved that, despite the apparently high switching costs, a new webmail client can quickly get a lot of traction. There’s room now to do to Gmail what Gmail did to everything else. The replacement should have some concept of workflow (”archive, but remind me to respond tomorrow”, “send, and alert me if I don’t get a response within a few days”), some concept of teams and colleagues (allow threads to be shared as a first-class object, rather than flailing around with forwards and CC lines), be some way smart about mining the semi-structured mails going through the system (flight booking emails should be automatically annotated with .ics files), know something about prioritization (I like Twitter DMs because of the assumption that they’ll go to a mobile device. If I’ve sent more than 20 emails to someone, they should have the option of copying their mail to me as an SMS).

Potentially most powerfully of all, developers should be able create their own plug-ins that run on the server. There should be an agreement between plug-in developers and the webmail provider that creating a plug-in automatically grants a royalty-free perpetual irrevocable worldwide (etc.) license to the provider, and that the source code to any plug-in may be merged into the main product. Though plug-ins have niche appeal, this could be a good source of new features, and a strong competitive advantage. I’d just fix Gmail if I could.

I’d happily pay for any service that got this stuff right."

Some good ideas in here. Server plug-ins could work in a manner similar to Google App Engine. A lot has been tried around better triage and data mining, with no clear winners yet. Also, there's a danger of feature overload somewhere in here. I'm not sure if we should replace one variant of feature overload (Microsoft Outlook) with another (integrated SMS / Twitter / Facebook / email). Maybe SMS, Twitter, Facebook, and email should instead all just be replaced with one thing that is the combination of all lessons learned with those protocols.