The IMAP Compatibility Matrix

There's a great discussion going on in the comments of my post about replacing IMAP, and also on YCombinator's Hacker News.

One frequent point made about my proposal is that IMAP already has a number of extensions that do things I suggested, such as the THREADS, SEARCH, and IDLE commands for conversations, search, and long polling ("push").

Commenter Bron (who works on a popular IMAP server) writes:

Also, IMAP plus a random mix of extentions becomes a HUGE compatibility matrix, and you have to offer fallback from all the nice stuff to really shitty workarounds just in case the server doesn't support a feature that makes your life bearable.

So - I'd love to see a decent competitor to IMAP as a protocol. If nothing else because it would reduce the compatibility matrix. Alternatively, I'd be happy for an IMAP5 to be defined which included all the good extentions - so anything which advertised support for IMAP5 had to support them.

Defining an IMAP5 standard might be a good alternative than something like reMAP, and adoption might be much faster. However, a potential IMAP5 would still remain a protocol inaccessible to many developers so it might not unleash the storm of creativity I'm hoping for.

Keep those comments coming.

How to Replace IMAP

IMAP is a complex and difficult protocol. Its original design, RFC 1730, dates back to 1994. Here's a sample IMAP session that shows you all of IMAP's quirks: It's a stateful protocol where you login and select a mailbox to operate on, it has an unconventional parenthesis-heavy representation for structured data, and fetching messages is a multi-step operation.

Design Outline

One of these days, I'll write up a spec for the mail access protocol of my dreams. I'm planning to call it the "reimagined Mail Access Protocol" (short: reMAP) which is handy because reMail already owns the domain name :-)

I would aim for a RESTful design with the following properties:

1. All communication over HTTP / HTTPS: Pure TCP connections are great, but for transferring large amounts of email, HTTP is the way to go. Problems like security, parallel downloads, persistent connections, caching, compression, download continuation via ranges, and so on have already been solved. There is no reason to solve them again.


2. Stateless: There's no reason to introduce state like IMAP does with its selected mailboxes. All you need is a HTTP session cookie used for authentication purposes. Session cookie also allow for things like OAuth. OAuth would let third parties get your permission to access your email without having to give them your username and password.


3. JSON and UTF-8: All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML. UTF-8 would be the only encoding allowed, since it is able to represent any character in the Unicode standard.


4. Conversations as first-order objects: Gmail, the iPhone SMS app, and Facebook's messaging system have shown the value of viewing messages not individually but in the context of conversations. In reMAP, the server would be responsible for grouping together messages. While you could still access individual emails, the first-order unit of data would be a conversation.


5. Labels, not folders: Labels are much for flexible than folders. Each conversation should have multiple labels, and the labels would be included when you request the message, rather than having to scan all folders for the message via IMAP.
   
6. Stable and unique IDs: IMAP has a UID for each message, but it changes the moment you move the message into a different folder. An IMAP server can also declare all UIDs to be invalid at any moment throughout the session. No more! reMAP would have stable and unique IDs for all conversations, emails, and attachments.


7. The beginning of the end for MIME: Yes, you could still get the MIME representation of each message that is sent. But MIME is a messy and complex beast. Instead of requesting the MIME-encoded message parts, you could just ask the server to give you the message as represented in plain text or HTML. Attachments can be downloaded in separate HTTP calls.


8. Push built in: The two prevailing methods for implementing push email are the IMAP IDLE command (not widely available in IMAP servers) and Microsoft's ActiveSync, which requires developers to purchase a license from Microsoft. In reMap, clients could just call an HTTP endpoint on the server which returns as soon as new messages are arrive.


9. Full-text index on the server: reMAP servers would need to maintain a full-text index of the contents of all messages. There's no reason clients should be required to download and index everything in order to do an exhaustive full-text search of your email.

Enormous Potential

I believe that one of the reasons for the lack of innovation in the email space is the lack of a simple yet powerful email access protocol. Every developer that wants to try something with email needs to first jump through the hoops of IMAP and MIME, or worse, the Outlook Object Model and MAPI. A new protocol like reMAP would lift this burden off their shoulders. We've seen what open, simple standards can do for innovation with Twitter's and Flickr's API. Now imagine unleashing the same sort of creativity to the vast ocean of data that is email.

Let me know what you think. My goal with this post was to encourage a discussion about this topic, and your comments are much appreciated.

If you liked this, consider following me on Twitter where I often tweet about email-related topics!

Dear Facebook, Please Let Me Reply to Your Message Notifications

Do you know this problem? Someone sends you a Facebook message. You want to reply, but instead of being able to just hit reply on the message, you have to perform a multi-step choreographed dance:

1. Click on the link in the email
2. Log into Facebook
3. Navigate Facebook's messaging interface to reply
4. Repeat this procedure when new messages roll in

This is hugely annoying. Instead of sending emails with a nonsensical address such as notification+pd=edfz@facebookmail.com and a Reply-To of noreply@facebookmail.com, can't Facebook just implement an email-reply-to-Facebook-message bridge? This is pretty simple to do - many Support systems (e.g. Kayako) already do this.

I've written (well, sketched) about Facebook vs. email before. When I have some time, I need to sit down and write my rant about how closed (Facebook, Twitter, Skype) and proprietary (Google Wave) systems are replacing email when they shouldn't. I think the underlying reason is that email's systems and protocols (SMTP, IMAP, MIME/RFC822, MS Exchange) are so hugely sucky, outdated, insecure, spammy, bug-prone, and stupidly designed. We need to engineer ourselves out of this mess.

Update: Wow, this post was just in time for Facebook's Message API, which makes the whole problem worse, not better: From the TC article: "The biggest addition — the Mailbox API — is also disappointing because it only lets users receive messages, not send them."

I need your IMAP Settings!

I'm planning to build an "IMAP Settings Directory" for an upcoming version of reMail Search, and I need your help!

Right now when you sign up for reMail Search, you need to manually enter your server settings. That means not only looking them up on your computer. You also need to type it in on the iPhone's small keyboard. That's painful.

In the next version of reMail Search, I want to take a different approach: You tell me who your email provider is, and reMail will pre-fill IMAP settings for you. I want to have this for the top 20 hosted email providers at a minimum.

If you're using a third-party provider to host your email, please leave a comment on this post with a link to their IMAP configuration page. This crowdsourced approach worked well for an Outlook-related request I had last year, so I'm going to try it again.

Here are two examples for what I want:

1. Mailtrust IMAP configuration page
   
2. Gmail IMAP configuration page