Predicting Whether You Need to Reply to an Email
Every couple of months, I try to catch up on email research work from academia. This weekend, I found this paper [1] by Drezde et al. from UPenn.
The problem they want to solve is predicting whether an email in your inbox needs to be replied to or not. This is very relevant research as it would allow users to quickly scan their email for items they actually need to act on. The result could just be a simple "needs reply" indicator in the list of incoming items.
How do they do this? The authors used a number of attributes to classify emails, such as whether the user frequently replies to the author, whether the email contains any question marks, and combinations of words appearing in the text.
The results aren't quite there yet. For the best test corpus, they achieve are 77% recall - which means that they find 77% of the emails that need replying. However, they come in at 76% precision, which means that 24% of emails they mark as "needs reply" don’t actually need replying.
Thus, reply prediction remains exciting. I'm hoping that they come up with a better classifier, and that someone then turns this into an industrial-grade email application.
[1] Mark Dredze, Tova Brooks, Josh Carroll, Joshua Magarick, John Blitzer, Fernando Pereira: Intelligent Email: Reply and Attachment Prediction, Intelligent User Interfaces 2008, Spain. [PDF]
The problem they want to solve is predicting whether an email in your inbox needs to be replied to or not. This is very relevant research as it would allow users to quickly scan their email for items they actually need to act on. The result could just be a simple "needs reply" indicator in the list of incoming items.
How do they do this? The authors used a number of attributes to classify emails, such as whether the user frequently replies to the author, whether the email contains any question marks, and combinations of words appearing in the text.
The results aren't quite there yet. For the best test corpus, they achieve are 77% recall - which means that they find 77% of the emails that need replying. However, they come in at 76% precision, which means that 24% of emails they mark as "needs reply" don’t actually need replying.
Thus, reply prediction remains exciting. I'm hoping that they come up with a better classifier, and that someone then turns this into an industrial-grade email application.
[1] Mark Dredze, Tova Brooks, Josh Carroll, Joshua Magarick, John Blitzer, Fernando Pereira: Intelligent Email: Reply and Attachment Prediction, Intelligent User Interfaces 2008, Spain. [PDF]
4 Comments:
Thanks for the mention.
I don't recall if we put this in the paper or not, but the results we got were close to how well people did on this task (labeling each others email). Looking at many of the misclassifications, its hard to see getting much better.
However, the accuracy of this problem depends on the UI. If you just want to show if an email needs a reply in the UI (I built such an extension for thunderbird) then this may not be good enough. However, if you used it as some other method, perhaps as part of prioritization or less confident indicators, it may be useful. We had some ideas about managing expected replies: emails for which I am waiting for a reply. In that system you may not need high precision but higher accuracy.
really Nice thanks for pointing this research out !
regards
John Jones
http://www.johnjones.me.uk
Mark -
Thanks for the reply - good points. If the results are as good as humans can classify email, it almost seems like the current results are an upper limit of what is possible.
Here at Xobni, Greg Duffy experimented with some SVN-based classifiers for reply prediction, and we got similar results as in your paper. Thus, a simple switch of algorithms doesn't help.
Let's chat sometime about different UI solutions for presenting the reply prediction labels.
Gabor
I advocate sorting the messages by what group the sender was in (like BiFrost, but hopefully with more than five groups!), then colouring the message based on how you were addressed -- to you and only you, cc you, bcc, etc. This very quickly gives you a 2D view (where one of the axes is color) of who it came from and who it went to. The location in that 2-space is a really good indicator of whether you need to respond or not, it is cheap to calculate both dimensions, and the wetware is good at making the final decision.
I talk about how to group-by-sender-group in my books
http://emailoverload.com
but at the time, didn't think that colour-coding was such a big deal. Now I think it is a bigger deal, see
http://www.emailoverload.com/outlook/ColorCoding.php
for how to do that in Outlook.
(It's also MUCH easier to set up colour-code than to group-by-sender!)
Post a Comment
<< Home