Akismet Marks Valid Comments As SPAM!

The Akismet anti-spam system is probably the most handy tool when it comes to managing submitted comments and trackbacks. It automatically protects bloggers from hundreds or thousands of spammy comments and now features a nice little statistics suite. However, lately, we haven’t been too impressed.

Recently, after commenting on several websites that used the Akismet plugin to filter comments, we noticed two things. First, our comments weren’t showing up. And second, we didn’t even get to see the familiar message noting that our comment had been received and that it was under moderation.

What was going on? Akismet.

Apparently, our comments — which were valid and pertinent to the subject matter — had been incorrectly classified as spam and automatically relegated to the spam folder of WordPress. And given that we often comment on popular and well-known blogs, the blog administrators would probably ignore the spam folder and thus, the goodies inside would be lost forever.

So why was Akismet doing this?

Well, it has to do with the way that the plugin works. Akismet is not simply a captcha that works independently on a blog. Instead, it works by tying into a shared Akismet database of known spam offenders. Additionally, it builds its list of offenders from people like you and me.

On the Akismet website, the FAQ page states:

When you mark a comment as "spam" in the plugin, the mistake is noted by Akismet and it learns from your submission.

So if some blog admin tags a valid comment as spam, then Akismet learns and the next time that the user is encountered, all comments are automatically classified as spam.

The problem with this system is that an admin might maliciously or unknowingly classify a valid comment as spam for any number of reasons (as opposed to just deleting the comment). And then on some other completely unconnected blog using Akismet, that user will be penalized.

The makers of Akismet know about this problem as well. Under the next subheading, Help! Akismet is catching a regular comment as spam!, it states:

Don’t worry, if you see a regular comment on your Akismet page, just click the "Not Spam" checkbox and submit and the comment will be sent back to Akismet as a mistake. The system will learn from your submission, though it may take a day or so in some cases. False positives, as they’re called, are extremely rare and we watch them closely.

Clearly, this system has some flaws and could be easily exploited. We will cover this later. But first, lets reverse the trend and stop submitting spam.

From Spam to Ham

To clear your spam status, all it usually takes is one intelligent admin who digs in the spam folder and finds your incorrectly classified comment. When they mark "Not Spam", you will be absolved. Unfortunately, this approach relies on the assumption that others will sift through their spam folders and that they will like or agree with your comment enough to promote you. And depending on how often you submit comments and whether or not admins review their spam, this might take a while. So instead, we suggest that you comment on one of your own blogs and mark yourself as "Not Spam". Or, if you do not own a blog, simply contact Akismet.

Gaming the Akismet System

We are not the first to have figured out this Akismet flaw: the false positives and the easy way to attack an unsuspecting blog commenter. Back in December of last year, for example, the folks over at Optempo discussed these difficulties in depth. And even earlier, The Great Startup Game had it’s say. But what does Akismet say?

Back to the FAQ page, it states:

Well without giving too much of the secret sauce away, we can safely say that it would be pretty difficult to poison Akismet. We use dozens of factors to determine the spamminess of a submission, and we also have an identity attached to everyone using and contributing to the system, which allows us to do some interesting things with weighting and clustering activity.

Sounds like rhetoric. And it isn’t convincing after having been classified as spam, along with scores of other popular commenters and well-known bloggers. Clearly, this phenomenon isn’t as rare as Akismet may believe.

Yet, we are stuck with this problem because an estimated hundreds of thousands of blogs use Akismet. And given that this has been a well-documented issue for well over a year now, without any significant changes to the underlying system, we the bloggers will have to take a bit more responsibility in reviewing our spam folders and all of the spam or ham within.

Secret sauce…

Have some Akismet tales of your own? Please share.