This topic comes up from time to time with both existing clients and potential clients. The approach that’s most often proposed is something akin to product ratings on Amazon, or movie ratings on Netflix. On the surface that seems like such an easy, simple idea. Why not do it?

We’ve spent a fair amount of time thinking about this in our product management meetings and have run into some problems with the practical side of offering it:

• This would be easy for a salesperson to do (when they take the time to “close the loop”), but would it be useful information? For instance, if a reference call or site visit didn’t go well, was it the fault of the customer reference? Was it a bad match for other reasons? Were the prospect’s expectations off base?
• If a simple 2 out of 5 stars (let’s call this a “one-click rating”), or “2″ on a scale of 1-5 system is used, but that doesn’t tell the whole story, then more context for the rating is needed. What additional questions need to be asked? How much time can you expect a salesperson to dedicate to closing the loop? If only 20% of reference activities, for instance, are qualified after the use, is that enough actionable information?
• Let’s assume you’re able to get the one-click rating information and some qualification data on each use (and that’s a BIG assumption). What will you do with any references that score lower? Will you stop using them? Coach them? Probably a combination, but this assumes you have enough to go on, and the time to act.
• One significant difference between the Amazon rating system for products or books, and something similar for references is volume. There may be hundreds or thousands of reviews about a book, there may be 4, 5 or 6 about a customer reference..over a 6-12 month period. If one of those is negative it will skew the overall score by quite a bit. Is that an accurate reflection on the customer’s ability to help? Or was the one low score not a result of the customer’s actions?
• When customers receive enough, and it may not take much, negative ratings (most likely without explanation) do you take them out of the “rotation”? If these ratings are visible to all users then the pool of references sales will want to use will get increasingly small. The chances for overuse of those customers with higher average scores will increase. Uh-oh, that wasn’t the goal!

We’d like your comments on this dilemma. In concept the idea has merit. In practice we’re stumped. There’s no point in offering a “sexy” feature like this if it’s ultimately meaningless, and perhaps even harmful.