Rating, reviewing and ranking systems – Part 1

Ranking and rating are activities that are almost entirely subjective by nature. They are based on a (usually unknown) value system. In the case of online review engines, the value system of a reviewer is seldom captured by the systems in question, and the value system of the person querying for reviews / rankings is also usually an unknown.

A while ago a colleague brought to my attention a site called cinemaclock, which showed movie ratings according to moviegoers. “It’s great”, she said. “It lets you see the reviews according to gender and agegroup!” Which is a great idea, I thought. But browsing through the reviews, you quickly realise that this device is not quite enough… In fact, there’s a raft of such devices that are used to mask how basic our review systems (still) are.

First, I’m gonna get some made-up terminology out of the way:
I’m gonna call any kind of rating/reviewing/ranking/recommendation system a “Review system”, or RS for short. And I’m gonna call anyone who winds up on a website sifting through reviews and ratings or trying to decide what to buy etc a “Seeker”.

Seeker Modes

– seekers in ‘survey’ mode are usually after some kind of overview or holistic understanding of the spread of responses to a particular type of product(s) or service. They’re probably doing some preliminary research.

– seekers in ‘decision’ mode usually want some kind of magic, intuitive ranking of a particular type of product or service, based on their own needs and their particular profile (including personal preferences).

Seekers will often vacillate between the two modes, especially when there are no intuitive tools to allow them to quantify or qualify their needs succinctly. This kind of vacillation (between modes of querying the system ) typically adds to the stress that may already have been induced by vacillations of the first kind: trying to choose between the products / services themselves.

I sometimes wish review engines would try harder. They still employ the same old strategies from back  when data-driven (or, as they used to be called, ‘dynamic’) websites became possible. And then the innovation kind of… stopped.

Current RS strategies

strategy #1 : distinguish ‘top reviewers’

In recognition of how much data the average seeker needs to wade through, several online review systems attempt to identify ‘quality contributers’ by (human?) monitoring of the population of reviewers. The site owner ‘promotes’ the reviewers that people seem to like, who provide lots of details, who review often… In short, people whose opinion everyone *ought to* value.

(A similar strategy is to ‘distinguish the the top reviews‘).

the problem: identifying top reviewers (critics with clout) leads to bias (because we all know clout begets clout. If I review movies for a leading newspaper then the chances of rottentomatoes.com promoting my viewpoint is greatly increased, for example). Admittedly, on rottentomatoes you can pick your own ‘favourite reviewers and critics’… but this requires you to spend time getting to know who those people are, and whenever your movie-oriented ‘value-system’ changes, presumably you’re going to have to spend another bunch of time picking new faves…

another problem: without ready-made top reviewers, or mechanisms to pick your own favorite reviewers, the site owner has to allocate resources (time, money) toward picking the best out from the pool of reviewers / reviews.

All attempts to bring quality reviews/reviewers to the fore is admirable, but that’s about all you can say about it. It attempts to route round a deeper problem: While the actual text of a review is often made available, no attempt is made at auto-tabulation or ‘keying’ of such long-form text. (And as it happens, tabulating from free text is a non-trivial problem – it usually requires several humans… Preferably ones who can count and type fast).

strategy #2 : Filtered searches

This is done by exposing a range of product attributes (some actual, some imposed – eg via tagging) and allowing the user to ‘drill down’ with a range of parameters.

the problem : After fighting with search forms, it is anti-climatic to be faced with a giant (or even worse, non-existent) list of ‘matches’ that you then have to wade through, probably by wading through a whole bunch of reviews for each item on that list. If you’re lucky you’ll get a few ‘sort by’ columns, courtesy of the code monkey behind the scenes, which you can at least play with while pretending you’re getting closer to a decision. While there will always be a level of ‘input activity’ expected from the end-user, the pay-off needs to match the effort they feel they’ve put in.

strategy #3 : side-by-side comparison charts

If you go to cellphones.ca you can see a neat little side-by-side product comparator tool. However this method is all show and not enough tell. It’s a strange idea that somehow the proximity – to each other – of the values for a specific attribute of 2 or more products of the same type is somehow enlightening. I’ve no doubt that it taps into some visual processing that inches us closer to a decision. But “inches” is the operative word.

strategy #4 : rating-fuelled ranking

You’ve all seen ’em. You go to a site and see ‘stars’ next to an item and are expected to rate it, in one or more dimensions, by giving it a certain number of stars. This simple process makes it a sinch to tabulate results and post rankings on the fly, but the rigidity of the process (fixed attributes, little or no long-form descriptions, etc) makes the rationale for the final ranking quite opaque. BUT: For very simple/free/disposable entities, this may not matter and star-ratings alone can be a good way to go.

I think out of all the strategies, it offers the most promising basis for looking at the next generation of RSes.

In my next post on rating & reviewing engines I’ll look at an approach that might help alleviate some of the problems mentioned here.

Leave a Reply

Your email address will not be published. Required fields are marked *