Yelp now shows that it de-values reviews from users who have only written a few of them.Yelp now shows that it de-values reviews from users who have only written a few of them. (Richard Vogel/Associated Press)Have you ever visited a review site in search of a restaurant recommendation, picked the best-rated venue, but ended up disappointed by the food and feeling the restaurant's reviewers had misrepresented it?

We've learned to rely heavily on community review sites to assist in everyday discovery tasks, helping us make choices on everything from finding a restaurant, plumber or used car to more important decisions, such as legal services or even doctors.

According to comScore and the Kelsey Group, nearly 25 per cent of Internet users reported reading online reviews prior to paying for an offline service, and 80 per cent of them claimed the reviews had a significant influence on their purchasing habits. Depending on what is needed, consumers were willing to pay a mark-up of 20 per cent to 100 per cent for services and products with excellent peer ratings on review sites.

That said, each of us likely has had an experience of feeling misled by a review site, and the media has no shortage of stories of businesses either shilling review sites (Belkin, Reverb, Avvo) or suing review sites claiming they've been unfairly represented (Yelp most recently, but also Angie's List).

To date there's been little explanation as to why and how we should trust these modern review and ranking services, and even less investigation into what really makes a review reliable.

This is especially relevant since consumers and businesses alike are starting to use community reviews to make decisions on high-ticket, high-cost-of-mistake purchases like legal or professional services.

The cost of picking a bad lawyer to represent you in a make-or-break deal is far larger than that of eating a bad burrito.

The review industry, which flourishes on the principles of information and business transparency, needs a heavy dose of transparency itself.

Gaming the system

Ranking is the most important function of any recommendation "system," be it a search engine, review site or friend. Humans are notoriously bad at choosing between too many things, so we've learned to rely on external signals like recommendations and reviews to narrow the set of possible choices. Since recommendations often influence purchasing decisions and money is at stake, however, there is a compelling interest for entities being reviewed to find ways to game the system.

Historically recommendation systems have opted to combat gaming by keeping their ranking algorithms secret. The assumption behind a hidden algorithm is that if entities being ranked don't know the rules of the game, they are incentivized to behave in a trustworthy way, since they're not sure if their gaming efforts will prove fruitful or harmful to their cause.

There are two significant problems with this approach, however.

First, the fact that the rules are unknown clearly doesn't stop businesses and individuals from trying to game the systems. The existence of entire industries specializing in so-called "black hat" SEO, or of businesses willing to pay for positive reviews, shows the interest to get rank by cheating is unabated. The result of these efforts is an information asymmetry known by economists as a "market of lemons," with consumers eventually trusting neither the underlying system nor anything ranked by it.

The second problem with secret rankings is even more pernicious. We are naturally — and rightfully —wary of the "because I said so" answer. This works against sites with undisclosed algorithms and can have very real consequences.

Learning from recent events: Yelp

In Yelp's case, now that we have some distance to evaluate it properly, some businesses "inferred" from the secrecy of Yelp's algorithm that it was favoring advertisers in its rankings, likening the site to running an extortion scheme.

Yelp's response, in my opinion, was right on. By providing more transparency into how and which reviews influence businesses' ranking on its site and how its ranking algorithm actively works to prevent gaming by malicious businesses, Yelp both started to address the offended businesses' complaints head-on and helped provide more trust in its rankings (and hence its service) overall.

With this move, Yelp has started to join a series of services that favor more transparent ranking algorithms. As more consumers grow wary of not knowing the factors behind recommendations, this trend will only gain momentum.

The future of the web-based review

The problem is that building ranking systems that are both transparent and hard to game is very difficult, and is one of the primary reasons services opted to go the secret algorithm route in the first place.

That said, there are a number of services today experimenting with various approaches at more transparent ranking algorithms. Among the most progressive are the following:

  • Quora uses votes from people you know — or at least whose identity you can explore by viewing their linked Facebook profile — to create a sense of affinity between the reader and the ranker.
  • Stack Overflow provides a similar voting model as Quora, but factors votes that hold more value from experts in certain topics.
  • Yelp now shows, for example, that it de-values reviews from users who have only written a few of them.
  • Digg uses a weighted ranking system where votes from community members who have been able to accurately predict trending topics hold more worth.
  • Thumbtack and RedBeacon both use information on whether you've actually purchased a service on their marketplace as a determinant on whether you can submit a review or not.
  • ChoiceVendor, the business-to-business review site I founded, displays the role and industry of the reviewer to help business decision-makers find reviews written by people who hold roles similar or relevant to their own. We also reserve the right to request invoices for services provided to create an additional level of guarantee that the reviewer has actually done business with the vendor he or she is reviewing.

These are but a subset of the sites, and we're still early on the learning curve for these types of transparent ranking systems. None of the algorithms above are foolproof in preventing gaming or optimally ranking entities. But as we improve relevancy and play cat-and-mouse with gamers, these algorithms will improve while still retaining their newfound transparency.

Anonymous out. Personal in.

The big loser here, I believe, will be anonymous reviews. Users are increasingly interested in not only what is being said or ranked, but also in who is saying it. The human element, and an understanding of the reader's affinity with the reviewer, is becoming increasingly important in creating the necessary trust needed to value the review.

By making the ranking system less one-model-fits-all and more subtle, the site simultaneously makes it harder to game.

Personalization, however, stands to be a big winner. "Objective" reviews (e.g., "this is the best Italian restaurant...") are already starting to lose their worth in favor of personalized reviews (e.g., "this is the best Italian restaurant for me..."). Netflix, for example, has done an amazing job providing personalized and relatively transparent recommendations.

Interestingly, personalized rankings also help in the battle against gaming. A rank-gamer wanting to fool me into buying his services by pretending to be "like me" on a personalized ratings site might be missing out on an opportunity to get someone else who isn't like me to buy his wares instead. By making the ranking system less one-model-fits-all and more subtle, the site simultaneously makes it harder to game.

We're at an interesting time with regards to the evolution of review sites, and of ranking as a whole. Demand for increasing transparency in the ranking algorithms will force sites to think of clever new ways to rank entities while being transparent to users, fair to things being ranked and simultaneously hard to game. As consumers and businesses continue to flock to recommendation sites as trusted advisors on a variety of important evaluation decisions, this problem and its myriad solutions will only increase in importance.

Yan-David Erlich is founder and CEO of ChoiceVendor.com, a site that helps small and midsized businesses connect with vendors and service providers. He also cofounded Mogad, a social recommendation service. Prior to ChoiceVendor and Mogad, Erlich worked on ranking algorithms at Google.