Review Scores Are Meaningless

Review Scores Are Meaningless

Here’s a hypothetical situation for you: The most anticipated new AAA shooter just came out, let’s call it “Military Shootface V: Grey and Brown Edition.” So you head on to your favorite review sites (starting with Cheat Code Central, of course) to check out the review scores. However, there is a problem. No two reviewers can seem to agree. Some reviewers rate the game with 9s and 10s while others only give it 6s or 7s. Which review is correct?

This seems like a stupid question, right? No review can be “correct” in a truly objective sense. Reviews are a matter of opinion, a subjective experience had by each individual reviewer. So these numbers can really just tell you how a reviewer felt about a game while playing it.

But what does that really mean? You probably don’t know any of these reviewers personally. You don’t know if these reviewers value graphics over plot or gameplay over replayability. You don’t know if the reviewer just doesn’t like shooters or if they really like shooters. And even if you knew these things, it would be hard to know exactly how that would affect any game’s eventual score. Say he was a huge shooter fanatic. Would he rate the game higher because he believes shooters are a superior breed of game, or would he rate the game lower because he has played enough shooters to truly see how good and bad they can get?

The fact of the matter is, you don’t know. I mean, let’s say we were trying to find a restaurant to eat at and I said, “Steve gives this place a seven.” When you ask, “Who is Steve?” I say, “I don’t know.” You’d flat out think I was insane, right?

Review Scores Are Meaningless

The biggest problem is that a complex opinion just cannot be expressed in the form of a number. If you went on to actually read every review, you’d see that most of the reviewers agreed that the game had decent gameplay and voice acting but flawed graphics. That tells you infinitely more about a game than simply looking at a number between one and ten. Yet, many outlets continue to provide scores because of fan demand. Much of the gaming community is only interested in the final meaningless score rather than the review. My theory is that kids these days just can’t be bothered to sit down and read. /OldMan #ReadingRainbow

In fact, the limits of human psychology make review scores inherently meaningless. Let’s say you are a halfway decent game reviewer and you get to review one new game every week. In a perfect world, games that you like would be rated better than games that you dislike. So if you liked “Spikey Hair JRPG 7” more than “Dribble and Shoot Basketball 2015,” you would rate Spikey Hair with a higher score than Dribble.

Except this doesn’t always happen. I’ve been a games journalist for six years now, so if I actually did review a new game every week, I would have reviewed 312 games by now. So to accurately judge every game against every other game I’ve ever reviewed, I’d have to remember 312 game scores and have been able to judge each title 100% accurately against my gaming preferences. This just won’t happen. In fact, most people’s long-term memory just isn’t that good, and looking up every single review score before writing would be an exercise in time wasting. Can you remember 312 review scores from reviews you have read?

So as long as you’re doing well in a COD game, you are constantly being handed short-term rewards.

So review scores are inevitably altered by external forces. Has the reviewer’s opinion of the genre or the franchise gone up or down? If rhythm games have been sucking recently, a halfway decent release will probably end up getting rated higher. Have the releases this year been good or bad? If the market has put out artistic blockbuster after artistic blockbuster, then the bar has been raised and games that are good but not complete blockbusters will be rated lower because of it. How is the reviewer feeling that day? If he got stuck in traffic, spilled coffee on his shirt, and his girlfriend left him, there’s a chance he might be harsher in a review. (This shouldn’t affect scoring in a professional environment, but it’s hard to say there’s not subconscious factors involved when giving out review scores.) So when a fan writes in hate mail to say, “How could you give Game X a 9 while you gave Game Y a 10?” the answer is always “human psychology.”

Numbers are an objective thing. 3 always means 3. The “average” score from 1-10 should be 5.5. So an “average” game should rate a 5.5. However, since reviews are subjective, this is not the case. Instead, we have come to regard 7s and 8s as the new “average” because of how harsh he number 5 seems. This review creep has lead reviewers all over the place to cluster their reviews around high numbers rather than low numbers. On an accurate mathematical bell curve of out-of-10 review scores, you would see approximately the same number of 8’s as you do 3’s. This is most certainly not the case.

Review Scores Are Meaningless

There is only really one way that numbers in the reviewing business can be somewhat accurate, and that’s when you take an average of all of them. And still, this meta-score doesn’t really track game quality as much as it tracks a trend of public opinion. If game scores average out to a 7.0 after 100 people reviewed the game, then you can be safe in assuming most people have had a somewhat positive experience with it. These are the same metrics we see used in services with Yelp and Amazon. Similarly, that’s why you may look at a restaurant with one five-star rating on Yelp more skeptically than you do a restaurant with 35 four-star ratings.

As long as the gaming populace is too busy to actually read the words of a review, then reviewers will have to continue to rate games with numbers. From IGN to Destructoid, these numbers are merely the tip of the iceberg and cannot even come close to representing a complex opinion. By simply looking at a score and refusing to read the thought process behind it, you are doing a disservice to both yourself and the reviewer who took time to articulate his or her point of view. So take some time to actually read, because while a picture is worth a thousand words, a couple paragraphs can be worth a million numbers.

Angelo M. D’Argenio
Lead Contributor
Date: December 26, 2012

*The views expressed within this article are solely the opinion of the author and do not express the views held by Cheat Code Central.*

blog comments powered by Disqus