Does Metacritic need some fixing?
By: Jeff Rivera
Metacritic is both a blessing and a curse. For all the usefulness we get out of it, there's no denying that its formulaic nature can cause unintended problems for people researching game scores and even for the game developers who have to hope for good enough Metacritic averages to convince publishers to green light future projects. So what conveniences and problems does it bring, exactly? And are these problems serious enough that we should be urging Metacritic to adjust their methodology?
Let's start out with the things that Metacritic does well. Metacritic makes it extremely convenient to read how multiple sites critically received a particular video game. If you're on the fence about a game, you can hop on over to Metacritic, and quickly see tons of reviews pulled together in one place. From there you can jump out to individual reviews for more detail. And while not every game gets reviewed by the bigger review outlets, you can generally find at least a handful of reviews for even the most obscure releases.
Another nice aspect of Metacritic is that you can often discover review sites that you've not visited before. Often times we get into a rut where we're reading reviews from the same two or three sites, and Metacritic helps us broaden our awareness. Metacritic even goes through a vetting process to ensure that spam contributions or fly by night sites don't clutter up the aggregation. When it comes down to it, Metacritic really does try to provide the best service possible.
There are some problems that I feel need to be addressed, however. As Metacritic works to create an average overall review score, they need to convert the scores from all the outlets they draw from into their own scale. For some sites, it's quite easy as they're already dealing with a number based scale, but for sites that use letter grades or a four or five star Metacritic breaks them down and converts them to a number scale. This is really problematic, because Metacritic is assuming that each star or letter grade is worth 20% of the total score, but these sites have chosen to abandon a number system for a reason and don't want each star acting in that manner. Even more extreme are the cases in which sites don't give any sort of score at all and Metacritic assigns their best guess.
Now I understand why Metacritic needs to convert letters and stars to numerical values, but I've seen many reviewers complain about this on Twitter, on podcasts, and in talking with them at media events. Gripes from reviewers aside, the inaccurate nature of the conversions is far more dangerous to developers.
Many people are unaware, but some publishers tie pay bonuses for developers into different types of goals. The most common bonus is tied to sales, which makes good sense, but in increasing cases, Metacritic averages are part of the equation. Also, it's becoming quite common for publishers to take past Metacritic averages into account when deciding whether or not to award development contracts. And unlike curious readers, publishers are looking at the hard average, not the reasons for why each individual score was given out, or how each score was translated to its Metacritic-friendly form.
So when I give a game a 3 out of 5 star grade, I'm saying it's a good game and that I'd recommend it, only with a little reservation. Metacritic would take my score and change it into a 60%, which is a rough score from outlets that deal with a ten point scale. Publishers are unhappy with a 60% score, despite my claim that the game is good.
I think Metacritic is due for an overhaul, because in theory the service in excellent; it just happens to be flawed. If sites are going to participate in Metacritic, they should still give up their score as normal, but they should also have to state whether or not the game is recommended or not. To go with this, Metacritic would display the site's scores with their reviews, but rather than an overall averaged score for all the outlets, they display the number of sites that recommend the game or not. This would be far more fair to developers and a better representation for Metacritic's readership.
And maybe this isn't the best solution, but at least it's a step in the right direction. If Metacritic's purpose is to help people more rapidly decide if a game is worth playing, I'd prefer a more accurate system than what we're getting now.
