WineMcGee.com

Why do most publications and retailers use a 100 point scale for rating wine? I found that a lot of other people have had the same question. Some great discussions online, notably at Dr. Vino and Fermentation, show that everyone has an opinion.  I’ve listened to the debate, read the scientific research, and I’ve come to a conclusion: the 100 point wine rating scale will go away.

What is the 100 point scale, anyway?

The 100 point wine rating scale was popularized by Robert Parker and his publication, The Wine Advocate, which was first published in 1978.  It is commonly suggested that Parker’s 100 point system gained popularity because it is simple and that everyone is accustomed to percentages.  But while the 100 point scale appears simple on the surface, when you dig deeper it’s actually quite confusing.

There are currently at least three major varieties of the 100 point scale within the US.  For the Wine Advocate and Wine Spectator  the  100 point scale really only ranges from 50-100, with 51 different possible values for the quality of wine, each point representing a change of about 2% in quality.  Two magazines that began later, Wine Enthusiast and Wine & Spirits, instead rate wines only from 80-100.  Steven Tanzer and the International Wine Cellar rate wines from 70-100.  Publications also often award points differently than one another.  For example, a study by Cornell found that Robert Parker frequently rated wines more than one point higher than Steven Tanzer.  So, a 90 point rating in one publication does not equal a 90 point rating in another.

And then there are the descriptions within the rating scales that explain what each range of numbers means.  Don’t stare at these too long or your head will wobble.  For example, the words  “outstanding, extraordinary, jaw-dropping, overwhelming, rocking, classic” are all actually used by prominent ratings systems to describe wines at various point spreads from 90-100 such as 90-94, 90-95, 96-98, 95-100.  I’ll bet you can’t guess which descriptor goes with which ratings range.  Isn’t this even a little more complicated than the National Threat Advisory level?  Now is Elevated worse than Guarded and Green is worse than Yellow?  Should I just hide now and you can come get me when it’s over?

The 100 point scale is still a new experiment.

For the rest of the world outside of the US, the 100 point scale is still not widely used.  In fact, many critics worldwide decry the use of any rating system, especially Parker’s, as they feel his bigger, richer, bolder tastes have negatively impacted winemaking worldwide.  Says famous wine writer and critic Hugh Johnson in his autobiography A Life Uncorked:

“If I missed the point of what Robert Parker was trying to do with his percentages of perfection, I made a mistake. To me his numbers game was simply irrelevant. He could score away as he liked, and I could enjoy my wine untroubled. It took years for the realization to dawn that his scoring was influencing the way wines were being made.”

British Master of Wine Jancis Robinson explained her preference to avoid ratings altogether, when possible, by saying:

“But, perhaps strangely for someone who studied mathematics at Oxford, I’m not a great fan of the conjunction of numbers and wine. Once numbers are involved, it is all too easy to reduce wine to a financial commodity rather than keep its precious status as a uniquely stimulating source of sensual pleasure and conviviality.”

In the US, prior to the popularization of The Wine Advocate, wine was frequently rated using the star system, or the 20 point UC Davis scale.  While the 100 point system may have been a great fit in the recent past, it wasn’t always this way, and it won’t necessarily always be this way.

Most people can’t discern enjoyment of wine in increments of 2%.

“I know of no one with three or four different glasses of wine in front of him or her, regardless of how good or bad the wines might be, who cannot say, ‘I prefer this one to that one’ “ – Robert Parker.

Everyone knows whether or not they like a specific wine.  But most casual wine drinkers don’t think in terms of rankings or degrees and would have a tough time consistently blind tasting four wines of the same style and ranking them from 1^st to 4^th with confidence.  And even when someone is able to discern noticeable difference between all four wines, more than likely at some point in the rankings there would be a toss-up:  “I like this one best and the other three are only decent” or “This one is disgusting, this one is delicious, and the other two are about the same.”

Even an expert certainly couldn’t taste 50 wines and arrange them from 1^st to 50^th from best to worst and then expect to do it the same tomorrow or even five minutes later.   At some point it doesn’t have to be a competition, because at some point it makes no sense.  Isn’t it enough to say “these two wines are both extraordinary” instead of “this wine is about 94 extraordinary, and this wine is 96 extraordinary?”  Even Parker has admitted that “probably the only difference between a 96-, 97-, 98-, 99-, and 100-point wine is really the emotion of the moment.”

Wine enjoyment is subjective – more like art than science.

“While some have suggested that scoring is not well suited to a beverage that has been romantically extolled for centuries, wine is no different from any consumer product.  There are specific standards of quality that full-time wine professionals recognize, and there are benchmark wines against which others can be judged.” – Robert Parker

Wine is very different from any run-of-the-mill consumer product.  There are not many consumer products that are rated and reviewed as extensively as wine.  But here are three that come to mind that are similar to wine:  music, movies, and food.  All of these things clearly fall into the ‘subjective’ category and their enjoyment is truly personal.

All wine critics will admit that wine is a subjective experience.  They may say, however, that a 100 point system still makes sense, because they are simply rating the wine based on their specific palate.  But even if critics think they can discern their enjoyment of wine in increments of 2%, consumers most certainly can’t.  Wine critics should understand that they are rating wine for the consumer.

Oddly enough, critics for both movies and music seem to realize that the experience is subjective and nearly always rate in thumbs up/thumbs down, star systems, or a numerical 10 point scale.  Ratings are only presented out of 100 when showing aggregate average ratings – such as on Rotten Tomatoes for movies or Metacritic for music reviews, which makes perfect sense.  When will the wine industry catch up?

Consumers don’t prefer the 100 point scale.  We use it because it’s there.

Many have suggested blindly that ‘consumers prefer the 100 point rating scale.’  When did this become self evident?  Consumer purchase behavior is a complex thing.  A recent study by Australian universities and wine institutions looked specifically at how shelf information influences consumer wine purchasing habits.

In 2001 George Akerlof, Michael Spence, and Joseph E. Stiglitz won the Nobel Prize in Economics for their theory on information asymmetry.  The theory explained that the price a consumer is willing to pay is related to their perceived “purchase risk” and their best-guess value of the product.  The Australian wine study set out to determine how specific shelf information could help decrease perceived purchase risk and help increase wine sales.

The study measured purchase behavior through two online experiments involving simulated wine shelves, in which regular red wine consumers were asked to choose a wine for dinner.  The online shelf simulation was earlier validated to show a very high correlation to actual AC Nielsen sales data – so it seemed to closely model the real-world purchasing process.  The study interestingly found that star ratings  had the highest impact on the purchasing process.  While a 90+ rated wine increased selection by about 10%, a five star rating increased selection by 18%.  Any wines rated at two stars (defined as above-average) or higher actually increased sales as much as a 90+ rating.

Would American consumers have reacted differently?  Maybe.  The 100 point rating scale is much more prevalent in the US than in Australia.  But do American consumers prefer the 100 point scale, or do we simply use whatever information is available to help us make our purchase decision?  If the star rating system is more intuitive than the 100 point scale, perhaps retailers would actually sell more wine using this system.

Experts can’t agree – sometimes not even with themselves.

In the fall of 2008 Robert T. Hodgson, a retired statistician turned wine marker, published studies on judge reliability and agreement of results among US wine competitions.  Since then, there has been much debate about his research.  A Wall St. Journal article titled “A Hint of Hype, A Taste of Illusion” and a myriad of blog discussions online further ignited the discussion in November 2009.

Hodgson’s research on judge reliability set out to determine how often a professional wine judge would produce consistent ratings when trying the same wine on separate occasions.  The data from 16 panels of four judges were analyzed and it was determined that a typical judge had a consistency range around +/- 4 points.  That is to say, for example, that on the first blind tasting they rated a wine an 88, while on the next occasion they rated the same wine a 92. These findings mirror the comments of many wine critics, including the quote from Robert Parker noted earlier in this posting that “probably the only difference between a 96-, 97-, 98-, 99-, and 100-point wine is really the emotion of the moment.”

It’s also commonly known that many experts don’t agree in their opinions about wine. Jancis Robinson and Robert Parker historically clashed in 2004 over their ratings for the new 2003 Bordeaux futures.  The wine was the 2003 Chateau Pavie, and while Parker awarded the wine a 95, Robinson awarded just 12 points out of a possible 20 and called the effort “ridiculous”.  A passive-aggressive war of words ensued and critics all over the world took sides.

Interested in the results from an earlier Cornell study on the similarity of Bordeaux ratings between Wine Spectator and Robert Parker, I wanted to see what correlation existed between a large set of Jancis Robinson’s recent Bordeaux ratings compared to Robert Parker’s.  I used the published ratings from http://www.bordoverview.com/ to analyze 2004-2008 Bordeaux reviews (older data was not available on their website).  I found that the results showed, not surprisingly, that the two critics had a very low level of correlation with a “coefficient of determination” of only 21%.  The coefficient of determination is a statistical measure that shows the proportion of the variance (fluctuation) of one variable that is predictable from the other variable. Essentially, looking at Robert Parker’s rating won’t tell you much at all about what Jancis Robinson thought about the wine.

A much smaller rating scale would be more meaningful.

If experts can’t agree, and even have trouble rating the same wine consistently individually, doesn’t this just reinforce the subjective nature of wine and the multitude of variables that determine ‘enjoyment’?  It’s a hoax then to continue to rate at such minute levels of detail.  If the standard of accuracy for a professional wine critic is +/- 4 points, it would make much more sense to reduce the range of possible values.  Even though many 100 point scales go from 50-100, nearly all rated wines fall into the 80-100 range.  An eight point swing is an astonishing 40% of this range.  That leaves a lot of “wiggle-room”.

Does this inconsistency among critics mean that they should stop rating wine?  Of course not – critical input is a great aid to consumers when buying wine – especially when a consumer finds that a critic’s palate matches their own.  However, I believe a reduced-range rating scale would be much more valuable, such as the star system.  As stated by Joshua Greene, editor for Wine and Spirits magazine, “It is absurd for people to expect consistency in a taster’s ratings. We’re not robots.”

Times they are a-changin’.

The wine world has changed dramatically the last ten years with the advent of the internet.  Consumers are becoming better educated.  Through thousands of blogs, and many public wine review and rating websites, the general population now has a place to talk about wine and even get involved in reviewing, rating, and sharing experiences.

The secret is out that the 100 point scale is not ideal and the online wine world has already drastically changed how wine is rated and reviewed.  CellarTracker hosts a website for cellar management and has over one million consumer wine reviews.   Wine.com, one of the largest online wine stores, allows consumers to rate wine using a five star system.  Snooth is another large consumer wine review website, and uses a five “wine glass” system to rate wines.  We’ve built WineMcGee to allow thumbs up, thumbs down, or OK reviews.

As time goes on, I believe the increasing knowledge and maturity of the US wine population and the availability of multiple rating systems will ultimately lead to the demise of the 100 point system as we know it.  I believe the future will primarily bring smaller rating scales (1-10, 5 star, etc) and systems that aggregate multiple rating sources to form true percentage ratings out of 100 – similar to Rotten Tomatoes and Metacritic.  If a major player in the industry were to move away from 100 point ratings, it could be the beginning of change.  If not, the new faces in the industry will continue to evolve and change will still come.  It’s not a matter of if, but when.