Blog

Thoughts on Jim Murray’s Whisky Bible 2010

At the Finest Spirits Festival in Munich last weekend I had the chance to have a few words with Jim Murray who had been placed in a rather forlorn corner of the basement. I took the chance and bought one of his Whisky Bibles which he was kind enough to sign for me. Unfortunatlely I did not have the time to attend his “Jim Murray Gives it to You Straight” masterclass scheduled very late at night.

So far I had only read second-hand reports about his ratings for some particular whiskies, most notably his repeated choice for an Ardbeg as Scotch Whisky of the Year. This move had provoked a lot of criticism in whisky world – especially on the internet in blog posts, forum threads and Twitter tweets – culminating in accusations that Jim Murray was on Ardbeg’s or LVMH’s payroll. I have to admit that I wasn’t immune to such thoughts myself. But I accepted Jim’s blog comments denying any affiliation and that the reason was merely that he liked Ardbeg very much.

Now that I have had a thorough look through the book, I am assured that his explanation was genuine. There are so many other whiskies with extremely high scores from many other distilleries that it is hard to believe that any money was involved in this.

The High Score Mystery

But this plethora of high scores has left me very thoughtful nevertheless. So many whiskies have extremely high scores – including some quite simple ones – that I am having a have a hard time to align my own impressions with those of Jim Murray. I know I am not alone with this feeling because Jim tackles this issue in his book where he admits being critizied for marking too highly. He defends the fact that 50 out of 950 score 95 or better, stating that this percentage of about 5% “sounds about right” to him.

I’m afraid I have to disagree on this. If you take culinary guides like Michelin or Gault Millau that rate restaurants, 95+ scores for whisky would be comparable to the top restaurants with three Michelin stars or a Gault Millau rating of 19 or 19.5 out of 20. Now would you expect 5% of all restaurants to be temples of ultimate culinary delight? Certainly not. Paris or London have thousands of restaurants, but only a handful have top scores. Most are not even considered to be included in these guides because they are just not good enough. The rate of top restaurants will probably be around one in a thousand, if at all. The underlying statistical law is the Gaussian distribution with a big bulge at the median value and very flat tails at both extremes.

The Whisky Bible takes a holistic approach and wants to list as many whiskies as possible. So we can expect a statisically relevant cross-section. 5% of all whiskies in absolute top positions would mean that the overall quality of whisky is extremely much higher than the quality of restaurant food. Is this really true?

The Dangers of the Compound Score

I think the answer to this mystery can be found by looking closer at Jim Murray’s approach to rating whisky. Jim’s ratings are diveded in four sub-scores for nose, palate, finish and balance. Each component is scored separately on a 25 point scale and then added together. His marks for each of the components are 20+ most of the times. So whenever a whisky is at least “decent” in Jim’s opinon, he is left with only 6 marks for each component.

This is effectively a five star rating which he strongly disapproves as a rating system for whisky in the introduction to the book as being too inaccurate, and rightly so. This is indirectly proven true by the introduction of half points because with only full points there would be too litte possibility to fine-tune a score.

And there are three more important things to consider with Jim Murray’s compound ratings:

1. The “Balance” score is not independent from the other three. In the author’s own words: “For a whisky to work well on the nose and palate, it should not be too one-sided in its character.”

   If this is to be taken seriously, an unbalanced whisky will never be able to receive top marks for nose and palate (and certainly also finish). So in a way, this fourth score only pretends to give additional information.

2. Because it’s a human who does the scoring and not a gas chromatograph, there is always a margin of error involved. Sampled on another day, any whisky might get a point more or less. Four scores mean four margins of error. 21/19/22/22 on day A could well give 20/18/21/21 on day B or 22/20/23/23 on day C. The resulting score would be anywhere between 80 and 88.
3. All four sub-scores are weighted equally. Does this hold true for everyone? I for one tend to weigh the nose far less than palate and finish. But in my experience, more whiskies have a better nose than palate than vice versa. And dow does the calculation take into account that the balance score is connected with the others?

Where is the Calibration mark?

The “continuous” 100 point rating system I adopted from the Malt Maniacs and that was also used by Michael Jackson to my knowledge (correct me if I’m wrong) uses 75 points as a sort of calibration mark. This would be a run-of-the-mill whisky that you can’t really call bad but that has nothing to write home about either.  When scoring a whisky using this system, the final score will be an assessment on how far away it is from that generic 75 point dram.

For a compound score to work well, you would need a calibration mark for each of the sub-scores. I would never accuse Jim Murray of not having some sort of neutral point for his scoring. But I tend to think that it is placed rather at a sub-score of 20 than at 18.75 (75/4). This would explain why his ratings are often significantly higher than other people’s.

But perhaps Jim’s ratings just reflect that he is generous with his scores because he likes whisky in general so much. Which is not a bad thing at all, mind you.

Does 95 Alwyas Equal 95?

But there is yet another thing that struck me browsing the Whisky Bible. A simple Irish blend like Jameson receives a score of 95 and – even more astonishing – the Swissky Exkusiv-Abfüllung from Switzerland gets 94.

What would Jim Murray say if presented with a head-to-head sampling of a Jameson and lets say an Ardbeg Airigh nam Beist that he scored 95 points as well. Or how about the Swissky and a Glenfarclas 17 (93 points)?

I can’t help thinking that these scores are not comparable. Coming back to the restaurant guide example, I guess giving the Jameson 95 points means something like “Here you can get the best Fish & Chips in town” while the Ardbeg score compares with the two star restaurant serving truffled lobster mousse with scallops.

Is this Book any Good Then?

Yes! The Whisky Bible is a great collection of tasting notes and scores for thousands of whiskies, and you really have to love Jim Murray’s writing style.

But you should be aware, that these are just one person’s opinions, worth as much or as little as any other’s. Just because a writer has a big name, it does not mean that you will like what he likes. I have found out that Jim Murray’s ratings have a very erratic correlation with my own, but this does not keep me from enjoying his book.

{ 5 comments }