Previously on Data Chug, we took a broad look at the Top 250 beers listed on the beer ranking website BeerAdvocate. We highlighted the top 10 beers, identified popular beer styles, ABV trends, the countries and states with the most beers in the Top 250, and more.
There were charts, jokes, and a cliffhanger ending that revealed two breweries that appeared a rather curious amount of times in the Top 250.
Today, we pick up where we left off by taking a closer look at what we affectionately refer to as “the outlier breweries.”
Now on with the Q & A…
The Outlier Breweries
Q1. Last time you mentioned that there were 210,023 different beers listed on BeerAdvocate, so to have a brewery with even a single beer in the Top 250 was impressive. But there were actually several breweries that had more than one beer in the Top 250. What can you tell me about those breweries?
A1. Right, so there were 86 breweries in total in the Top 250, and 45 of those breweries had more than one beer on the list. In fact the average number of beers per brewery in the Top 250 was 2.91, so it wasn’t all that unusual for a brewery to have more than one beer in the Top 250.
However, what was unusual was that two breweries, namely Trillium Brewing Co. and Hill Farmstead Brewery, both had exactly 20 beers each in the Top 250. This means that nearly 1 out of every 6 beers in the Top 250 was either from Trillium or Hill Farmstead. If having one beer in the Top 250 is impressive, what do you call having 20?
Q2. Really impressive? I don’t know— it seems a little bit more than unusual to me. But Trillium and Hill Farmstead weren’t the only outlier breweries with a statistically high number of beers in the Top 250, so what were the other ones?
A2. We identified a total of 8 outlier breweries. Basically any brewery with more than 6 beers in the Top 250 would be considered a statistical outlier as you can see in the box plot graph below:
Of the outlier breweries, there were four that had 7 beers each in the Top 250 which included Russian River, Toppling Goliath, 3 Fonteinen, and Prairie Artisan Ales. Next, you had Brasserie Cantillon with 8, Tree House with 13, and as you know, both Hill Farmstead and Trillium both had 20 beers each in the Top 250. By the way, 3 Fonteinen and Cantillon are both Belgian breweries, the rest are American.
Interestingly, even though there are only 8 outlier breweries out of a total of 86 breweries in the Top 250, those 8 breweries contributed a combined total of 89 beers to the Top 250, or 35.60%.
Q3. Okay, so there are 8 outlier breweries. But still, having 20 beers in the Top 250 like Hill Farmstead and Trillium did just seems crazy. So what’s going on with those breweries?
A3. Alright, so imagine that Hill Farmstead and Trillium only produced 20 beers per brewery, and also imagine that every single one of those beers were in the Top 250. Basically, we thought it would be pretty strange if 100% of the beers brewed by any given brewery were all in the Top 250.
Q4. So that’s what you found? Every beer brewed by Hill Farmstead and Trillium were in the Top 250?
A4. Not exactly. After we looked at the total number of beers currently brewed by each outlier brewery, we found there wasn’t anything necessarily unusual about either Hill Farmstead or Trillium when compared to most of the other outlier breweries.
For example, when we pulled the numbers in mid-February of 2018, Hill Farmstead was currently producing 124 different beers, and of those 124, 20 were in the Top 250. In other words, 16.1% of the beers currently produced by Hill Farmstead were in the Top 250.
Trillium was at 14.9%. As for the rest of the outliers, 3 Fonteinen was at 26.9%, Cantillon at 22.86%, Tree House was at 20.96%, Toppling Goliath at 14.28%, Russian River at 11.67%, and Prairie Artisan Ales was relatively low in comparison at 4.7%. If you look at it this way, both Trillium and Hill Farmstead weren’t sticking out, but were right in the middle of the pack of the outliers. The average percentage of beers contributed per brewery of the outliers was 16.54%.
Q5. So what you’re saying is that it’s just a numbers game? Basically, the more variety of beers that a brewery produces, the more that the chances increase that one of those beers will make it into the Top 250? If that’s true, then those percentages that the outlier breweries were putting up would be similar for every brewery, including the non-outliers in the Top 250. So what do the percentages for the non-outlier breweries look like?
A6. Okay, so there were initially 8 outlier breweries identified out of a total of 86 breweries in the Top 250, which means there were 78 non-outlier breweries.
The Lost Abbey was one of the non-outliers, producing a total of 57 beers, 2 of which appeared in the Top 250. In other words, 3.5% of the beers The Lost Abbey produced were in the Top 250. Likewise, Avery was at 1.7%, Deschutes at 0.56%, Firestone at 3.2%, Other Half Brewing Co. was at 2.02%, etc.
After chugging the numbers, it turned out that an outlier brewery was now any brewery that had more than 9.79% of its beers it currently produced in the Top 250.
Q7. Wait, so where does that leave Prairie Artisan Ales? It was originally an outlier brewery because it had 7 beers in the Top 250. But because those 7 beers represented only 4.7% of the brewery’s total production, it’s not considered an outlier anymore?
A7. That’s right. However, looking at the numbers relative to the quantity of different beers produced per brewery just reinforced the fact that the other outlier breweries were unusual, and appearing in the Top 250 wasn’t just a numbers game. In other words, greatness in the brewing world isn’t simply a matter of throwing enough stuff at the wall until something sticks, at least not according to an analysis of BeerAdvocate’s data.
What’s more, we identified 4 new outlier breweries that had more than 9.79% of the beers they currently produced appearing in the Top 250. Those breweries were Alpine Beer Company at 10.71%, Brouwerij Rodenbach N.V. at 12.5%, Brasserie de Rochefort at 33.33%, and Brouwerij Westvleteren at a staggering 66.66%. Of the now 11 outlier breweries, 5 were Belgian, and the other 6 were American.
Q8. Alright, these percentages just seem insane. I mean, is there some conspiracy type stuff going on? What does all this mean?
A8. Ah, now there’s a question. In statistical analysis, sometimes the juiciest information can be obtained by looking more closely at the outliers.
In this case, if you take the data from BeerAdvocate at face value, it could mean that the outlier breweries like Tree House, Toppling Goliath, Trillium, Hill Farmstead, etc. are simply out-preforming all other breweries in the world at a statistically significant level, with respect to specific beer styles.
There’s certainly a case to be made that a brewery like Cantillon, founded in 1900, simply specialized and excelled in a limited number of beer styles over many years, and became a world leader par excellence in those specific and relatively sparsely produced categories like Gueuze and Lambic, which are the only kind of beers Cantillon makes. 8 out of 35 beers from Cantillon appear in the Top 250.
This pattern is similar for the other Belgian outlier breweries such as Brouwerij Westvleteren, a brewery that at first glance seems remarkably unusual with 66.66% of its beers appearing in the Top 250. However, we should also consider that the brewery, founded in 1838, only currently produces 3 beers in total, 2 of which were in the Top 250. One of those beers, Westvleteren 12, was rated a perfect 100/100 by RateBeer, and is considered by some sources to be the best beer in the world.
You may include Rochefort (founded 1595), Rodenbach (1821), and 3 Fonteinen (1887) into that list of Belgian breweries that produce a relatively small variety of beers, but have perfected those beers over generations, and are consistently recognized as world-class breweries, often epitomizing the beers styles they produce.
Similar to the production range of Brouwerij Westvleteren, Rochefort only produces 3 beers, 1 of which (or 33%) was in the Top 250. Rodenbach essentially invented and mastered the Flanders Red style, the only style of beer it brews, so it wasn’t surprising to see 1 of the 8 beers it produces (Caractère Rouge) in the Top 250. Likewise, 3 Fonteinen specializes predominately in Lambic and Gueuze, and contributed 7 of its 26 beers to the Top 250.
Q9. Okay, maybe that explains the Belgian breweries somewhat, but what about the American outliers?
A9. Unlike the Belgian outlier breweries that typically only brew a small variety of beer styles, most of the American outliers brew a wider range, but they tend to specialize or excel at one particular beer style.
For example, the majority of beers contributed by Alpine (founded 1999), Tree House (2012), and Trillium (2013) in the Top 250 are hop-forward beer styles.
Russian River, founded in 1997, is widely known for its Pliny the Elder and Pliny the Younger double and triple IPAs, both of which appear in the Top 250 and were also rated 100/100 by RateBeer. In addition to mastering the double IPA, a beer style the brewery is credited with inventing, Russian River was one of the first American breweries to specialize in sour and barrel-aged beers, and led in the American Wild Ale category with 5 beers in the Top 250.
Toppling Goliath (founded in 2009) is a bit of a mixed bag with 3 examples of imperial stout in the Top 250, along with 4 examples of hop-forward styles. Likewise, Hill Farmstead (founded 2010) has 12 hop-forward styles in the Top 250, and the rest are a mix of saisons, stouts and a single porter.
But even though Hill Farmstead produces a variety of beers compared to most of the other outliers, it was awarded the title of “Best New Brewery in the World” by RateBeer in 2011, and “Best Brewery in the World” every year from 2013-2017.
Q10. Interesting, but when I looked at BeerAdvocate’s list of all American IPAs and there were literally beers there with PERFECT scores. None of those perfect scoring beers appeared in the Top 250, so what gives?
A10. You’re correct, you might find several beers with perfect scores listed under many different beer style categories, but many of those perfect scores came from only one reviewer. The fact is that BeerAdvocate’s ranking algorithm weights a beer’s score based on the number of reviews the beer has had, which is why those higher rated beers don’t appear in the Top 250.
What this means is that there very well could be and probably are better commercial examples of certain beer styles out there than those that appear in the Top 250, but it may take a little while until more of the BeerAdvocate user base has a chance to weigh in on those beers. What this suggests is that the many of the current outlier breweries will not be there forever.
Nonetheless, we should probably expect to see some of the Belgian outlier breweries in the Top 250 for a while, particularly those breweries producing time-intensive beers styles like Lambic, Gueuze and Flanders Red. These beer styles often take upwards of three years to produce, not to mention the perhaps generations of knowledge needed to perfect those styles. All of this makes it cost-prohibitive for other breweries to attempt, which limits the respective competition.
For now, what we can say with certainty is that there are a relatively small number of breweries on BeerAdvocate’s Top 250 that are contributing a statistically significant number of beers to that list.
And outliers typically demand attention especially in situations where it’s clear that the outliers were not due to simple data collection errors. By the way, the data we used is open to the public to independently verify. We’ve even made it easy to sort the data by state, country, brewery, score, etc. In other words, feel free to chug the numbers yourself. Here’s the link: BeerAdvocate’s Top 250.
Q11. You know, we spent all this time talking about the “Top 250 beers,” but did you ever consider the source of that data? What I mean is that if the user base is predominately English-speaking Americans, doesn’t that skew the data in favor of American breweries?
A11. You’re not wrong. I wonder, is that the same as saying “you’re not not not wrong?” Anyways, what I mean is that you’re right, or not not right.
Basically, it’s more likely that the results of the beers in the BeerAdvocate’s Top 250 had less to do with a conspiracy, and more to do with the demographics of the website’s user base and how the algorithm functions. Even so, outliers still exist within this particular data set, and statistically speaking, those outlier breweries deserve a spotlight.
Of course, interpretation of that data can be tricky.
Q12. Okay, so you’re admitting that BeerAdvocate’s Top 250 is just one data set and it may be limited by certain factors including its user base which you speculate is dominated by American English-speakers. Rarity and hype might also be factors. Come to think of it, if you’re trying to independently verify if there is anything special about those outlier breweries you mentioned, don’t you think you should look more thoroughly at other data sets?
Like, did you even think to analyze Untappd’s list of Top Rated beers to see how it compares to BeerAdvocate’s list?
A12. “Untappd” you say? You mean the geosocial networking service and mobile phone application that allows its users to rate beers and earn digital badges to increase the users average time spent on the app? That Untappd? Why, yes, we did think to analyze that data.
In fact, we did more than just think about it. We pulled the data, chopped it up, ran a statistically analysis, and put it all together just for you.
Q13. Alright, let’s have it. Anything juicy?
A13. Oh, it’s juicy alright— juicy like a big hazy NE IPA. Let me see, I know I put that article somewhere for you…
Ah, there it is:
Next on Tap… Data Chug: Analysis of Untappd’s Top Rated Beers
Hi, I’m Dan: Beer Editor for Beer Syndicate, Beer and Drinking Blogger, Beer Judge, Gold Medal-Winning Homebrewer, Beer Reviewer, American Homebrewers Association Member, Shameless Beer Promoter, and Beer Traveler.
Dan with Owner and Brewer Jean-Pierre van Roy of Brasserie Cantillon Seen Sporting a Hill Farmstead T-Shirt.
