The Agony and Ecstasy of Beer Reviews
From the Thoughtful to the Troll, Every Beer Gets an Opinion
When Scott Smith, founder and owner of East End Brewing in Pittsburgh, started selling his beer more than a decade ago, rating websites like BeerAdvocate and RateBeer were valuable resources for his young brewery. In the pre-social-media era, the forums on the sites provided insightful access to consumers, and reviews provided a “first line” for feedback about inconsistencies and quality issues.
As East End has grown and established quality-control procedures in-house, and now that customers can interact directly with the brewery via Twitter and Facebook, he doesn’t check rating websites, which now include Untappd, as frequently for feedback. He also stays away for another reason many brewers can identify with: criticism, gossip, and commentary from the beer Internet’s peanut gallery.
“The beer websites, in particular the forum community on BeerAdvocate, can be pretty toxic, and I use that term with some reluctance, but I find that’s the case,” says Smith. “I see horrible things written about brewers that I know, beers that I know, by random, anonymous people. The Internet is certainly filled with trolls, and it seems like the forums on BeerAdvocate have more than its per capita allotment.”
Indeed, for brewers who have invested significant time, energy and money into making a beer, scrolling through reviews that dismiss it as “meh” and forum threads about overhyped or overrated breweries can be exasperating.
“You sweat and you work and you pour your life into something and then you get like, ‘Eh, I dumped it.’ You get these reviews that have no information; you can’t even really tell if anybody actually tasted it. There’s definitely people posting on there who post a lot more thoughtfully and offer more descriptions, but it’s pretty tough,” says Randy Mosher, author of Tasting Beer and a partner in two Chicago breweries, 5 Rabbit Cerveceria and Forbidden Root. [Mosher is also a columnist for All About Beer Magazine.]
Online beer communities and reviewing sites can also feel like “a detached little bubble compared to the rest of the beer world,” says Mosher, in which imperial IPAs, lambics and barrel-aged stouts reign at the top of the list and members obsess over tracking down rare beers. While he tracks what is being said about his beers online, Mosher says he prefers semi-professional or professional feedback from competitions that are judged blindly or from a more traditional feedback mechanism.
“In a perfect world, we do get feedback from consumers: They buy our beer. They wear our shirts, buy our beer, and we thrive in the marketplace. So that really is the kind of feedback we love to have,” says Mosher.
A Part of the Drinking Experience
Google the name of a beer and review site links will likely show up in the first five search results. The scores are posted on shelf tags at stores like Whole Foods or local beer shops and on beer bar menus. It’s not uncommon to see savvy consumers stand in front of rows of bombers and six-packs, smartphone in hand, checking a beer’s score and scrolling through reviews.
Before that beer may even end up on a shelf, the store’s beer buyer might look up the beer online before placing his order with the distributor. And all these numbers are crunched and compiled into lists that are picked up by local and national media, touting the best beer in the world or the best new brewery in the world, drawing beer lovers to seek out beers that everyone’s seemingly talking about.
Consumer beer rating websites have democratized beer reviewing and given license to hundreds of thousands of drinkers to log on, share their opinion and assign a beer a numerical rating. That power has also shifted the relationship between brewery and consumer and changed the discourse of beer culture.
“Beer Drinkers With Opinions”
Todd Alström logged the first online beer review for BrewGuide.com, what would become BeerAdvocate.com, on Aug. 22, 1996. It was for a pint of Berkshire Brewing Co. Steel Rail Extra Pale Ale, served at the Eastside Grill in Northampton, Massachusetts. He rated it 3.85 out of 5 and described it as “a very light ale, with an extremely refreshing amount of carbonation.”
In the 19 years since that first review, 582 other reviewers have logged on to rate Steel Rail Extra Pale Ale, and BeerAdvocate has amassed more than 10 million ratings and reviews. Alström, who was developing video games and tinkering with computers from a young age, and his similarly technically inclined brother Jason Alström originally built the website as a place to post their reviews of beers. They later added forums for users to talk about beer, and in 2000 they rebranded as BeerAdvocate and introduced functionality that allowed the growing user base to review beers by attribute—appearance, aroma, taste and feel—and give the beer a rating.
Since then, they’ve experimented with attribute weights, flexible rating styles and quick ratings, with single-score “ticking” and no attributes or text. Recently they returned to their roots with attribute-based ratings that “encourage users to think about the beer and share their experience with the rest of the community with a thoughtful review,” says Todd Alström. The rating system in place today is based on the Beer Judge Certification Program (BJCP) score sheet: Appearance is 6 percent, smell is 24 percent, taste is 40 percent, mouthfeel is 10 percent, and overall is 20 percent.
He doesn’t foresee any other major updates to the rating system, but they are highly considering adding some sort of commenting system that would allow breweries or business owners to claim their page and communicate more directly with reviewers. “There are no beer experts, just beer drinkers with opinions’ is an old saying that Jason and I coined and displayed on the site in the early days of introducing user reviews. It was our way of telling our users that their opinion was just as important as the so-called pros, which we still believe to this day,” says Alström via email.
In recent years there have been new entrants to the beer review arena, most notably the mobile app Untappd, which launched in 2010 and counts more than 2.1 millions users. With the software, users have the option to rate a beer from one-half to five stars and can dash off a few notes or thoughts. Check-ins from friends appear in a newsfeed stream, allowing users to see who is drinking what and where and to “toast” it or comment on the check-in. “There has to be some sort of rating system for any type of beer just to know if it’s good or not, just like you have Rotten Tomatoes for movies or IMDB,” says Greg Avola, co-founder of Untappd. “But I firmly believe that the best recommendations come from your friends, and that’s what we try to focus a lot on at Untappd. That’s an important part that gets overlooked when it comes to ratings of beer, and that’s what we’re all about in terms of discovery.”
When BeerAdvocate was launched, the site allowed its growing user base to review a beer by attribute (look, smell, taste and feel), and give it a rating. A similar philosophy guided RateBeer, a review site founded by Bill Buchanan and now run by executive director Joe Tucker, who discovered the site in 2000.
Tucker, who was developing an interest in wine at that time, had seen that reviews from prolific wine critic Robert Parker could make or break a wine or winery, and found the process inherently flawed. “I knew from my physiological psych background that that was ridiculous because we all have different palates, we all have different aroma sensitivities, those chemical senses that we have are varied in humans and so one person’s taste isn’t another person’s taste, physiologically. So to have one source saying, ‘Yeah, this is good,’ or ‘This is bad,’ is completely preposterous,” says Tucker.
Instead, RateBeer made sense: a community powered by thousands of palates and ratings and tastes. Today the site has 357,000 active members and 6.24 million reviews and wields significant influence.
Tucker and RateBeer experienced that influence in 2005 when RateBeer named Westvleteren 12, a quadrupel ale made by the small Trappist brewery at St. Sixtus abbey in Vleteren, Belgium, the best beer in the world. The ranking was picked up by major news outlets like CNN, which drove drinkers to the brewery and drew attention to a small brewery in a quiet area that wasn’t quite prepared for it, says beer writer Tim Webb, co-author of Good Beer Guide Belgium and Pocket Beer Guide.
“This abbey is in a quiet area of West Flanders; there’s little country lanes going to it; the nearest town is about two or three kilometers away. It’s supposed to be a very quiet place where you can just plow the fields and have some contemplation,” says Webb. After the announcement, lines of eager beer hunters formed daily outside the abbey, and the Belgian media reported fistfights in line. Meanwhile, the brewery couldn’t even begin to keep up with the newfound demand.
“That was a good kick in the pants,” says Tucker. “It really got us to think about our math and the way we were delivering our information, the way we were putting together our list, the way we were notifying people about it.” Now, they try to be a little more gentle, Tucker says. He learned to talk to the media to explain what the honors mean and adjusted the math to decrease the volatility of ratings and stabilize the rankings, which are compiled from user reviews.
Arizona Wilderness Brewing Co. stands in a plaza along a busy six-lane road in Gilbert, Arizona. These days, construction crews are working alongside brewers as the 2-year-old brewpub is going through a sooner-than-expected expansion.
Jonathan Buford opened the brewery in September 2013 with partners Brett Dettler, the business manager, and brewmaster Patrick Ware. In its first year, Arizona Wilderness brewed about 100 different beers and highlighted ingredients grown by local farmers: herbs, spices and citrus from an urban farm in Gilbert and a heritage grain called Sonoran White Wheat that it uses in Belgian styles and Berliner weisses. That experimentation, frequent turnover of new beers and the fact that the beers were only sold on premise created a buzz locally and drove drinkers to check out what was happening in Gilbert.
Five months after opening, RateBeer released its annual Top New Brewers in the World list. The list, compiled from user ratings of beers, ranked the top 10 highest-rated new breweries. Out of more than 2,600 breweries that opened in 2013, Arizona Wilderness was named the best new brewer in the world.
“We weren’t prepared for what was coming next, which was just a massive amount of response of people wanting to see why we were called this,” says Buford. “Effectively, it catapulted us into a new realm of growth that we had never planned for. It’s like cutting your legs off and learning how to walk. You can’t learn quickly enough.”
It forced them to skip from year one to year five of their business plan in one month to meet demand, he says. They scrambled to keep beers on tap, added a few more 15-barrel fermenters to expand capacity, beefed up their staff and hired an experienced brewpub general manager, Matt McCormack. After nine months, they sought to expand with a new brewery next door, scheduled to open in spring 2015, which will have the capacity to produce about 2,200 barrels annually (they brewed 1,000 barrels in their first year) and includes a bigger kitchen and a humidity- and temperature-controlled barrel-aging and sour beer room.
The Hype Factor
The most difficult piece to nail down numerically or theoretically in beer ratings may be the “hype factor”—the buzz that surrounds a brewery and the expectations drinkers might set for a certain beer or brand. Buford reflects on the pressure to meet demands and the opportunities that it afforded them. “It felt like a bottomless pit we never would get out of,” he says of the crush after the title. “A year later, boy, are we having fun again. We’re being ourselves. [We’re] not so worried what people want from us, think about us, expect from us.”
The title itself, while an honor, has always felt a bit soulless anyway, he says. They never used it in advertising or put it on beer labels or their website. And what Buford has always been most proud of is not that their brewery was named the best in the world, but that they built a brewery in Gilbert. “Really, what I want to contribute to our hype is that we have brought culture to an area that was dead of culture,” says Buford.
“The most valuable piece for me is going to be a validated panel backed up with some kind of analytical equipment,” says Matt Brynildson, brewmaster of Firestone Walker Brewing Co. “That’s not what you’re going to find online. That’s a completely different type of feedback mechanism.”
Once a beer leaves the brewery, it often faces variables often out of the brewers’ control: how it’s stored, how it’s poured into a glass and at what temperature, how long a drinker stores a bottle, or whether the beer travels out of its intended footprint in the hands of a beer trader.
Compare that with the Great American Beer Festival or the World Beer Cup, where a brewer ships his beer directly to be tasted blind by a validated panel of judges, typically industry professionals, and only against other beers of the same style.
It’s a similar practice at the Beverage Testing Institute (BTI) in Chicago. There, panelists judge only 25-35 beers in one session to prevent palate fatigue. They taste at the same time every day, usually 10 a.m. to 1 p.m., in a temperature-controlled room at 68 degrees, outfitted for maximum natural light. The beer is chilled overnight and served between 53 and 58 degrees in Riedel stemware. Panelists taste all products blind and do not know any of the brands or prices—just the category they are tasting that day—and scores are translated to a 100-point scale. Rankings from BTI appeared in this magazine for many years.
While beer consumers may just as carefully smell, taste and critique a beer, an online panel is more dissonant than harmonized. That means diverse opinions and a slew of feedback for a brewery to sift through, says Jeremy Danner of Boulevard Brewing Co. As a full-time brewer turned ambassador brewer, Danner works between the brewing and marketing departments, runs much of the brewery’s social media and scans online forums and rating websites. “There’s two mindsets in that breweries can care or they cannot care about what people are saying online. I think it’s about being able to have a critical eye to sift through what reviews or blogs or opinions you really find valuable or influential,” he says.
In a perfect world, Danner says, beers would be rated on objective quality and execution of intentions versus personal preference.
“I think the best and most fair way to rate a beer is based on the intentions of the brewer. Our pilsner [KC Pils] is the second-lowest-rated beer on Untappd, and it is a really good beer and 100 percent of what we intended it to be. But because it’s not an exciting or extreme, crazy beer, it suffers in reviews,” says Danner.
Words Can Hurt
More painful than the sting of a negative review is the fallout from rumors and incorrect information about beers, which can be dangerous for a brewery’s reputation. Discrepancies range from the minor—Smith of East End Brewing says that its 3.8% ABV Southern English Nut Brown Ale was listed as 8.0% on a rating site—to the more serious, like claims of a beer being infected.
Cory King, founder of St. Louis’ Side Project Brewing, experienced the latter a year and a half ago. The rumor started when several reviewers on BeerAdvocate posted that they thought Side Project’s The Origin, a hefty blended dark beer aged in rye barrels with vanilla beans, was infected, describing the off-flavor of buttered popcorn, which is indicative of diacetyl. King had the beer tested at nearby Schlafly Beer. The results came back negative for infection, but the damage was done.
“Even though its [sic] not infected, it sucks that all it takes is one person to post it online and now people are questioning it. I can defend myself and my beers on here, but I cannot reply to people reviewing The Origin as infected, and all it takes is one palate to prematurely cry infection to cause a whole stir among the beer community of the other forums,” King wrote on the beer website STL Hops’ forum.
King’s frustration highlights an inherent problem with any online beer review: If it’s not possible to know any reviewer’s taste or palate, how can you trust his or her opinion of a beer?
Jake Austin, founder of Austin Street Brewery in Portland, Maine, says he stays off Untappd for that reason.
“I avoid Untappd [because] it just seems like an easy way to spout off about beer, but you don’t actually know how well-educated they are; you don’t know their palate, what they like for beers,” says Austin. “I think you have to take Untappd with a gigantic grain of salt as far as judging what you want to drink.”
Yet Austin still checks BeerAdvocate and RateBeer reviews about once a week, which he feels provides greater context about the drinker and more in-depth reviews. As a 1-year-old brewery, he wants to make sure his opinion of the beer is matching up with consumers’ thoughts. So far, he says, things have been lining up perfectly.
Heather Vandenengel talks about beer on the Internet on Twitter at @heathervandy.