Tag Archives: evaluation

Judging Coffee and Beer: Answer to DoubleShot Coffee Company

DoubleShot Coffee Company: More Espresso Arguments.

I’m not in the coffee biz but I do involve myself in some coffee-related things, including barista championships (sensory judge at regional and national) and numerous discussions with coffee artisans. In other words, I’m nobody important.

In a way, I “come from” the worlds of beer and coffee homebrewing. In coffee circles, I like to introduce myself as a homeroaster and blogger.

(I’m mostly an ethnographer, meaning that I do what we call “participant-observation” as both an insider and an outsider.)

There seem to be several disconnects in today’s coffee world, despite a lot of communication across the Globe. Between the huge coffee corporations and the “specialty coffee” crowd. Between coffee growers and coffee lovers. Between professional and home baristas. Even, sometimes, between baristas from different parts of the world.
None of it is very surprising. But it’s sometimes a bit sad to hear people talk past one another.

I realize nothing I say may really help. And it may all be misinterpreted. That’s all part of the way things go and I accept that.

In the world of barista champions and the so-called “Third Wave,” emotions seem particularly high. Part of it might have to do with the fact that so many people interact on a rather regular basis. Makes for a very interesting craft, in some ways. But also for rather tense moments.

About judging…
My experience isn’t that extensive. I’ve judged at the Canadian Eastern Regional BC twice and at the Canadian BC once.
Still, I did notice a few things.

One is that there can be a lot of camaraderie/collegiality among BC participants. This can have a lot of beneficial effects on the quality of coffee served in different places as well as on the quality of the café experience itself, long after the championships. A certain cohesiveness which may come from friendly competition can do a lot for the diversity of coffee scenes.

Another thing I’ve noticed is that it’s really easy to be fair, in judging using WBC regulations. It’s subjective in a very literal way since there’s tasting involved (tastebuds belong to the “subjects” of the sensory and head judges). But it simply has very little if anything to do with personal opinions, relationships, or “liking the person.” It’s remarkably easy to judge the performance, with a focus on what’s in the cup, as opposed to the person her-/himself or her/his values.

Sure, the championship setting is in many ways artificial and arbitrary. A little bit like rules for an organized sport. Or so many other contexts.

A competition like this has fairly little to do with what is likely to happen in “The Real World” (i.e., in a café). I might even say that applying a WBC-compatible in a café is likely to become a problem in many cases. A bit like working the lunch shift at a busy diner using ideas from the Iron Chef or getting into a street fight and using strict judo rules.

A while ago, I was working in French restaurants, as a «garde-manger» (assistant-chef). We often talked about (and I did meet a few) people who were just coming out of culinary institutes. In most cases, they were quite good at producing a good dish in true French cuisine style. But the consensus was that “they didn’t know how to work.”
People fresh out of culinary school didn’t really know how to handle a chaotic kitchen, order only the supplies required, pay attention to people’s tastes, adapt to differences in prices, etc. They could put up a good show and their dishes might have been exquisite. But they could also be overwhelmed with having to serve 60 customers in a regular shift or, indeed, not know what to do during a slow night. Restaurant owners weren’t that fond of hiring them, right away. They had to be “broken out” («rodés»).

Barista championships remind me of culinary institutes, in this way. Both can be useful in terms of skills, but experience is more diverse than that.

So, yes, WBC rules are probably artificial and arbitrary. But it’s easy to be remarkably consistent in applying these rules. And that should count for something. Just not for everythin.

Sure, you may get some differences between one judge and the other. But those differences aren’t that difficult to understand and I didn’t see that they tended to have to do with “preferences,” personal issues, or anything of the sort. From what I noticed while judging, you simply don’t pay attention to the same things as when you savour coffee. And that’s fine. Cupping coffee isn’t the same thing as drinking it, either.

In my (admittedly very limited) judging experience, emphasis was put on providing useful feedback. The points matter a lot, of course, but the main thing is that the points make sense in view of the comments. In a way, it’s to ensure calibration (“you say ‘excellent’ but put a ‘3,’ which one is more accurate?”) but it’s also about the goals of the judging process. The textual comments are a way to help the barista pay attention to certain things. “Constructive criticism” is one way to put it. But it’s more than that. It’s a way to get something started.

Several of the competitors I’ve seen do come to ask judges for clarifications and many of them seemed open to discussion. A few mostly wanted justification and may have felt slighted. But I mostly noticed a rather thoughtful process of debriefing.

Having said that, there are competitors who are surprised by differences between two judges’ scores. “But both shots came from the same portafilter!” “Well, yes, but if you look at the video, you’ll notice that coffee didn’t flow the same way in both cups.” There are also those who simply doubt judges, no matter what. Wonder if they respect people who drink their espresso…

Coming from the beer world, I also notice differences with beer. In the beer world, there isn’t really an equivalent to the WBC in the sense that professional beer brewers don’t typically have competitions. But amateur homebrewers do. And it’s much stricter than the WBC in terms of certification. It requires a lot of rote memorization, difficult exams (I helped proctor two), judging points, etc.

I’ve been a vocal critic of the Beer Judge Certification Program. There seems to be an idea, there, that you can make the process completely neutral and that the knowledge necessary to judge beers is solid and well-established. One problem is that this certification program focuses too much on a series of (over a hundred) “styles” which are more of a context-specific interpretation of beer diversity than a straightforward classification of possible beers.
Also, the one thing they want to avoid the most (basing their evaluation on taste preferences) still creeps in. It’s probably no coincidence that, at certain events, beers which were winning “Best of Show” tended to be big, assertive beers instead of very subtle ones. Beer judges don’t want to be human, but they may still end up acting like ones.

At the same time, while there’s a good deal of debate over beer competition results and such, there doesn’t seem to be exactly the same kind of tension as in barista championships. Homebrewers take their results to heart and they may yell at each other over their scores. But, somehow, I see much less of a fracture, “there” than “here.” Perhaps because the stakes are very low (it’s a hobby, not a livelihood). Perhaps because beer is so different from coffee. Or maybe because there isn’t a sense of “Us vs. Them”: brewers judging a competition often enter beer in that same competition (but in a separate category from the ones they judge).
Actually, the main difference may be that beer judges can literally only judge what’s in the bottle. They don’t observe the brewers practicing their craft (this happens weeks prior), they simply judge the product. In a specific condition. In many ways, it’s very unfair. But it can help brewers understand where something went wrong.

Now, I’m not saying the WBC should become like the BJCP. For one thing, it just wouldn’t work. And there’s already a lot of investment in the current WBC format. And I’m really not saying the BJCP is better than the WBC as an inspiration, since I actually prefer the WBC-style championships. But I sense that there’s something going on in the coffee world which has more to do with interpersonal relationships and “attitudes” than with what’s in the cup.

All this time, those of us who don’t make a living through coffee but still live it with passion may be left out. And we do our own things. We may listen to coffee podcasts, witness personal conflicts between café owners, hear rants about the state of the “industry,” and visit a variety of cafés.
Yet, slowly but surely, we’re making our own way through coffee. Exploring its diversity, experimenting with different brewing methods, interacting with diverse people involved, even taking trips “to origin”…

Coffee is what unites us.

Taste and Judgement (Draft)

This post isn’t ready to be written. So this is just a placeholder. But, given my RERO mantra, I guess I should still publish it as a “placeholder” of sorts.

I recently served as judge in the Canadian Barista Championship (CBC), here in Montreal. That championship is the national competition pitting against one another baristas (espresso artists) from regional competitions. Rules and regulations (PDF) for this championship closely follow those set by the World Barista Championship (WBC).

Participating in this event, I got to think about taste, evaluation, (inter)subjectivity, coffee, Montreal’s culinary scene, and food culture generally.

Some videos from the event are available, through the event organizer’s uStream channel.

The event was blogged by Anthony Benda. Despite being busy preparing for his café’s grand opening, Anthony managed to give an excellent performance during the championship, especially on the first day. I wasn’t on the panel of judge for his performance but I have reason to believe that Anthony’s performance was really quite good.

I also got to think about my own involvement in such events.

Being a judge at barista championships is still somewhat new to me. I judged during the Eastern regional championship, back in June, and this was my first national championship. I still think that the “barista judge” label fits and I did mention it on occasion, with lots of disclaimers. Most judges at that event were coffee professionals of one type or the other (from equipment distributors to barista champions). My impression is that, despite my limited experience and my somewhat indirect connections to the coffee industry, I was accepted as a peer by other judges.

More importantly, I sincerely think that the judging at this competition was exceedingly fair. My strong perception is that we achieved a high degree of consistency in our judging, both at an individual level and through the group. A large part of what I perceive to be a resounding success comes from the work of WBC’s Brent Fortune, who trained and calibrated the judging team.

One thing I kept thinking about was how different barista judging is from judging homebrewed beer. I haven’t acted as a homebrew judge myself but many of my friends have and I proctored an exam for the Beer Judge Certification Program (BJCP). Put simply, homebrew competitions are stricter than barista championships. And I mean this to imply that BJCP competitions are in some ways less reasonable than barista championships, though the WBC could learn a thing or two from the BJCP.

Barista championships are based on fairness and impartiality. Though it’s mentioned on occasion, “objectivity” isn’t the core principle in judging. Not emphasizing “objectivity” allows for a sensible approach to tasting since, after all, tasting is as subjective as any other form of sensory perception. In other words, acknowledging the subjective nature of tasting brings realism to WBC-style competitions.

The reason I emphasize the “subjectivity” issue is that homebrewing competitions seem to exist in a radically different world, a world in which “objectivity” is an absolute goal. Though the BJCP style guidelines may allow for some room for variation in judging beer aroma, appearance, flavour, and mouthfeel, the main approach is object-based and a direct connection between a judge’s experience and precise measurements of the beer’s characteristics is assumed. Several homebrew judges do use the term “objective” fairly frequently and “subjectivity” is a “bad word” in many of the homebrew circles which give a lot of weight to those homebrew competitions.

One thing I find fascinating about this distinction between WBC and BJCP competitions is that, to some extent, the coffee professionals are less, well, “anal” than the homebrewers. The level of technical expertise may be as high in both domains. The drinks themselves are comparable on many levels, including in terms of chemical complexity. But the approaches taken to evaluate those drinks are radically different.

I also got to think about the connections (actual and potential) between Montreal’s strong beer scene and its renascent coffee scene. It certainly was fun to have beers at Benelux with a number of participants in the Canadian Barista Championship, including coffee writer Felipe Gonzalez. Myriade’s grand opening, on Monday, will likely serve as an opportunity for me to discuss Montreal’s coffee scene in more depth.

Of course, any of this could be the start of a long monologue on my part. But it’s probably better if I leave this post as it is, to serve as a placeholder for further discussion of taste, evaluation, subjectivity, coffee scenes