The Artisans Cup Judging

Posted by on Oct 4, 2015 | 27 Comments
The Artisans Cup Judging

In their Sunday afternoon panel discussion Ryan and Chelsea Neil expressed a desire for creating professional standards in bonsai and for consistency and transparency in judging of bonsai exhibits. To that end, the judges at The Artisans Cup were separated during judging so they would not influence each other and they were informed in advance that the raw scores for all judges would be published on The Artisans Cup blog.

Bay Island Bonsai members and Intensive Students who study with Boon Manakitivipart are all familiar and practiced in scoring trees on a 30 point scale. The purpose of the exercise is to teach a method for evaluating each aspect of a tree against other trees of the same species or of similar character. Boon’s rubric emphasizes the importance of the trunk by awarding 1-10 points, the branching is scored 1-5, the nebari receives 1-5 points and the container merits a similar 1-5 score. The last five potential points are scored on the overall impression, giving a judge the chance to score higher or lower for trees that may be unusual but still merit consideration.

The rubric for the judges as published on The Artisans Cup blog uses a single score 0-60. While the instructions are extensive for aspects of the tree and display composition that should be evaluated, no specific guidelines for each aspect are detailed numerically. Instead, consistency in judging across all trees is emphasized. While we could speculate that more instructions were provided on the actual scoring sheet, it is safe to assume for this exercise that this is the extent of the directions the judges received.

Individual highlights: Peter Warren awarded top marks to Eric Schikowski's collected Mountain Hemlock.

Individual highlights: Peter Warren awarded top marks to Eric Schikowski’s collected Mountain Hemlock.

The published results give bonsai enthusiasts an unprecedented opportunity to use statistics as a tool for learning. Herewith, a rudimentary analysis of some trees from the event which I hope will give some insight into what five highly-respected judges believe qualifies as a great bonsai at one of America’s top bonsai shows.

The average score for all trees accounting for the discarded low and high scores in the rubric was 43 with a standard deviation of 4. Predictably, two standard deviations above mean (51) gives us the two winners which went to a non-statistical tiebreaker.

Colin Lewis gave his top score to Doug Paul's collected Rocky Mountain Douglas Fir.

Individual highlights: Colin Lewis gave his top score to Doug Paul’s collected Rocky Mountain Douglas Fir.

High Scoring Trees

Let’s take a look at the set of trees that were one standard deviation above the mean (47) or higher; there were 14 trees in this category. Based on the information in the program, here are some statistics:

13 of 14 were conifers, the single non-conifer was a Buttonwood.
Correction: All 14 of the top 14 were conifers. The Buttonwood scored 46.
11 of the 14 highest scores went to domestically collected wild trees. 3 trees were Japanese imports, 1 origin not listed.
14 of 14 highest scores went to trees that were at least 18″ across or 18″ tall.
7 of the 14 were a juniper species.
3 of the 14 were white pines.
2 of the 14 were Redwoods.
1 was a Mountain Hemlock.
1 was a Douglas Fir.

Low Scoring Trees

11 single trees and 3 multi-tree displays scored one standard deviation below the mean or lower. Here are some statistics in this set:

10 of 14 were nursery stock or field grown.
2 of 14 were domestically collected wild trees.
2 of 14 origin unspecified.
There is no clear size trend among low scoring trees.
2 of the 14 were a juniper species.
8 of the 11 single trees were conifers.
3 of the 11 single trees were broadleaf.

Boon's top scoring tree of the exhibit was John Kirby's California Juniper.

Individual highlights: Boon’s top scoring tree of the exhibit was John Kirby’s California Juniper. Correction: Boon picked Eric Schikowski’s collected mountain hemlock.

Individual Highlights

So which tree was the favorite of each judge based on raw scores? Three are shown above, here are all of them:

Peter Warren awarded top marks to Eric Schikowski’s collected Mountain Hemlock.
David Degroot awarded the high score to two trees: The large Japanese Black Pine from the Pacific Bonsai Museum and Jim Gremel’s field-grown ‘Kishu’ Chinese juniper.
Colin Lewis gave top prize to Doug Paul’s collected Rocky Mountain Douglas Fir.
Walter Pall was most impressed by Bob Shimon’s collected coast redwood.
And Boon marked John Kirby’s collected California Juniper as most deserving Correction: Boon picked Eric Schikowski’s collected mountain hemlock.

As expected from the Rubric, each judges’ high score was not counted in the final average for their favorite tree.

Conclusions

While statistics may tell us a few things about this show they may also be misleading with a show that was only 71 displays deep. The trends on the high scoring trees are clear with a near-unanimity for collected conifers. 68% of the entire show was made up of conifers, 92% of the high scoring trees were conifers. The trend in sizing is also pretty clear – medium to larger trees scored higher.

Based on these results, it’s safe to say that if you want to impress these bonsai judges, you’d better have large collected conifers. Indeed, if we consider the basic criteria for judging any bonsai, it is logical that the collected conifers scored highly. The collected trees have trunks that are readily available in larger sizes and that already have a lot of character.

What is more difficult to understand is why some of the best broadleaf trees in the exhibit did not score higher. There were certainly a number of very high quality entries. Did the collected conifers score higher because they inherently possess more bonsai character? Or was it because the deciduous in the show were not as well-developed in this case?

Perhaps this means that we all have a lot of work to do when it comes to creating deciduous and broadleaf trees that will be of equal or higher quality to the collected conifers of the Western US. Many of these conifers had been in training for fewer than 10 years while the large broadleaf trees had been in training almost uniformly for 20+ years. The conclusion I draw is that it takes more time and dedication to grow good broadleaf material than it does to create stunning bonsai from collected conifers. Perhaps in The Artisans Cup in 2020 we will see a broadleaf tree in the top 5 in the scoring. Will it be yours?

27 Comments

  1. Walter Pall
    October 5, 2015

    I agree to most everything you are saying. Modern Bonsai has finally fully arrived in America. In this style it is most important to impress the viewer with whatever means. The monster conifers have the most talent for this. Quiet, more classical and zenny trees should be there but will by definition not score high. Loud scores high. You don’t have to like it, just understand.
    Walter Pall

  2. Ross Clark
    October 5, 2015

    It’s a good start, but the extreme bias toward high scoring only a few mostly western and Oriental species certainly raises questions. Also, it would be logical to suppose that holding this event in the extreme northwestern part of the U.S. would make it more difficult for Easterners to participate, in spite of offering facilitated transportation. And perhaps the scoring system could reflect a wider spectrum of viewpoints.

    Also, here’s a question for you…..When nature has done all of the work over many years (as is the case with collected trees), is it accurate to say that a bonsai artist has “created” a bonsai?

    • Eric Schrader
      October 5, 2015

      To answer your question – I think if you saw what goes into the transformation from a wild tree to bonsai you would likely agree that the final product is created. It would be extremely rare to find a collected conifer that needed no work. The process is faster than scratch growing but it is an art, distillation and manipulation are key to this creative process.

    • kora dalager
      October 6, 2015

      Ross:
      Please know, that there is also a National Bonsai Exhibition every other year in Rochester, NY. In 2016 it will be the 5th show. It is certainly an extra ordinary commitment in time and cost, to transport trees across the US. So can you say, these are the best trees in the US, no, you can only say, these are the trees submitted, and accepted for this show.

  3. dave crust
    October 6, 2015

    The remarkable thing to me about the scoring of trees was not how the judges did but how the final score was tallied. There was a lot of hub-bub and some confusion about tie-breaking. This was nearly completely unnecessary. If the cumulative tree scores had been expressed with out rounding and instead were given a whole and percentage number it would have been a no-brainer.

  4. Judging Bonsai at the Cup: The Limits of the Point System | Bonsai Bark
    October 6, 2015

    […] Eric Schikowski’s collected Mountain Hemlock was one of my favorites. It was also Peter Warren's first choice. I borrowed this photo and the one below from Eric Shrader's PHUTU blog. […]

  5. ceolaf
    October 6, 2015

    I’ve a ton to say on this topic, but I will limit it to three comments.

    1) The judges made clear during the “Ask the Judges” panel that that they each ignored the “rubric”, judging this exhibition as each of them (each an experienced judge) has judged in the past.

    2) The one command in the scoring instructions for the judges was to be consistent. “(We are not telling you what is most important, we are only telling you to be consistent in how important you think each aspect is,” says the scoring guide.

    And yet, each of the judges made clear in the panel that they disagree with this. For example, they said — in their different ways — that different species must be judged differently. Boon mentioned that he adjusted for degree of difficulty of the species, for example.

    So, unfortunately, the scoring was really not much different — if at all different — than it would have been without Ryan and Chelsea’s efforts to do something new/better with it.

    3) You commented that it would be expected that the winner would be two standard deviations above the mean. Out of 71 trees, we might expect around 2 trees beyond two standard deviations above the mean if the distribution was normal. But don’t think this was a normal distribution. If you squint, you can see a bell, but it is skewed and therefore not centered on the mean. (This is the most common problem with any statistical inferences beyond basic descriptive statistics, that basic statistical assumptions are violated. In this case, you assumed a normal distribution, but that is not the case.)

    Perhaps you could publish a chart of the distribution of the scores? Perhaps a pair of charts, one of the total scores, and one with the high and low dropped?

    • Eric Schrader
      October 6, 2015

      Ceolaf –

      I chose not to attend the judges panel simply so I could spend more time in the exhibit Sunday afternoon. I attended Ryan and Chelsea’s panel because I had never heard them speak in such a forum before. I think your three comments are insightful.

      1. This is human nature – if you are an expert, even when another expert tells you something you will tend to trust your own judgement first. At least in my case, I may later be influenced by what the other people say after considering their perspective for some time.

      2. I disagree that Boon’s interpretation is inconsistent. It fits into the Rubric as there was instruction to take into account species: “Design is a good representation of the species as it occurs in the natural environment, OR design is a good representation of a shape as a bonsai. ”

      3. I should have disclaimed being a statistician at the outset. I wanted to crunch numbers as it’s fascinating to do so, but my understanding of statistics comes mostly from being 1 standard deviation below the mean on most of my college chemistry exams. People who were 2 STDEV above were awarded A’s.

      Thanks for your thoughtful comments.

    • Peter Warren
      October 7, 2015

      Ceolaf, without wishing to be overly confrontational, please give evidence to back up your claim that we judges made clear that we ignored the rubric. It is doing the bonsai world, the judges and the organisers a disservice to make such a statement without any evidence. I said nothing of the sort in the panel discussion.

      Personally, I stuck to the rubric and was as consistent as possible. I thought it was a great way of judging, very fair and very honest, as all of my comments in the panel showed. This will out in time once the video is online.

      As for the idea that Boon mentioned, different species being judged differently. Take the example of ramification on deciduous trees. A zelkova can become incredibly well ramifed. An Apple tree or Quince cannot because of their growth habit.

      Consistent judging here and what Boon was alluding to is to ask the question ” Does this Zelkova show a good level of ramification compared to other Zelkova?”, equally “Does this apple show good ramification compared to other apple?”

      If you conpare the ramification of the apple versus the zelkova, the zelkova wins.

      This is being objective, the whole point of the rubric, subjectivity was taken out of the equation as much as possible. My “favourite” tree scored low points because of the rubric due to health issues. Other trees that I really liked scored low because of the rubric….because it was the DISPLAY that was judged, not just the tree. Many of the great trees were let down by poor display techniques. I can think of one deciduous tree that was very highly liked by a number of judges but scored low as a display. (This was discussed only at the panel at the end of the show before anyone gets upset)

      Read through the rubric, look at the scores, look at the pictures again and listen back to the video when it is online and then come back and tell me that I and the other judges disagreed with the rubric. We all thought it a breath of fresh air and the best system we had ever been asked to use. Fair and objective but still with room for personal preference as to weighting of importance.

      As for the deciduous trees. Go back and look at them again. Look at health, look at the leaf quality, look at the distribution of leaf size, look at the styling, look at the overall display, look at the quality of the scissor work, look for multiple branches or even buds forming at one node, look for the presence of wire scars, look at the mossing, look at the pot combination, look at the table suitability, look at the craftsmanship and quality of the two, look at the ramification, the primary and secondary branch structure, look at the presence and health of internal branching, the nebari and the scar healing, take a breath and step back and appreciate the feel of the display, ask yourself the questions, “how could this be better?” ,”has the artist achieved what they set out to do and has it been executed to the fullest?”. Do all of that in less than 90 seconds and you should get an idea of how much they were considered and judged.

      Im a massive fan of deciduous trees and especially the homegrown american decidious trees, I have made no secret of the fact that this aspect of American bonsai beats European bonsai hands down. I would have loved it if one of them had won, but allowing personal preferences and agendas into the judging would have been going against the rubric and the spirit of the show. Fact of the matter was that the deciduous trees were slightly lacking in comparison to the conifers.

      Its great that such interest has been created in the whole process and for anyone who disagrees, please put forward a better system that is not open to corruption and personal politics (the main reason for dropping the the top and bottom points), and is not impossibly rigid with fifteen categories of winners.

      Intelligent discussion (statistical based or not 🙂 pushes things forward. Comments with no basis in truth do not.

      • Symon
        October 8, 2015

        Thanks for the insights Peter. In your opinion how did these trees compare to trees in the Kokufu exhibition? Probably off topic, but how does the judging in Japan differ from what happened here?

        • Peter Warren
          October 8, 2015

          I would say that about 10% of the trees would walk into kokufu as is, 50% would do with a bit more ramification: foliage density and there are three or four trees that could win it in the future.

          Judging compared to Japan? Very different. In Japan there is no transparency or defined rubric.

          I think it important however to stop looking towards Japan and using it as a yardstick or reference point. It’s time for American* bonsai to start feeling comfortable in it’s own skin both in terms of collected and home grown trees. The inferiorty complex that you suffer from is restrictive and unnecessary. Hopefully the Artisans Cup was a collective therapy session.

          * same is true for European and Rest of world.
          Lots can be learnt from Japan equally lots should never be copied.

      • Ray Norris
        October 29, 2015

        Well said Peter. I agree it was a breath of fresh air to the judging system. all of the judges are experienced and I really enjoyed all their comments about he show. Kudos to Ryan and Chelsea and
        all who made this show possible.
        Thank you very much for your thoughts.

  6. Zack Clayton
    October 6, 2015

    I have been at several shows where a self appointed expert declares that a really fine deciduous tree is “Not a real bonsai”. As artists and practitioners we know better. But, from the show results, is there a subconscious bias that informs our scoring? At shows where the categories are divided and a best is awarded in each category, I would stack up the best deciduous against the best conifer and challenge you to make much difference. I think that an underlying bias does exist. Jin and shari add a Wow! factor that most deciduous can’t match.

    • Elizabeth Johansen
      October 7, 2015

      My question would be why don’t people do jin and shari on deciduous trees? Really old deciduous trees get them as well. I’ve a large maple next door to me that is a mess of jin and shari. There is an old apple down the road that is a fifteen foot stump at a slight angle with one remaining living branch that has branched out in a broom pattern. It has all the wow factors that if reproduced in bonsai form would surely be a show favorite.

  7. John Kirby
    October 6, 2015

    Judging or scoring multiple items is always complicated. It is particularly complicated when there are a lot of unique subjects to be judged. On some review panels for competitive processes, training judges to use the full range of scores available is a first step, or normalize the distribution after scoring and before analysis, then you can compare apples to apples. another approach would be to have a pre-screen/triage to limit the number of trees to be actually judged and then apply the scores to the top ten or some other limited number.many approaches can work. Since most of us in the Bonsai world are not statisticians/data modelers, the exact method chosen may not be intuitively obvious to us, but the fact that a careful and thoughtful system is used should be a relief to the black box judging we get at most shows. I believe that we should have great hope that the transparency and openness of the process from the Artisan’s Cup will encourage others to do so at judged events. I think we owe Ryan and Chelsea a big thank you for this, along with the congratulations for the classy show.

    • Eric Schrader
      October 6, 2015

      I’ll reiterate John’s thanks to Ryan and Chelsea. I find the chance to analyze the results openly to be a great gift to the community. It furthers the intelligent discussion of bonsai as an art form to be able to get inside the thoughts of 5 professionals.

  8. Cheryl Sykora
    October 6, 2015

    And I thought I was the only one that analyzed the scores. Wow! very interesting. I did notice that the winners consistently scored high across all the judges making me wonder the importance of throwing out the top and bottom scores. also a better way of breaking ties would be to add all the judges’ scores together and average rather than what was done in my opinion. then it would have remained the judges’ decision. Very brave to publish scores and allow us to all pick it apart though! Thanks Eric for a very interesting post.

  9. bonsai Mohegan
    October 7, 2015

    Eric, greetings from the northeast. This is my visit to you r site. I would like to ask if you moderate comments, or if you invite ALL discussions?

    • Eric Schrader
      October 7, 2015

      Well, that’s an interesting question. I have to say that my blog has not previously hosted a debate. I do moderate comments as without doing so I could end up with things on my blog that are completely ludicrous. However, to date I have only deleted comments that were either completely off-topic (like “how do I sign up for emails?”) or obviously spam. I don’t intend to censor people here but I would urge you and everyone to keep your comments to an intelligent debate and on-topic.

      My purpose in writing this article was to call people’s attention to the fascinating way in which the published scoring and judging rubric have allowed us the opportunity to elevate the way that we evaluate trees. Secondarily it was to incite a discussion of how smaller or broadleaf trees could have scored better or been developed better.

      My hope would be that the debate leads to meaningful strides forward, perhaps to establishing a standard judging format for bonsai where people could use scores as teaching tools and for evaluating material that they are considering purchasing.

  10. Paul Pikel
    October 14, 2015

    Actually there were only 13 trees of a score of 47 or above. 5 had a score of 46 including the buttonwood which was incorrectly ‘corrected’ above

    • Eric Schrader
      October 14, 2015

      Hi Paul! Congratulations on such a wonderful buttonwood and on being a TAC exhibitor.

      I count the following trees as scoring 47 or higher:

      4,5,8,14,27,30,31,32,38,52,53,56,61 and 64.

      Note that the program incorrectly identifies tree #53, it was a collected Colorado Douglas Fir by Scott Elser, not a cork oak as the program states. I believe that this was a last-minute change.

      • Paul Pikel
        October 14, 2015

        Thanks very much. It was great to be part of. I appreciate the clarification of the scores you used. I’ll check it out.

  11. Julie Trigg
    October 14, 2015

    fascinating article. Appreciate all the viewpoints, even though I found some confusing. I feel it is so important to have a fair system like this. Hopefully, this will eliminate the politics involved in so many shows. Thank you all!

  12. Ed Trout
    October 15, 2015

    I regret that I was not able to attend this event, but none the less fully supported it’s concept. I applaud Ryan & Chelsea, and Michael, and all the AC organizers for their dedication and work to pull this off. I also believe that we should all do bonsai for our own pleasure, and the fact some others like our work is a bonus. When we enter our work in an arena like this, we must be prepared for any and all critical comments that will come. But criticism evokes thoughts, and when we turn those thoughts into creative improvement, only positive results are realized. And we move forward, and everyone wins. I believe that this show, and Bill V’s previous national shows are the catalysts to move North American forward that we should all appreciate.

  13. Mark D'Cruz
    October 15, 2015

    I thinks the whole premise of proposing a point system to score art, beauty and nature, which is what bonsai is all about is where the problem lies. Trying to quantify one’s emotional stirrings, into rigidly sliced out section and prescribed check boxes is quite impossible and perhaps fit more for engineering and qualitative sciences rather than subjective and emotive art forms.

    Unfortunately, our very way life, our schooling, university, work and reward system seems to want us to be controlled to such ‘point based’ systems.

    I think when we get a panel of people renowned in their arts to make subjective and emotive judgments within the strict confines of a point-system it will bring out counter-intuitive results, albeit only Conifers in the top tier, or the skew to collected materials.

    Ask people to define why they are moved by something, but then restraining them to a handful of pre-defined teaching aids is naïve and prone to fracture.

    For the sake of openness and brevity you could have a point system to rank art within a panel, but confining the panel to a scheme is belittling the panel’s maturity.

  14. Jamie A.
    October 24, 2015

    I’ve also been thinking that the whole competition angle within the bonsai world is a bit problematic.

    I noticed on The Artisan Cup’s website that they want the 3rd Cup to be help at New York’s Museum of Modern Art (MOMA), and I really appreciate this push towards having bonsai recognized as an art, rather than a craft. But a lot more goes into this transition than simply trying to hold a competition at an art gallery.

    MOMA doesn’t have prize competitions, they have curators who themselves make judgments about what work is important, and then buy that art. And, when they have traveling exhibits that they bring to MOMA, these are themselves exhibits chosen by the curators, and not competitions with points. In a sense, the art world gives even more authority to the “experts” than does the bonsai world. While this has problems, especially when it comes to introducing new forms of art into a world dominated by big institutional players, but it reflects something that the art world understands about the nature of art – and that the bonsai world doesn’t yet understand.

    Essentially, what makes good art is not really how it stacks up against another piece of art, in some sort of rubric based competition, but often, what curators are looking for, are pieces of art that somehow advance the discipline and dialogue of art, by doing something that hasn’t yet been done. Certainly technical proficiency helps, as the more proficient someone is the better they can use techniques to say something new, but technical proficiency is secondary. Instead, the discourse of art is all about using the medium to advance our understanding of the world, and also, our understanding of art itself.

    For instance, the work that seems to have received the most discussion from the Artisans Cup was David Crust’s Larch in the Vacuum. Well, why? Because, I think, he did something new, and in doing something new, provoked a lot of thought about what bonsai is. To my mind, what he helped demonstrate is the way in which nature doesn’t really exist as something distinct from the human world, as the moment we start talking about “nature,” we’re already there modifying it, and “creating” a romantic vision of it. In other words, the human world is always interacting with nature, even if we like to imagine a stylized nature that exists independent of human interaction, because that very stylized view of primeval nature if a human idea about nature imposed on the world. In fact, that vision of unspoiled nature, to my mind, lies at the heart of what draws so many of us to bonsai, as we’re all often very drawn to nature. But it’s nice to be reminded of the fact that nature doesn’t exactly exist as we imagine it, because the idea of “nature” that we have is an idea of nature that WE have, and not nature itself.

    Anyways, this point is neither here nor there. The issue is not what that piece means, but that it clearly is advancing the medium of bonsai in interesting ways, which people can recognize even if they don’t like the work. But if bonsai is an art form, it seems like these are the works we should be praising, because its trying to say something new. But this presupposes an entirely different motivation to bonsai practice, one that isn’t motivated by winning competitions, which is certainly fun, but which is also a somewhat parochial goal if measured against the desire to advance the art form. Although, perhaps would be less parochial if those competitions were explicitly geared towards recognizing those works that truly took the medium in new and unexpected directions – which is likely hard to measure via a rubric.

    So, the question, for me, is how can bonsai develop in this direction, assuming we believe it is or should be considered an art form, given that it is still a fairly unusual art form. We don’t have museums and curators, who might be expected to scour the world for the most important works. Granted, we have a few of these, but bonsai is something that takes place in peoples backyards, and within commercial nursery’s, which presents interesting challenges. Bonsai competitions certainly raise the technical practices of bonsai enthusiasts, but how does a discipline transition from valuing that which can be measured in a head to head competition, such as the technical features of a tree, to instead valuing that which a bonsai “curator” might value, such as the tree’s ability to advance the medium of bonsai as art? That requires a big cultural shift, although, the community is probably already moving in that direction.

    I hope that Ryan Neil manages to stage the 3rd Cup at MOMA, and I can see them being interested in it. But I’d also guess that it might be their first “competition.” I suspect that until the community figures a way to organize an exhibit of works that are curated, rather then in competition with one another, that the community will not really have “arrived” as an art form. Although, the fact that we do have books dedicated to the artistic vision of certain artists probably indicates that we’ve been moving in this direction for a long while.