Have you ever had a result at a contest that you just can’t understand, maybe your band played well but still ended up near the bottom of the pile? Or maybe a band that obviously struggled with the test piece were placed a lot higher than they, or anyone in the audience, expected? Of course this has happened to you, after all “that’s contesting”…
Often when you get disappointing or puzzling results, you think that maybe the adjudicators were looking for a different interpretation, or only focusing on a few select passages and ignoring other sections of the piece. Or maybe you wonder if the adjudicators were unduly influenced by the opening few bars and immediately pigeon-holed you into the bottom half of the placings. Or sometimes you wonder if they might even have got you mixed up with another band! Amongst all these possibilities, have you ever considered the influence of draw position on the final result?
In what follows, research undertaken by The Contest Sleuth reveals that, for many sizes of contest, achieving a podium placing (1st, 2nd or 3rd) is not independent of draw position. In other words, contrary to what you would like to believe, draw position does influence your chance of getting into the prizes, as well as your chance of coming near the bottom of the results table.
Disclaimer: before continuing, please note that the purpose of this article, and the associated research, is not to discredit the performances or placings of any bands, or to throw doubt on the work of any adjudicators either past or present. The motivation was to stimulate thought and discussion amongst the wider brass band community and the decision makers in order to (hopefully) improve the experiences of contesting for everyone involved.
Research & Results
Research by The Contest Sleuth on a huge amount of data from hundreds of contests (test piece and own choice contests with closed adjudication, as well as a smaller number of entertainment contests with open adjudication) has revealed significant differences in the number of times a band from each draw position has been award a podium placing (e.g. 1st 2nd or 3rd). The dataset focuses on contesting in the UK over a period of over 40 years and encompasses results from all sections, Championship through to 4th section. Contest results have been separated according to the number of bands taking part and the results analysed to investigate if there is a relationship between draw position and placing, or whether there is no/not enough evidence to suggest that draw position influences the results. As very good and very poor results generally matter the most to competing bands, exploring any connections between draw position and top 3 and bottom 3 placings has been the primary focus of the research.
To get an idea of the findings, let’s first take a look at the distribution of podium placings with draw position for a contest/section with a field of 12 bands, e.g. a typical Championship section regional contest. We’ll later look at a field of 20 bands and then see what insights we can gain from these two examples.
Results for a Field of 12 Bands
Let’s get straight to the data. Figure 1 shows the total number of podium finishes (1st, 2nd and 3rd) from bands in each draw position for a field of 12 bands. The data encompasses the results of 222 contests, so we’d expect each composite bar of the chart to have a height of roughly 56 (19 firsts, 19 seconds and 19 thirds) if draw position didn’t influence the results. Clearly, this is not the case.
The discrepancies are such that the probably of achieving this distribution if result were independent of draw position is less than 0.1% (see the later section on the statistical significance of the results). Instead, very few bands have claimed podium finishes from the dreaded number 2 draw or from being drawn number 3. On the other hand, being draw 9th, 12th, 11th or 8th appears to give you a much better chance at claiming a podium finish. Looking at just 1st place (the gold bars in Figure 1) shows that you are more likely to triumph if drawn 8th, 11th or 12th.
Does this mean that it doesn’t matter how well a band drawn number 9 or 12 plays? No. The data does, however, suggest that draw position does matter when considering performances of a similar/comparable standard. For example, if the best 6 performances are all of a comparable standard and are from bands drawn 2nd, 3rd, 10th, 9th, 11th and 12th, the data suggests that the podium finishers would be bands drawn 9th, 11th and 12th and that the other accomplished performances will likely not get the credit they deserve.
Results for a field of 20 bands
Let’s now take a quick look at the results for a field of 20 bands, see Figure 2 below. This could be a section at the Spring Festival, National Final or a larger section at a regional contest.
The data in Figure 2 is based on the results from 141 contests, so we’d expect each composite bar on the chart to have a height of 21. Again, this is clearly not the case. Looking at the total height of the bars, which indicates a greater number of podium placings, success is generally more likely for bands in the second half of the draw than in the first, the exception being for bands drawn 9th. Also worthy of note is that of the 141 contests in our database, only 6 podium places were achieved off the dreaded number 2 draw, compared with over 30 podium placings from bands drawn number 9. Looking at just 1st place (gold bars), more bands have gained victory from a number 14 draw and no bands have won off a number 3 draw (based on the data we have).
Variation in placings compared with the assumption that the draw doesn’t matter
While the above results are very interesting, it is perhaps even more telling to look at the variation of the recorded results compared with the results which would arise if draw position made no difference to the result. We’ll look at this data for the same two section sizes of 12 and 20 bands.
For a field of 12 bands, the results are shown below for both top 3 placings (green bars) and bottom 3 placings (red bars).
As you can see, we’ve annotated these plots and categorised the different draw positions. Bad draws (shaded in red) are those with a greater number of bottom three results and a lower number of top 3 results than you would expect if draw position makes no difference. Conversely, good draws (gold shading) are those where the historical data shows that more bands have achieved a podium placing and fewer bands have achieved a bottom 3 result. There are also draw positions where fewer bands do very well and fewer bands do badly, i.e. you could think of these draw positions as those that are likely to give you a mid-table result. For a field of 12 bands, the draw positions can be separated quite nicely into these three categories, with the three best draw positions being 9, 11 and 12 and the worst three positions being 2, 4 and 6. Maybe the legend of the dreaded number 2 draw isn’t a myth after all…
We can make a similar chart for contests with 20 competing bands, see below.
For this section size, the three categories of ‘good draws’, ‘bad draws’ and ‘mid-table draws’ still apply, but there appears to be an additional category. We’re going to call this extra category ‘high risk & high reward’. The draw positions in this category (9, 10 and 14 in the field of 20 bands) are those with a higher occurrence of good results and a higher occurrence of poor results. You could, therefore, view bands draw in these positions as being the bands that the adjudicator remembers the most vividly. If you pull off a good performance from these draw positions, then you’re probably going to do very well indeed. However, play poorly and you’re probably going to end up near the very bottom…
Statistical Significance / Hypothesis Testing
You may be wondering whether the results presented above are statistically significant and whether the discrepancies between draw positions which have been highlighted above can be easily explained away. The short answer is that we’ve drawn on enough data to show that, if draw position didn’t influence the results, then the probability of getting the results shown above (i.e. for contests with 12 or 20 bands) is less than 0.1%.
Looking at different section sizes, the probability of achieving the observed results if the placings were independent of draw position would be <1% for section sizes of 8, 12, 13, 15, 16, 17, 18, 19 and 20 bands. In other words, we can be pretty confident that draw position does matter for these section sizes!
Interestingly, we don’t have enough evidence to suggest that draw position influences the results for contests with 7, 9, 10, 11 or 14 bands. This could be that we haven’t collected enough data yet, or that there’s something about these section sizes that means that the influence of draw position isn’t as significant.
Concluding Remarks / Suggestions for the Future
While we all understand that, despite the best efforts of organisers and adjudicators, brass band contesting may never be a completely fair and transparent process, it seems that there are a number of issues with the status quo. For contests like the Spring Festival and the National Finals, with around 20 bands per section, it is clear that many of the optimum draw positions are in the second half of the draw. Can we, in good faith, seriously consider just carrying on a system which appears to skew results so considerably?…
While we’ve done some fact-finding on this issue, we don’t pretend to know all of the answers, or how to solve this problem. That being said, we’ve made a few suggestions below which may be worth considering.
- Consider smaller sections. The data we have suggests that, in general, the results from smaller sections are not influenced as much by draw position. At a regional level, this could be achieved by splitting up or redrawing regional boundaries or creating new regions.
- Consider a greater number of comfort breaks for adjudicators to try and replicate the effect of smaller sections. We’re not sure if this will work, but it may be worth a go?
- Look into ways of helping adjudicators remember early performances, particularly in large sections. Audience members often remember bands through a ‘visual hook’ such as the colour of the uniforms. Following this logic, it may be worth trialling say different coloured paper for the adjudicators’ remarks for each band? Or maybe even switching to Open adjudication – although we guess the jury is still out on whether this is actually fairer?
- Consider training adjudicators to ‘lock in’ points scores after each performance (maybe using Steven Mead’s criteria-based system?), rather than (presumably) waiting until the very end to decide the placings and then fudging the points scores to decrease by one point for each band below the winner (I think we’ve all see this before…)
- Many contests are now recorded, maybe the adjudicators should be given the opportunity to listen back to parts of each performance to help them make their decision? Although this may obviously delay proceedings, we think that many banders would prefer a more considered decision rather than a faster one.
Thank you for taking the time to read this article. Please let us know your thoughts (either leave a comment below or send us an email, details on the contact us page) and do feel free to share this article. As mentioned earlier, the aim is not to foster malcontent or to discredit the fine work done by adjudicators, so please don’t take this the wrong way! Also, let us know if you’d like a copy of the underlying data.
We plan to make predictions for the upcoming contests using just the draw position, and maybe some knowledge on the current form of the bands, so please do look out for those.
If you want to know what the podium placing charts look like for more section sizes, check out the images below (click to expand).
