-
Bhangra discussion is still going strong. Join us in our Facebook group!
Which competition first used judging deliberation?
- Thread starter KushK
- Start date
yraparla
SwizzeeMusic.com
- Messages
- 2,072
It was at least when I was starting out, maybe 2007-2009. Potentially even before that as well. Keep in mind that we didn't even have judges meetings because they'd be gone as soon as show was over. There was no notion of giving out the scores, or handing back notes. This is all fairly recent stuff (Burgh, Blowout, Bruin, Elite 8, and others leading the charge).
Back then Judges were getting slammed on the regular because rubrics weren't accounting for traditionality and that was the hot ticket at the time. So deliberations allowed judges to take that non-rubric element into account. Now traditionality is everywhere in rubrics and a totally useless category since literally no teams represent early 2000s bhangra anyway (even Empire doesn't fit that category). So we need deliberations now to take non-rubric elements into account to avoid cookie-cutter 'folk' teams winning everything.
I prefer deliberations, but with a caveat. Firstly, fuck points. Rubrics and points are arbitrary, rushed, and unrepresentative of a performance. What does a 10/15 mean? an 11/15? Nothing. You know it means nothing because judges use points differently themselves! There's no way to see a number at a comp and know what it means.
What matters are relative points. If you see a judges spread of points, now you have a better sense of what they mean. But it's still stupid because some judges will have an overall lower median/mean or a very high variance. How can a judge possibly give anything above a 90 boggles my mind, but it may be perfectly reasonable to someone else. Even among judges at the same comp, with the same rubric, you'll get inconsistent points where two judges who might think the exact same of a team but will give diff points.
So the hybrid solution is to have judges rank teams based on points, then let judges decide how to place them using the points and rubric as a guide.
Here's my caveat: Deliberations are where the committee needs to step in and protect the teams. Judges should be very tightly controlled on changing placings and have legitimate arguments for doing so. Any arguments should either be in line with the rubric, or be factors that the rubric did not account for. We can't have judges using rubric for points and then using their own preference for placing.
Deliberations are meant to enhance the rubric process, not override it. As long as judges can make a sound argument, I think it's fine. ultimately the only thing we remember are the placings, not the points. So ignore the points at the very end and let judges put out the placings they can live with so we can fairly hold them accountable.
That caveat is hugely important. We've seen judges go off the rails and just do w/e they want (cornell blowout comes to mind recently where they didn't even place after racking up most points). My recommendations is take top 4 teams by points. No team can move more than one spot (so number 4 could only be number 3, number 1 must place 1st or 2nd) and points have to be reasonably close. I think this balances the respect for points and gives judges flexibility to adjust.
my quick thoughts
Back then Judges were getting slammed on the regular because rubrics weren't accounting for traditionality and that was the hot ticket at the time. So deliberations allowed judges to take that non-rubric element into account. Now traditionality is everywhere in rubrics and a totally useless category since literally no teams represent early 2000s bhangra anyway (even Empire doesn't fit that category). So we need deliberations now to take non-rubric elements into account to avoid cookie-cutter 'folk' teams winning everything.
I prefer deliberations, but with a caveat. Firstly, fuck points. Rubrics and points are arbitrary, rushed, and unrepresentative of a performance. What does a 10/15 mean? an 11/15? Nothing. You know it means nothing because judges use points differently themselves! There's no way to see a number at a comp and know what it means.
What matters are relative points. If you see a judges spread of points, now you have a better sense of what they mean. But it's still stupid because some judges will have an overall lower median/mean or a very high variance. How can a judge possibly give anything above a 90 boggles my mind, but it may be perfectly reasonable to someone else. Even among judges at the same comp, with the same rubric, you'll get inconsistent points where two judges who might think the exact same of a team but will give diff points.
So the hybrid solution is to have judges rank teams based on points, then let judges decide how to place them using the points and rubric as a guide.
Here's my caveat: Deliberations are where the committee needs to step in and protect the teams. Judges should be very tightly controlled on changing placings and have legitimate arguments for doing so. Any arguments should either be in line with the rubric, or be factors that the rubric did not account for. We can't have judges using rubric for points and then using their own preference for placing.
Deliberations are meant to enhance the rubric process, not override it. As long as judges can make a sound argument, I think it's fine. ultimately the only thing we remember are the placings, not the points. So ignore the points at the very end and let judges put out the placings they can live with so we can fairly hold them accountable.
That caveat is hugely important. We've seen judges go off the rails and just do w/e they want (cornell blowout comes to mind recently where they didn't even place after racking up most points). My recommendations is take top 4 teams by points. No team can move more than one spot (so number 4 could only be number 3, number 1 must place 1st or 2nd) and points have to be reasonably close. I think this balances the respect for points and gives judges flexibility to adjust.
my quick thoughts
yraparla
SwizzeeMusic.com
- Messages
- 2,072
Another idea i've been kicking around is having a merit-demerits system. Rubric is intact, but describes the weight of each category. Judges note things they like (points) and things they didn't like (demerits). Apply the points and demerits to each rubric category (let's say you can give a value of 1-3 per each note). Then using the weighting you get a score. Judges take top 4 and deliberate.
Benefit is that points/demerits are directly tied to tangible parts of the routine (if you even out you just end up at the average). So at the end, you have a list of all noteworthy items, categorized by rubric, and when judges deliberate they just spent time applying the rubric to their notes so they're already thinking in terms of the rubric.
I think this frees judges from the nonsense of deciding if a team garnered an 8 or a 9. Make no mistake, judges get about a minute to score roughly 10-12 categories. You think they're that precise? Averages often come down to 1-2 points sometimes, that's well within margin of error. This system would alleviate that, you have to earn the merit/demerit instead of magic handwaving by judges
sorry side note but I think it's a cool idea. I'd rather do away with points entirely, but that's not feasible, this is kind of an interesting compromise.
Benefit is that points/demerits are directly tied to tangible parts of the routine (if you even out you just end up at the average). So at the end, you have a list of all noteworthy items, categorized by rubric, and when judges deliberate they just spent time applying the rubric to their notes so they're already thinking in terms of the rubric.
I think this frees judges from the nonsense of deciding if a team garnered an 8 or a 9. Make no mistake, judges get about a minute to score roughly 10-12 categories. You think they're that precise? Averages often come down to 1-2 points sometimes, that's well within margin of error. This system would alleviate that, you have to earn the merit/demerit instead of magic handwaving by judges
sorry side note but I think it's a cool idea. I'd rather do away with points entirely, but that's not feasible, this is kind of an interesting compromise.
Apparently, the new olympic ice skating scoring goes something like this; someone on reddit posted a decent rundown:Swi said:
So a comp committee can decide these kinds of base values for the judges, and then the judges' job is to decide how much to add or subtract.
Swi these are some really good points! So how do feel about a Judge who in scores places a team 5th or 6th but then during deliberations ends up placing them first?
Just some things I saw with the judging this past weekend at Fever, which have made me curious as in to how the community feels and if you all feel it is working?
Just some things I saw with the judging this past weekend at Fever, which have made me curious as in to how the community feels and if you all feel it is working?
Saab
Today is a gift
- Messages
- 991
I think the issue you describe would be related to the margin of victory i.e. difference between team one to team five. If all that separates team one from five is two points - three points, I can see a judge maybe changing the score in relation to other factors (re-thinking scoring, discussions about the rubric, competition input, missing something, issues/discussion in deliberation, performances around teams, bias are all factors).
Again, competitions should intervene if their isn't justification, and it's important for judges to be able to convey fully why a switch like that was made. I don't think I have ever heard of a team placing 5th or 6th winning first, and the justification for something like that would need to be conveyed to all affected teams by the judges/judge.
Again, competitions should intervene if their isn't justification, and it's important for judges to be able to convey fully why a switch like that was made. I don't think I have ever heard of a team placing 5th or 6th winning first, and the justification for something like that would need to be conveyed to all affected teams by the judges/judge.
yraparla
SwizzeeMusic.com
- Messages
- 2,072
It's really hard to speak abstractly Ankush so I don't want to say anything specific. In general you would like to see judges stay consistent between points and their vote, but at the same time it's not in a vacuum. Judges discuss things, work through concepts and if at the end they change their mind I usually don't have a problem with that. It takes the whole group to make changes, not just a single judge.
Also in deliberations, you're contrasting specific teams, so the judge could potentially be in a situation where his number 2 and number 4 teams are being discussed for 1st. Maybe he picks the number 4 team based on the criteria the other judges are comparing. You don't' have to hold yourself to the points, you hope that your points reflect your subjective feelings, but in the moment if you feel the right decision is to vote for your 4th ranked team, it's ok because at least 2 other judges had them as first! Margin of points really matter here as well. If all 4 teams were clustered in points and way better than the other teams, then maybe it's ok.
The judges conference is a weird, insular, fast-paced situation. You don't get benefit of a replay, you only have each others notes, and the discussion moves fast. really fast. You discuss tons of elements quickly and minds can change very fast. A judge may have missed something and not given proper credit to a team. On hearing another judge make their arguments, minds can change.
Also in deliberations, you're contrasting specific teams, so the judge could potentially be in a situation where his number 2 and number 4 teams are being discussed for 1st. Maybe he picks the number 4 team based on the criteria the other judges are comparing. You don't' have to hold yourself to the points, you hope that your points reflect your subjective feelings, but in the moment if you feel the right decision is to vote for your 4th ranked team, it's ok because at least 2 other judges had them as first! Margin of points really matter here as well. If all 4 teams were clustered in points and way better than the other teams, then maybe it's ok.
The judges conference is a weird, insular, fast-paced situation. You don't get benefit of a replay, you only have each others notes, and the discussion moves fast. really fast. You discuss tons of elements quickly and minds can change very fast. A judge may have missed something and not given proper credit to a team. On hearing another judge make their arguments, minds can change.
siddyp
Tough times never last, but tough people do.
- Messages
- 1,270
If rubrics are going to combine many elements into big sections, there should be deliberation imo. The points can become unreliable way too easily.
-----------
If rubrics are sectioned into detailed parts, then deliberation COULD be taken away and heres how I think it could/should be done:
cleanliness of the dancing is a section of its own.
cleanliness of formations is a section of its own.
creativity of dancing is a section of its own.
formational creativity is a section of its own.
and I could keep going listing many dif options, and the competition could have whatever sections it wants..
But what this does is allow the judge to give points based on each individual aspect of the WHOLE performance.
If you're judging, you have in your head, what is a 10/10 when it comes to cleanliness of the dancing. Apply that judgement to what you just saw from that team. (for every section.) If a judge can't do this, IMO they aren't qualified.
By doing this, it will offer a real breakdown of where teams were good and not so good. Then by points, however each section is weighted, should add up to an pretty close/final conclusion.
(I think deliberation should still be there in this case as an option ONLY if the points are close to make sure the right outcome is there.) Otherwise, given that the judges are qualified, the points should truly stack up the top teams.
See this as an example: http://bhangrateamsforum.com/discuss/main-bhangra-discussion/my-life-is-bhangra-judging-workbook/
^^ I think it'd be sweet if more comps did this and had more charts that broke it all down.
-----------
If rubrics are sectioned into detailed parts, then deliberation COULD be taken away and heres how I think it could/should be done:
cleanliness of the dancing is a section of its own.
cleanliness of formations is a section of its own.
creativity of dancing is a section of its own.
formational creativity is a section of its own.
and I could keep going listing many dif options, and the competition could have whatever sections it wants..
But what this does is allow the judge to give points based on each individual aspect of the WHOLE performance.
If you're judging, you have in your head, what is a 10/10 when it comes to cleanliness of the dancing. Apply that judgement to what you just saw from that team. (for every section.) If a judge can't do this, IMO they aren't qualified.
By doing this, it will offer a real breakdown of where teams were good and not so good. Then by points, however each section is weighted, should add up to an pretty close/final conclusion.
(I think deliberation should still be there in this case as an option ONLY if the points are close to make sure the right outcome is there.) Otherwise, given that the judges are qualified, the points should truly stack up the top teams.
See this as an example: http://bhangrateamsforum.com/discuss/main-bhangra-discussion/my-life-is-bhangra-judging-workbook/
^^ I think it'd be sweet if more comps did this and had more charts that broke it all down.
Sue Sylvester
kinnell
- Messages
- 467
Personally, I think deliberation serves an important role if the goal of the judges is to place the teams most accurately. If the placings are completely based on points and scoresheets were taken away after each performance, then you rely on a process that essentially attempts to evaluate teams on a more absolute scale, rather than a more ideal, relative scale against the performances of other teams. I've heard of competitions that take scoresheets immediately after performances even though the rubric requires the judge to score components of the performance (creativity, execution, cleanliness..etc) that can only be evaluated after judge has seen the entire performance, which doesn't really give the judges enough time to truly look back. Deliberation also serves to attempt to offset the disadvantage teams that perform earlier in the show (in the first half) have over the teams that perform later in the show. Discussion amongst the judges also gives them the opportunity to evaluate performances from a bigger picture and to share opinions and provide information/insight that the other judges may have missed or may not have considered.
As mentioned by Swi, deliberation does need to be controlled though. I know some competitions take the highest four team scores combined form all the judges and let judges deliberate only on those teams. Points are important, at that point, in order to get teams to that deliberation table at which points they stop to hold weight in the final placings. Since judges may have differing opinions, you can always get a situation where a judge with a different opinion may not have the exact top 4 teams with the combined top 4 with the scores of all the judges.
Has this ever happened before to your knowledge, Ankush? For a very brief second there, I thought you might have been insinuating that it happened at Bhangra Fever this year because you put both those statements so close to each other!
(hahah no worries though, I know that you're not... because you got the actual scores of all the judges at Fever last night so you know that didn't happened!) Gotta love these hypotheticals though - definitely promoted some great discussion on judging!
As mentioned by Swi, deliberation does need to be controlled though. I know some competitions take the highest four team scores combined form all the judges and let judges deliberate only on those teams. Points are important, at that point, in order to get teams to that deliberation table at which points they stop to hold weight in the final placings. Since judges may have differing opinions, you can always get a situation where a judge with a different opinion may not have the exact top 4 teams with the combined top 4 with the scores of all the judges.
Ankush said:
Has this ever happened before to your knowledge, Ankush? For a very brief second there, I thought you might have been insinuating that it happened at Bhangra Fever this year because you put both those statements so close to each other!
(hahah no worries though, I know that you're not... because you got the actual scores of all the judges at Fever last night so you know that didn't happened!) Gotta love these hypotheticals though - definitely promoted some great discussion on judging!
amancheema
Active Member
- Messages
- 240
...
DamanSingh
Member
- Messages
- 849
Haha, I also thought you were insinuating that being the case at Fever, which it certainly was not. The 5th/6th place team did not jump up to first. Of course, you're alluding to a hypothetical situation.Ankush said:
I'm curious. What things are you referring to?
Haha! Not getting into Bhangra Fever judging at all. I 100% agree with the results at Bhangra Fever.
I was just looking at the scoresheets from this past weekend and it made me think of a hypothetical situation and see how you all felt about it? I also help organize WBBC so this helps us learn how the circuit feels about certain things.
This has turned into a really good discussion however! I'm glad I started this =)
I was just looking at the scoresheets from this past weekend and it made me think of a hypothetical situation and see how you all felt about it? I also help organize WBBC so this helps us learn how the circuit feels about certain things.
This has turned into a really good discussion however! I'm glad I started this =)
siddyp
Tough times never last, but tough people do.
- Messages
- 1,270
A lot of false assumptions imo...KarnSingh said:
Judges remember ALOT because they are in that frame of mind. They are writing stuff down without taking their eyes off the performance. (it's amazing how much one can remember when writing stuff down while it happens) They do remember energy levels/josh from each team.
And judges aren't robots. The goal of the teams is to impress the panel. which consists of different judges. not the same judge multiplied by 3 or 4 or 5. If you have a panel of judges that think the EXACT same thing, then it isn't a fair competition. <-- i dont even think it's possible to have a panel of judges that think the exact same way. So while you want judges to not bring in their bias, they all have specific things they want and with that, they'll judge to the rubric.
To your point about fairness, all teams get the same treatment with points and deliberation. They all face the same short amount of time to be given points, and they are given the same treatment in deliberation. Also, rubrics can be funny. And they can do funny things when they shouldn't. Which has the capability of screwing up point allocations. Which makes deliberation beyond necessary.
And teams don't win or place with 1 awesome gimmick that stood out. If all else is equal between them and the teams being discussed, then yes that'll put them over the top given the rubric and/or judging panel wants it. It comes down to the big picture.
yraparla
SwizzeeMusic.com
- Messages
- 2,072
Karn, I'm pretty sure I detailed a framework that addresses many of your issues, please take some time to re-read what I wrote:
What are they discussing? Who is better based on the spirit of the rubric. Who left a lasting impression that night.
Equal time after performance? This is a faulty assumption that all errors for each team lead to offsetting scores. A team that has more gimmicks or 'impact' elements will actually benefit more from a quick score than a team that provided a meticulous and complicated routine.
Your issues seem to be about judges in general. Yes, they can have personal interest or bias, but that's not something that goes away by staying with points alone.
Think of it this way:
We all view bhangra performances differently. There's a balance between the different elements of josh, nakhra, choreo, execution, difficulty, formations, etc. We all have our own personal balance that informs how well we receive a performance. A rubric is supposed to be a quantitative representation of that balance. But it's only ONE balance, the competition's preferred balance. It's guaranteed to be different from what every single person in the audience is feeling. So it's impossible imo to make a rubric that will satisfy even a majority of people watching. How many times have teams heard, 'oh we liked you, but rubric said this.' That's such nonsense, the rubric is supposed to match our general notions. they're not commandments handed down that we must worship. They have limitations, we know the limitations, so the deliberation is a complement to produce better results. Scores group teams so we can discuss just the ones that match the rubric well, then judges deliberate so they can use their years of experience to include non-rubric elements.
If I had my way, we'd have 9 judges and just do an instant runoff system. Judges score teams like normal on the rubric, the scores are added up and given to judges so they have reference. They submit a ranking of all the teams. Then you just do instant run off, dropping the team with the lowest vote every time until you get to the winner. But competitions are conservative and teams are paranoid and judges are not involved in the process (they just show up and judge) so we're stuck with what we have.
The Alternative Vote Explained
Sorry this was kind of all over the place, but to your point Karn. I disagree strongly. Talk to more people who have judged, as you said you don't have much experience with it. Every judge wants deliberation because they know the system is faulty and they would rather adjust and put out placings they really believe in (they can also not change anything and stick with their scores, but at least it's their choice then).
So who's facilitating? Committee is allowing judges to do so. They have the right to either let points stand or allow judges to deliberate.
What are they discussing? Who is better based on the spirit of the rubric. Who left a lasting impression that night.
Equal time after performance? This is a faulty assumption that all errors for each team lead to offsetting scores. A team that has more gimmicks or 'impact' elements will actually benefit more from a quick score than a team that provided a meticulous and complicated routine.
Your issues seem to be about judges in general. Yes, they can have personal interest or bias, but that's not something that goes away by staying with points alone.
Think of it this way:
We all view bhangra performances differently. There's a balance between the different elements of josh, nakhra, choreo, execution, difficulty, formations, etc. We all have our own personal balance that informs how well we receive a performance. A rubric is supposed to be a quantitative representation of that balance. But it's only ONE balance, the competition's preferred balance. It's guaranteed to be different from what every single person in the audience is feeling. So it's impossible imo to make a rubric that will satisfy even a majority of people watching. How many times have teams heard, 'oh we liked you, but rubric said this.' That's such nonsense, the rubric is supposed to match our general notions. they're not commandments handed down that we must worship. They have limitations, we know the limitations, so the deliberation is a complement to produce better results. Scores group teams so we can discuss just the ones that match the rubric well, then judges deliberate so they can use their years of experience to include non-rubric elements.
If I had my way, we'd have 9 judges and just do an instant runoff system. Judges score teams like normal on the rubric, the scores are added up and given to judges so they have reference. They submit a ranking of all the teams. Then you just do instant run off, dropping the team with the lowest vote every time until you get to the winner. But competitions are conservative and teams are paranoid and judges are not involved in the process (they just show up and judge) so we're stuck with what we have.
The Alternative Vote Explained
Sorry this was kind of all over the place, but to your point Karn. I disagree strongly. Talk to more people who have judged, as you said you don't have much experience with it. Every judge wants deliberation because they know the system is faulty and they would rather adjust and put out placings they really believe in (they can also not change anything and stick with their scores, but at least it's their choice then).