Make judging great - some low hanging fruit

[arnab]

After judging a bunch of East coast competitions over the past three years, I was really excited to see some of our new (and some old) dancers bringing up their judging concerns on some of the other threads. These are important issues on the circuit, and I’m glad these conversations are out in the open – I think dancers and teams work too hard for there to be any ambiguity or uncertainty in judging practices.

Below, I’ve listed out some issues that I think are “low hanging fruit” in the judging community – issues that we could fix this season if we all made a collective effort. Before we discuss some of those, I think it’s important to realize that the state of Bhangra judging (at least in US) is not THAT bad. In general, we have knowledgeable people who have the teams’ best interest at heart, and I have never witnessed any obvious explicit or implicit bias during my time as a judge. That being said, there’s a lot of work to do. I’d give the current state of judging practices a B-. We generally get the job done, but a lot of gains to made in improving rubrics and processes.

Now, onto the issues…

· Rubric Scoring: The vast majority of competitions now use simple averages or sums to tally judge scores and rank teams. Personally, I think this is a little crazy. For example, I’m a judge with scores that will range from about 50-99 out of 100. About a third of judges I meet will have score ranges 75-100 out of 100. With simple averages/sums, my scores will weigh about double of the other judge with narrower ranges. Hey, that’s cool with me… but probably a little unfair to the other judge whose opinion should count equal to mine. Solution: re-weigh scores in Excel to correct for differences in score ranges, so that judges have equal influence in a simple sum/average. I think Faizan’s system actually did this, but can’t quite recall right now.

· Deliberation: Generally, committees and judges will put in a fair amount of time into making the rubric just right. We need to do the same for how we want to run deliberation. First of all, decide what the impact of deliberation will be. The standard I see now is any team in the top 5 can move up or down 2 spots based on deliberation (so a first place in points can’t be deliberated out of placing). I have judged competitions where any team can move any number of spots in deliberation. That can’t happen.

There also needs to some process to deliberation. It’s usually only 10-15 minutes of discussion, so we have to decide beforehand what gets discussed. At the very least, committees PLEASEEEE tally the scores before deliberation – there’s no sense in tallying for 7-8 minutes in a 10-15 min session. In putting together a structured deliberation, things you might include:

• Revisiting teams that scored <3-4 points apart
• Discussing the merits of every team in the top 5, or discussing any teams that a judge thinks should have a placing spot but doesn’t because of points
• Going through every judge’s rationale for their top 3 – I think this is really important to give every judge a chance to make their points in deliberation. Often, in this kind of setting, the loudest voices dominate the room, and I’ve seen quieter (often female judges) opinions get squashed. Everyone has the authority and right to make their points heard.

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

· Changing scores after performances: This is pretty common practice these days, and I think for the sake of transparency teams should know if this is allowed at the competition. Inherently, I don’t think this is an obviously unethical thing to do. Most judges prefer the ability to modify scores for past performances as they see more performances, in an effort to create a distribution of scores they will reflects the distribution of performance across teams. On the other hand, a good judge should have seen enough performances to know where on the scale a performance should fall. Personally, I do not change scores after performances are over, and it’s why I maintain a wide range of scores – to give myself room to score another team higher or lower later in the night. If you do allow changing scores, be aware of two issues:

• You won’t be able to get final score tallies until all performances are over if you allow changing until the last performance because judges will hold onto their score sheets. As mentioned before, this eats into deliberation time. PLEASE committees, find some communication channel between judges and their liaisons (text, email, groupme, whatever) to report scores as the performances go on, so you can tally even if people are holding onto score sheets.
• Judges may feel tempted by situations where they may score Team A 80/100 points, and then watch Team B later in the night. In their head, they think Team B was actually worse than Team A, but when they score, they actually come to 85/100 points for Team B. They then go back to Team A’s score sheet and bump them up a few points in random areas to reflect their opinion. This happens all the time. It is NOT the right way to do things.

IMO, I think it’s time we do away with this practice of changing scores. It’s not transparent, and we already have a mechanism for capturing error and “unmeasurable” elements with deliberation. If you think your score is not reflective of a performance, own your mistake in deliberation and make your point. I still think we should able to hold onto our scores for earlier performances as reference, but changing scores just introduces shadiness at multiple steps in the process.

· Post-Comp Judge’s meetings: Some people brought up the notion that post-comp judges meetings are worthless these days – I totally agree. It’s just so hard to get in real feedback over 5-6 minutes, and everyone is just so hurried to get to the afterparty it makes it unenjoyable for everyone. In places like Canada/Toronto where you’re starting comps much earlier in the day, I imagine it’s easier to give teams all the time they want and have good discussion. In the US, comps are usually ending at 9-9:30 PM and we’re struggling to get in just a few words before the venues kick us out and buses leave for the hotel.

At the very least, I think meetings should START, not end with questions from the captains. Usually we are giving teams general feedback or telling them things they already know (e.g. “yeah, I took off some points when you broke a saap on stage” – yeah, no shit). By the time we’re done rambling, there might be 1 minute to ask the questions that teams have been waiting to talk about. We rush answers, tell them to email us with other questions, they never do, and that’s that. Let’s at least get through what teams really want to know if we’re going to suffer through these meetings.

Long post, but the devil is in the details here. If you read all the way through, thank you! I’d love to hear more solutions, or other issues that you’ve noticed during your time on the circuit.


PS – I’ll be judging Big 10 this coming spring. I’ll be bringing up these issues with the committee and other judges, and if you’re reading this and judging at other places this year, I hope you do the same.

 

[mafzal]

Fantastic post. PM'd you! I'm working on a resource packet compiling all of these best practices, tips, and ideas together

 

[MohitoK]

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

These are some great points!


My biggest issue with rubrics at basically every comp is large point categories with no clearly defined scoring breakdown of subcategories. For example, in a "traditional" category that's worth 15 points, I think there should be a clearly defined breakdown of how those 15 points will be awarded/deducted. How many points will be given for satisfying a variety of traditional moves? How many points are given for traditional dancing style elements like nakhra and modeh? etc...


I know some judges are really good about dividing up the categories themselves, but creating a standard for every rubric will hopefully lead to more consistency amongst all the scores and better feedback for teams.

 

[ManrajBajwa]

After judging a bunch of East coast competitions over the past three years, I was really excited to see some of our new (and some old) dancers bringing up their judging concerns on some of the other threads. These are important issues on the circuit, and I’m glad these conversations are out in the open – I think dancers and teams work too hard for there to be any ambiguity or uncertainty in judging practices.

Below, I’ve listed out some issues that I think are “low hanging fruit” in the judging community – issues that we could fix this season if we all made a collective effort. Before we discuss some of those, I think it’s important to realize that the state of Bhangra judging (at least in US) is not THAT bad. In general, we have knowledgeable people who have the teams’ best interest at heart, and I have never witnessed any obvious explicit or implicit bias during my time as a judge. That being said, there’s a lot of work to do. I’d give the current state of judging practices a B-. We generally get the job done, but a lot of gains to made in improving rubrics and processes.

Now, onto the issues…

· Rubric Scoring: The vast majority of competitions now use simple averages or sums to tally judge scores and rank teams. Personally, I think this is a little crazy. For example, I’m a judge with scores that will range from about 50-99 out of 100. About a third of judges I meet will have score ranges 75-100 out of 100. With simple averages/sums, my scores will weigh about double of the other judge with narrower ranges. Hey, that’s cool with me… but probably a little unfair to the other judge whose opinion should count equal to mine. Solution: re-weigh scores in Excel to correct for differences in score ranges, so that judges have equal influence in a simple sum/average. I think Faizan’s system actually did this, but can’t quite recall right now.

· Deliberation: Generally, committees and judges will put in a fair amount of time into making the rubric just right. We need to do the same for how we want to run deliberation. First of all, decide what the impact of deliberation will be. The standard I see now is any team in the top 5 can move up or down 2 spots based on deliberation (so a first place in points can’t be deliberated out of placing). I have judged competitions where any team can move any number of spots in deliberation. That can’t happen.

There also needs to some process to deliberation. It’s usually only 10-15 minutes of discussion, so we have to decide beforehand what gets discussed. At the very least, committees PLEASEEEE tally the scores before deliberation – there’s no sense in tallying for 7-8 minutes in a 10-15 min session. In putting together a structured deliberation, things you might include:

• Revisiting teams that scored <3-4 points apart
• Discussing the merits of every team in the top 5, or discussing any teams that a judge thinks should have a placing spot but doesn’t because of points
• Going through every judge’s rationale for their top 3 – I think this is really important to give every judge a chance to make their points in deliberation. Often, in this kind of setting, the loudest voices dominate the room, and I’ve seen quieter (often female judges) opinions get squashed. Everyone has the authority and right to make their points heard.

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

· Changing scores after performances: This is pretty common practice these days, and I think for the sake of transparency teams should know if this is allowed at the competition. Inherently, I don’t think this is an obviously unethical thing to do. Most judges prefer the ability to modify scores for past performances as they see more performances, in an effort to create a distribution of scores they will reflects the distribution of performance across teams. On the other hand, a good judge should have seen enough performances to know where on the scale a performance should fall. Personally, I do not change scores after performances are over, and it’s why I maintain a wide range of scores – to give myself room to score another team higher or lower later in the night. If you do allow changing scores, be aware of two issues:

• You won’t be able to get final score tallies until all performances are over if you allow changing until the last performance because judges will hold onto their score sheets. As mentioned before, this eats into deliberation time. PLEASE committees, find some communication channel between judges and their liaisons (text, email, groupme, whatever) to report scores as the performances go on, so you can tally even if people are holding onto score sheets.
• Judges may feel tempted by situations where they may score Team A 80/100 points, and then watch Team B later in the night. In their head, they think Team B was actually worse than Team A, but when they score, they actually come to 85/100 points for Team B. They then go back to Team A’s score sheet and bump them up a few points in random areas to reflect their opinion. This happens all the time. It is NOT the right way to do things.

IMO, I think it’s time we do away with this practice of changing scores. It’s not transparent, and we already have a mechanism for capturing error and “unmeasurable” elements with deliberation. If you think your score is not reflective of a performance, own your mistake in deliberation and make your point. I still think we should able to hold onto our scores for earlier performances as reference, but changing scores just introduces shadiness at multiple steps in the process.

· Post-Comp Judge’s meetings: Some people brought up the notion that post-comp judges meetings are worthless these days – I totally agree. It’s just so hard to get in real feedback over 5-6 minutes, and everyone is just so hurried to get to the afterparty it makes it unenjoyable for everyone. In places like Canada/Toronto where you’re starting comps much earlier in the day, I imagine it’s easier to give teams all the time they want and have good discussion. In the US, comps are usually ending at 9-9:30 PM and we’re struggling to get in just a few words before the venues kick us out and buses leave for the hotel.

At the very least, I think meetings should START, not end with questions from the captains. Usually we are giving teams general feedback or telling them things they already know (e.g. “yeah, I took off some points when you broke a saap on stage” – yeah, no shit). By the time we’re done rambling, there might be 1 minute to ask the questions that teams have been waiting to talk about. We rush answers, tell them to email us with other questions, they never do, and that’s that. Let’s at least get through what teams really want to know if we’re going to suffer through these meetings.

Long post, but the devil is in the details here. If you read all the way through, thank you! I’d love to hear more solutions, or other issues that you’ve noticed during your time on the circuit.


PS – I’ll be judging Big 10 this coming spring. I’ll be bringing up these issues with the committee and other judges, and if you’re reading this and judging at other places this year, I hope you do the same.

Yea and also when awarding points judges should consider with the number of dancers on stage coz sometimes is not fair that u have 12 dancers and some other team have 16 dancers and the team with 12 people wins simply because they are much synchro taking to the account that they have less people to synchro.

 

[scaplash]

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

I like a lot of the points you brought up, and one thing that makes it easier for me to judge is when I am watching a team, I like to actually divide it by segments and judge individual segments on components of the rubric. Its certainly not perfect as some rubric components are definitely holistic. But I think it makes it easier to give teams feedback in terms of what segments worked and which ones didn't for whatever reasons. It also I think helps combat bias when judging a team, often time a team might 5 out of 6 pretty clean segments, but if they have a segment that is memorably unclean, it can be easy to let that trickle into your judgment and deflate their scores more than necessary or vice versa. The hype of one segment can make you forget that a bunch of other segments were actually pretty weak.

 

[srikarran]

After judging a bunch of East coast competitions over the past three years, I was really excited to see some of our new (and some old) dancers bringing up their judging concerns on some of the other threads. These are important issues on the circuit, and I’m glad these conversations are out in the open – I think dancers and teams work too hard for there to be any ambiguity or uncertainty in judging practices.

Below, I’ve listed out some issues that I think are “low hanging fruit” in the judging community – issues that we could fix this season if we all made a collective effort. Before we discuss some of those, I think it’s important to realize that the state of Bhangra judging (at least in US) is not THAT bad. In general, we have knowledgeable people who have the teams’ best interest at heart, and I have never witnessed any obvious explicit or implicit bias during my time as a judge. That being said, there’s a lot of work to do. I’d give the current state of judging practices a B-. We generally get the job done, but a lot of gains to made in improving rubrics and processes.

Now, onto the issues…

· Rubric Scoring: The vast majority of competitions now use simple averages or sums to tally judge scores and rank teams. Personally, I think this is a little crazy. For example, I’m a judge with scores that will range from about 50-99 out of 100. About a third of judges I meet will have score ranges 75-100 out of 100. With simple averages/sums, my scores will weigh about double of the other judge with narrower ranges. Hey, that’s cool with me… but probably a little unfair to the other judge whose opinion should count equal to mine. Solution: re-weigh scores in Excel to correct for differences in score ranges, so that judges have equal influence in a simple sum/average. I think Faizan’s system actually did this, but can’t quite recall right now.

· Deliberation: Generally, committees and judges will put in a fair amount of time into making the rubric just right. We need to do the same for how we want to run deliberation. First of all, decide what the impact of deliberation will be. The standard I see now is any team in the top 5 can move up or down 2 spots based on deliberation (so a first place in points can’t be deliberated out of placing). I have judged competitions where any team can move any number of spots in deliberation. That can’t happen.

There also needs to some process to deliberation. It’s usually only 10-15 minutes of discussion, so we have to decide beforehand what gets discussed. At the very least, committees PLEASEEEE tally the scores before deliberation – there’s no sense in tallying for 7-8 minutes in a 10-15 min session. In putting together a structured deliberation, things you might include:

• Revisiting teams that scored <3-4 points apart
• Discussing the merits of every team in the top 5, or discussing any teams that a judge thinks should have a placing spot but doesn’t because of points
• Going through every judge’s rationale for their top 3 – I think this is really important to give every judge a chance to make their points in deliberation. Often, in this kind of setting, the loudest voices dominate the room, and I’ve seen quieter (often female judges) opinions get squashed. Everyone has the authority and right to make their points heard.

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

· Changing scores after performances: This is pretty common practice these days, and I think for the sake of transparency teams should know if this is allowed at the competition. Inherently, I don’t think this is an obviously unethical thing to do. Most judges prefer the ability to modify scores for past performances as they see more performances, in an effort to create a distribution of scores they will reflects the distribution of performance across teams. On the other hand, a good judge should have seen enough performances to know where on the scale a performance should fall. Personally, I do not change scores after performances are over, and it’s why I maintain a wide range of scores – to give myself room to score another team higher or lower later in the night. If you do allow changing scores, be aware of two issues:

• You won’t be able to get final score tallies until all performances are over if you allow changing until the last performance because judges will hold onto their score sheets. As mentioned before, this eats into deliberation time. PLEASE committees, find some communication channel between judges and their liaisons (text, email, groupme, whatever) to report scores as the performances go on, so you can tally even if people are holding onto score sheets.
• Judges may feel tempted by situations where they may score Team A 80/100 points, and then watch Team B later in the night. In their head, they think Team B was actually worse than Team A, but when they score, they actually come to 85/100 points for Team B. They then go back to Team A’s score sheet and bump them up a few points in random areas to reflect their opinion. This happens all the time. It is NOT the right way to do things.

IMO, I think it’s time we do away with this practice of changing scores. It’s not transparent, and we already have a mechanism for capturing error and “unmeasurable” elements with deliberation. If you think your score is not reflective of a performance, own your mistake in deliberation and make your point. I still think we should able to hold onto our scores for earlier performances as reference, but changing scores just introduces shadiness at multiple steps in the process.

· Post-Comp Judge’s meetings: Some people brought up the notion that post-comp judges meetings are worthless these days – I totally agree. It’s just so hard to get in real feedback over 5-6 minutes, and everyone is just so hurried to get to the afterparty it makes it unenjoyable for everyone. In places like Canada/Toronto where you’re starting comps much earlier in the day, I imagine it’s easier to give teams all the time they want and have good discussion. In the US, comps are usually ending at 9-9:30 PM and we’re struggling to get in just a few words before the venues kick us out and buses leave for the hotel.

At the very least, I think meetings should START, not end with questions from the captains. Usually we are giving teams general feedback or telling them things they already know (e.g. “yeah, I took off some points when you broke a saap on stage” – yeah, no shit). By the time we’re done rambling, there might be 1 minute to ask the questions that teams have been waiting to talk about. We rush answers, tell them to email us with other questions, they never do, and that’s that. Let’s at least get through what teams really want to know if we’re going to suffer through these meetings.

Long post, but the devil is in the details here. If you read all the way through, thank you! I’d love to hear more solutions, or other issues that you’ve noticed during your time on the circuit.


PS – I’ll be judging Big 10 this coming spring. I’ll be bringing up these issues with the committee and other judges, and if you’re reading this and judging at other places this year, I hope you do the same.

Yea and also when awarding points judges should consider with the number of dancers on stage coz sometimes is not fair that u have 12 dancers and some other team have 16 dancers and the team with 12 people wins simply because they are much synchro taking to the account that they have less people to synchro.

I Disagree with this. If Team A has 16 dancers (who aren't in sync) and Team B has 12 (who are in sync), as a judge you must view it as "Team A wasn't in sync while Team B was."


Competitions have performance rules where they have a range for the number of dancers the performance must have. Like "8-20 dancers". 8/12/16/20 dancers should not be considered when a rubric has no place for it.

 

[ManrajBajwa]

After judging a bunch of East coast competitions over the past three years, I was really excited to see some of our new (and some old) dancers bringing up their judging concerns on some of the other threads. These are important issues on the circuit, and I’m glad these conversations are out in the open – I think dancers and teams work too hard for there to be any ambiguity or uncertainty in judging practices.

Below, I’ve listed out some issues that I think are “low hanging fruit” in the judging community – issues that we could fix this season if we all made a collective effort. Before we discuss some of those, I think it’s important to realize that the state of Bhangra judging (at least in US) is not THAT bad. In general, we have knowledgeable people who have the teams’ best interest at heart, and I have never witnessed any obvious explicit or implicit bias during my time as a judge. That being said, there’s a lot of work to do. I’d give the current state of judging practices a B-. We generally get the job done, but a lot of gains to made in improving rubrics and processes.

Now, onto the issues…

· Rubric Scoring: The vast majority of competitions now use simple averages or sums to tally judge scores and rank teams. Personally, I think this is a little crazy. For example, I’m a judge with scores that will range from about 50-99 out of 100. About a third of judges I meet will have score ranges 75-100 out of 100. With simple averages/sums, my scores will weigh about double of the other judge with narrower ranges. Hey, that’s cool with me… but probably a little unfair to the other judge whose opinion should count equal to mine. Solution: re-weigh scores in Excel to correct for differences in score ranges, so that judges have equal influence in a simple sum/average. I think Faizan’s system actually did this, but can’t quite recall right now.

· Deliberation: Generally, committees and judges will put in a fair amount of time into making the rubric just right. We need to do the same for how we want to run deliberation. First of all, decide what the impact of deliberation will be. The standard I see now is any team in the top 5 can move up or down 2 spots based on deliberation (so a first place in points can’t be deliberated out of placing). I have judged competitions where any team can move any number of spots in deliberation. That can’t happen.

There also needs to some process to deliberation. It’s usually only 10-15 minutes of discussion, so we have to decide beforehand what gets discussed. At the very least, committees PLEASEEEE tally the scores before deliberation – there’s no sense in tallying for 7-8 minutes in a 10-15 min session. In putting together a structured deliberation, things you might include:

• Revisiting teams that scored <3-4 points apart
• Discussing the merits of every team in the top 5, or discussing any teams that a judge thinks should have a placing spot but doesn’t because of points
• Going through every judge’s rationale for their top 3 – I think this is really important to give every judge a chance to make their points in deliberation. Often, in this kind of setting, the loudest voices dominate the room, and I’ve seen quieter (often female judges) opinions get squashed. Everyone has the authority and right to make their points heard.

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

· Changing scores after performances: This is pretty common practice these days, and I think for the sake of transparency teams should know if this is allowed at the competition. Inherently, I don’t think this is an obviously unethical thing to do. Most judges prefer the ability to modify scores for past performances as they see more performances, in an effort to create a distribution of scores they will reflects the distribution of performance across teams. On the other hand, a good judge should have seen enough performances to know where on the scale a performance should fall. Personally, I do not change scores after performances are over, and it’s why I maintain a wide range of scores – to give myself room to score another team higher or lower later in the night. If you do allow changing scores, be aware of two issues:

• You won’t be able to get final score tallies until all performances are over if you allow changing until the last performance because judges will hold onto their score sheets. As mentioned before, this eats into deliberation time. PLEASE committees, find some communication channel between judges and their liaisons (text, email, groupme, whatever) to report scores as the performances go on, so you can tally even if people are holding onto score sheets.
• Judges may feel tempted by situations where they may score Team A 80/100 points, and then watch Team B later in the night. In their head, they think Team B was actually worse than Team A, but when they score, they actually come to 85/100 points for Team B. They then go back to Team A’s score sheet and bump them up a few points in random areas to reflect their opinion. This happens all the time. It is NOT the right way to do things.

IMO, I think it’s time we do away with this practice of changing scores. It’s not transparent, and we already have a mechanism for capturing error and “unmeasurable” elements with deliberation. If you think your score is not reflective of a performance, own your mistake in deliberation and make your point. I still think we should able to hold onto our scores for earlier performances as reference, but changing scores just introduces shadiness at multiple steps in the process.

· Post-Comp Judge’s meetings: Some people brought up the notion that post-comp judges meetings are worthless these days – I totally agree. It’s just so hard to get in real feedback over 5-6 minutes, and everyone is just so hurried to get to the afterparty it makes it unenjoyable for everyone. In places like Canada/Toronto where you’re starting comps much earlier in the day, I imagine it’s easier to give teams all the time they want and have good discussion. In the US, comps are usually ending at 9-9:30 PM and we’re struggling to get in just a few words before the venues kick us out and buses leave for the hotel.

At the very least, I think meetings should START, not end with questions from the captains. Usually we are giving teams general feedback or telling them things they already know (e.g. “yeah, I took off some points when you broke a saap on stage” – yeah, no shit). By the time we’re done rambling, there might be 1 minute to ask the questions that teams have been waiting to talk about. We rush answers, tell them to email us with other questions, they never do, and that’s that. Let’s at least get through what teams really want to know if we’re going to suffer through these meetings.

Long post, but the devil is in the details here. If you read all the way through, thank you! I’d love to hear more solutions, or other issues that you’ve noticed during your time on the circuit.


PS – I’ll be judging Big 10 this coming spring. I’ll be bringing up these issues with the committee and other judges, and if you’re reading this and judging at other places this year, I hope you do the same.

Yea and also when awarding points judges should consider with the number of dancers on stage coz sometimes is not fair that u have 12 dancers and some other team have 16 dancers and the team with 12 people wins simply because they are much synchro taking to the account that they have less people to synchro.

I Disagree with this. If Team A has 16 dancers (who aren't in sync) and Team B has 12 (who are in sync), as a judge you must view it as "Team A wasn't in sync while Team B was."


Competitions have performance rules where they have a range for the number of dancers the performance must have. Like "8-20 dancers". 8/12/16/20 dancers should not be considered when a rubric has no place for it.

Then if comps say 8-20 dancers. Then the team would bring like 8 of their best dancers and win the comp where else teams who have like 12 dancers who are good but since there are more people there might be more mistakes as well.So should we consider the number?

 

[srikarran]

After judging a bunch of East coast competitions over the past three years, I was really excited to see some of our new (and some old) dancers bringing up their judging concerns on some of the other threads. These are important issues on the circuit, and I’m glad these conversations are out in the open – I think dancers and teams work too hard for there to be any ambiguity or uncertainty in judging practices.

Below, I’ve listed out some issues that I think are “low hanging fruit” in the judging community – issues that we could fix this season if we all made a collective effort. Before we discuss some of those, I think it’s important to realize that the state of Bhangra judging (at least in US) is not THAT bad. In general, we have knowledgeable people who have the teams’ best interest at heart, and I have never witnessed any obvious explicit or implicit bias during my time as a judge. That being said, there’s a lot of work to do. I’d give the current state of judging practices a B-. We generally get the job done, but a lot of gains to made in improving rubrics and processes.

Now, onto the issues…

· Rubric Scoring: The vast majority of competitions now use simple averages or sums to tally judge scores and rank teams. Personally, I think this is a little crazy. For example, I’m a judge with scores that will range from about 50-99 out of 100. About a third of judges I meet will have score ranges 75-100 out of 100. With simple averages/sums, my scores will weigh about double of the other judge with narrower ranges. Hey, that’s cool with me… but probably a little unfair to the other judge whose opinion should count equal to mine. Solution: re-weigh scores in Excel to correct for differences in score ranges, so that judges have equal influence in a simple sum/average. I think Faizan’s system actually did this, but can’t quite recall right now.

· Deliberation: Generally, committees and judges will put in a fair amount of time into making the rubric just right. We need to do the same for how we want to run deliberation. First of all, decide what the impact of deliberation will be. The standard I see now is any team in the top 5 can move up or down 2 spots based on deliberation (so a first place in points can’t be deliberated out of placing). I have judged competitions where any team can move any number of spots in deliberation. That can’t happen.

There also needs to some process to deliberation. It’s usually only 10-15 minutes of discussion, so we have to decide beforehand what gets discussed. At the very least, committees PLEASEEEE tally the scores before deliberation – there’s no sense in tallying for 7-8 minutes in a 10-15 min session. In putting together a structured deliberation, things you might include:

• Revisiting teams that scored <3-4 points apart
• Discussing the merits of every team in the top 5, or discussing any teams that a judge thinks should have a placing spot but doesn’t because of points
• Going through every judge’s rationale for their top 3 – I think this is really important to give every judge a chance to make their points in deliberation. Often, in this kind of setting, the loudest voices dominate the room, and I’ve seen quieter (often female judges) opinions get squashed. Everyone has the authority and right to make their points heard.

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

· Changing scores after performances: This is pretty common practice these days, and I think for the sake of transparency teams should know if this is allowed at the competition. Inherently, I don’t think this is an obviously unethical thing to do. Most judges prefer the ability to modify scores for past performances as they see more performances, in an effort to create a distribution of scores they will reflects the distribution of performance across teams. On the other hand, a good judge should have seen enough performances to know where on the scale a performance should fall. Personally, I do not change scores after performances are over, and it’s why I maintain a wide range of scores – to give myself room to score another team higher or lower later in the night. If you do allow changing scores, be aware of two issues:

• You won’t be able to get final score tallies until all performances are over if you allow changing until the last performance because judges will hold onto their score sheets. As mentioned before, this eats into deliberation time. PLEASE committees, find some communication channel between judges and their liaisons (text, email, groupme, whatever) to report scores as the performances go on, so you can tally even if people are holding onto score sheets.
• Judges may feel tempted by situations where they may score Team A 80/100 points, and then watch Team B later in the night. In their head, they think Team B was actually worse than Team A, but when they score, they actually come to 85/100 points for Team B. They then go back to Team A’s score sheet and bump them up a few points in random areas to reflect their opinion. This happens all the time. It is NOT the right way to do things.

IMO, I think it’s time we do away with this practice of changing scores. It’s not transparent, and we already have a mechanism for capturing error and “unmeasurable” elements with deliberation. If you think your score is not reflective of a performance, own your mistake in deliberation and make your point. I still think we should able to hold onto our scores for earlier performances as reference, but changing scores just introduces shadiness at multiple steps in the process.

· Post-Comp Judge’s meetings: Some people brought up the notion that post-comp judges meetings are worthless these days – I totally agree. It’s just so hard to get in real feedback over 5-6 minutes, and everyone is just so hurried to get to the afterparty it makes it unenjoyable for everyone. In places like Canada/Toronto where you’re starting comps much earlier in the day, I imagine it’s easier to give teams all the time they want and have good discussion. In the US, comps are usually ending at 9-9:30 PM and we’re struggling to get in just a few words before the venues kick us out and buses leave for the hotel.

At the very least, I think meetings should START, not end with questions from the captains. Usually we are giving teams general feedback or telling them things they already know (e.g. “yeah, I took off some points when you broke a saap on stage” – yeah, no shit). By the time we’re done rambling, there might be 1 minute to ask the questions that teams have been waiting to talk about. We rush answers, tell them to email us with other questions, they never do, and that’s that. Let’s at least get through what teams really want to know if we’re going to suffer through these meetings.

Long post, but the devil is in the details here. If you read all the way through, thank you! I’d love to hear more solutions, or other issues that you’ve noticed during your time on the circuit.


PS – I’ll be judging Big 10 this coming spring. I’ll be bringing up these issues with the committee and other judges, and if you’re reading this and judging at other places this year, I hope you do the same.

Yea and also when awarding points judges should consider with the number of dancers on stage coz sometimes is not fair that u have 12 dancers and some other team have 16 dancers and the team with 12 people wins simply because they are much synchro taking to the account that they have less people to synchro.

I Disagree with this. If Team A has 16 dancers (who aren't in sync) and Team B has 12 (who are in sync), as a judge you must view it as "Team A wasn't in sync while Team B was."


Competitions have performance rules where they have a range for the number of dancers the performance must have. Like "8-20 dancers". 8/12/16/20 dancers should not be considered when a rubric has no place for it.

Then if comps say 8-20 dancers. Then the team would bring like 8 of their best dancers and win the comp where else teams who have like 12 dancers who are good but since there are more people there might be more mistakes as well.So should we consider the number?

Team's still go with more dancers than the minimum. With 8 members you are possibly limiting your team from being more creative with formations, which have a place in the rubric. Team's also dance for more than winning. Leaving an impression, pushing the limits, doing something new and creative.


If team's were to be judged on number of dancer's, might as well have it as separate categories.

 

[SGupta7]

After judging a bunch of East coast competitions over the past three years, I was really excited to see some of our new (and some old) dancers bringing up their judging concerns on some of the other threads. These are important issues on the circuit, and I’m glad these conversations are out in the open – I think dancers and teams work too hard for there to be any ambiguity or uncertainty in judging practices.

Below, I’ve listed out some issues that I think are “low hanging fruit” in the judging community – issues that we could fix this season if we all made a collective effort. Before we discuss some of those, I think it’s important to realize that the state of Bhangra judging (at least in US) is not THAT bad. In general, we have knowledgeable people who have the teams’ best interest at heart, and I have never witnessed any obvious explicit or implicit bias during my time as a judge. That being said, there’s a lot of work to do. I’d give the current state of judging practices a B-. We generally get the job done, but a lot of gains to made in improving rubrics and processes.

Now, onto the issues…

· Rubric Scoring: The vast majority of competitions now use simple averages or sums to tally judge scores and rank teams. Personally, I think this is a little crazy. For example, I’m a judge with scores that will range from about 50-99 out of 100. About a third of judges I meet will have score ranges 75-100 out of 100. With simple averages/sums, my scores will weigh about double of the other judge with narrower ranges. Hey, that’s cool with me… but probably a little unfair to the other judge whose opinion should count equal to mine. Solution: re-weigh scores in Excel to correct for differences in score ranges, so that judges have equal influence in a simple sum/average. I think Faizan’s system actually did this, but can’t quite recall right now.

· Deliberation: Generally, committees and judges will put in a fair amount of time into making the rubric just right. We need to do the same for how we want to run deliberation. First of all, decide what the impact of deliberation will be. The standard I see now is any team in the top 5 can move up or down 2 spots based on deliberation (so a first place in points can’t be deliberated out of placing). I have judged competitions where any team can move any number of spots in deliberation. That can’t happen.

There also needs to some process to deliberation. It’s usually only 10-15 minutes of discussion, so we have to decide beforehand what gets discussed. At the very least, committees PLEASEEEE tally the scores before deliberation – there’s no sense in tallying for 7-8 minutes in a 10-15 min session. In putting together a structured deliberation, things you might include:

• Revisiting teams that scored <3-4 points apart
• Discussing the merits of every team in the top 5, or discussing any teams that a judge thinks should have a placing spot but doesn’t because of points
• Going through every judge’s rationale for their top 3 – I think this is really important to give every judge a chance to make their points in deliberation. Often, in this kind of setting, the loudest voices dominate the room, and I’ve seen quieter (often female judges) opinions get squashed. Everyone has the authority and right to make their points heard.

· Rubric creation: There have been a few times where committees will unilaterally determine the rubric without judges input. I don’t think this is a good idea. For one thing, judges are mostly busy, young professionals who won’t really put that much time into critically thinking about the rubric unless there is some structured time to discuss it. That means, for many judges, the first time they REALLY are thinking hard about the rubric is at the competition. Not good.

I’m a big fan of having a draft rubric that the committee comes up with, and then the judges can edit as needed. That way, we can understand if the committee is looking to reward more innovative/risky performances or stay on the folk/conservative side of things. When creating the rubric, each section should be as discrete as possible. Just about every rubric I’ve used has significant overlap between sections, where points are allocated to creativity in 2-3 areas, or where several very different criteria are lumped into one section. This is a BIG place where subjectivity comes into play. If energy, nakhra, and synchronization are all in the same section, how many points do I allocate to each of these? Different judges will have different answers, and I think that creates a suboptimal measurement of the performance.

When rubrics are done, committees should have judges practice with them and score old performances on Youtube. For most comps, the first time a judge actually uses a rubric will be the first team of the night, which is unfair. Every rubric has a learning curve, and we should be practicing using the rubric’s logic long before the teams take the stage. Also, it’s almost inevitable that I find things that I want to change about the rubric after I actually use it to score a team.

· Changing scores after performances: This is pretty common practice these days, and I think for the sake of transparency teams should know if this is allowed at the competition. Inherently, I don’t think this is an obviously unethical thing to do. Most judges prefer the ability to modify scores for past performances as they see more performances, in an effort to create a distribution of scores they will reflects the distribution of performance across teams. On the other hand, a good judge should have seen enough performances to know where on the scale a performance should fall. Personally, I do not change scores after performances are over, and it’s why I maintain a wide range of scores – to give myself room to score another team higher or lower later in the night. If you do allow changing scores, be aware of two issues:

• You won’t be able to get final score tallies until all performances are over if you allow changing until the last performance because judges will hold onto their score sheets. As mentioned before, this eats into deliberation time. PLEASE committees, find some communication channel between judges and their liaisons (text, email, groupme, whatever) to report scores as the performances go on, so you can tally even if people are holding onto score sheets.
• Judges may feel tempted by situations where they may score Team A 80/100 points, and then watch Team B later in the night. In their head, they think Team B was actually worse than Team A, but when they score, they actually come to 85/100 points for Team B. They then go back to Team A’s score sheet and bump them up a few points in random areas to reflect their opinion. This happens all the time. It is NOT the right way to do things.

IMO, I think it’s time we do away with this practice of changing scores. It’s not transparent, and we already have a mechanism for capturing error and “unmeasurable” elements with deliberation. If you think your score is not reflective of a performance, own your mistake in deliberation and make your point. I still think we should able to hold onto our scores for earlier performances as reference, but changing scores just introduces shadiness at multiple steps in the process.

· Post-Comp Judge’s meetings: Some people brought up the notion that post-comp judges meetings are worthless these days – I totally agree. It’s just so hard to get in real feedback over 5-6 minutes, and everyone is just so hurried to get to the afterparty it makes it unenjoyable for everyone. In places like Canada/Toronto where you’re starting comps much earlier in the day, I imagine it’s easier to give teams all the time they want and have good discussion. In the US, comps are usually ending at 9-9:30 PM and we’re struggling to get in just a few words before the venues kick us out and buses leave for the hotel.

At the very least, I think meetings should START, not end with questions from the captains. Usually we are giving teams general feedback or telling them things they already know (e.g. “yeah, I took off some points when you broke a saap on stage” – yeah, no shit). By the time we’re done rambling, there might be 1 minute to ask the questions that teams have been waiting to talk about. We rush answers, tell them to email us with other questions, they never do, and that’s that. Let’s at least get through what teams really want to know if we’re going to suffer through these meetings.

Long post, but the devil is in the details here. If you read all the way through, thank you! I’d love to hear more solutions, or other issues that you’ve noticed during your time on the circuit.


PS – I’ll be judging Big 10 this coming spring. I’ll be bringing up these issues with the committee and other judges, and if you’re reading this and judging at other places this year, I hope you do the same.

Yea and also when awarding points judges should consider with the number of dancers on stage coz sometimes is not fair that u have 12 dancers and some other team have 16 dancers and the team with 12 people wins simply because they are much synchro taking to the account that they have less people to synchro.

I Disagree with this. If Team A has 16 dancers (who aren't in sync) and Team B has 12 (who are in sync), as a judge you must view it as "Team A wasn't in sync while Team B was."


Competitions have performance rules where they have a range for the number of dancers the performance must have. Like "8-20 dancers". 8/12/16/20 dancers should not be considered when a rubric has no place for it.

Then if comps say 8-20 dancers. Then the team would bring like 8 of their best dancers and win the comp where else teams who have like 12 dancers who are good but since there are more people there might be more mistakes as well.So should we consider the number?

Team's still go with more dancers than the minimum. With 8 members you are possibly limiting your team from being more creative with formations, which have a place in the rubric. Team's also dance for more than winning. Leaving an impression, pushing the limits, doing something new and creative.


If team's were to be judged on number of dancer's, might as well have it as separate categories.

There are pros and cons to bringing fewer or more dancers. With fewer dancers you can potentially be cleaner, but your formations may not be as creative as a team that brings 16 dancers. I think these things tend to average themselves out in the rubric so in short, no, number of dancers shouldn't matter. Most music teams bring 12-16 dancers anyway so the disparity isn't that big.

 

[arnab]

Practically, number of dancers is not a huge issue. Teams generally understand that there is a lot to gain from bringing more dancers in terms of formations and creativity, but weigh that with the risk of a messier set. Most judges (including myself) will not correct synchronization scores for number of dancers, for the reasons some others have given. There's no good systematic way to do that, and as I mentioned, bringing 16-20 dancers is a calculated risk that teams should prepare for.


All in all, if you want to bring 16 to a competition, don't worry... 16 person sets place all the time. Just be careful to make sure your dancers don't look cramped on a stage with small dimensions - that's where I see points docked most of the time for larger teams.

 

[Archana]

Great points Arnab! Here are some more based on my experience.

So one thing I have experimented with judging is having a deliberation with all the judges well before the competition, with the chair of judging of course. What we did was use 2 teams from the previous year of that competition, watched their videos and judge on that rubric. It was like our homework. We then uploaded it into a doc sharing file so that all the judges and chair could see. Then we had a google video call and we looked at each judges score for each team and see where we were all standing. Upon deliberation with each of us judges, we found a common ground on how to we want tackle each part of the rubric. Yes, preferences come into play, but after finding a common ground, all of our scores for the competition were around the same area. This actually worked really well and highly recommend it to competitions. Also, as hard as it may seem, please please, competitions need to figure out what kind of competition they want, folk or modern or a mix. Then find judges based on that. At one competition, after all of us judges were chosen, we fixed up the rubric and had a talk with the competition of what they want. In the end they decided to go folk/traditional. The line-up kind of changed where some teams dropped out and others were brought in. However, one of the judges was more so on the modern side and he was still the judge. This, then, brings in a conflict of interest because his scores varied widely.

Also, in regards to pre-comp judges meeting, I highly think that is a waste of time. This needs to be done (based on the experimented plan) after the judges deliberate on the rubric with all of the captains via video chat. Leave the competition weekend for the teams to prep, etc. All questions should be answered weeks before the competition.

Judges should not change scores. Like I said, a deliberation between judges and how to score with the rubric should fix that.

 

[Nimit]

One quick point about how to balance judges who score higher or lower. You can make each individual judges score not out of 100 but out of the highest score that judge gave at that competition. Example:

Raw Scores:
Team 1
Judge A - 95/100
Judge B - 65/100
Judge C - 80/100

Team 2
Judge A - 80/100
Judge B - 50/100
Judge C - 85/100

Adjusted Scores:
Team 1
Judge A - 95/95
Judge B - 65/65
Judge C - 80/85

Team 2
Judge A - 80/95
Judge B - 50/65
Judge C - 85/85


Averaging these new values might give a more accurate and balanced representation!

 

[vaibhav]

Great Points, Arnab! I agree with so many of your points on effective judging strategies. It would be great to see many more competitions begin to implement these things in the near future!
I would just like to add to and reiterate several of the points I have noticed to be effective at many competitions and especially at Bruin, where I've had the chance to run judging for the past four years.


Selection of Judges

• Choose Judges that understand what your competition is about and what your committee wants/expects from its teams in terms of performance.
• We at Bruin are huge fans of large judging panels (6-7 judges). Larger sample size, assuming that the judges judge to your competitions rubric and beliefs, should ideally provide more precision and accuracy in competition results.
• Look to diversify your judges lineup (i.e. male & female representation, younger & older judges, etc.). Teams can get some great, well-rounded feedback from a more diverse panel, I've noticed.

Pre-Comp Judges Meeting

• Typically hold a Google Hangout Judges meeting at least 1.5-2 months before the competition date.
• Pro tip- If your university/school/college email is associated with google, use that email to create a Google Hangout with 15 users!
• Give teams an ample amount of time to prepare a routine to cater to your competition, rubric, and judges (whether they all be traditional, modern, mixed, etc.)
• I know some competitions have a free Q&A sessions while others have teams submit Q's before hand and have judges answer them at a later date, but personally I'm not a big fan of the latter. What has seemed to be effective at the Bruin pre-judges meeting and also at Blowout's Pre-Judges meeting from this year is Structuring the meeting by having judges go through each judging category (i.e. Creativity, Synchronization, etc.) by describing their expectations for a team that specific category, and what they expect from a first place winning team at that competition in the context of that category. This will provide teams a clear understanding of what each judge expects.

Judges Group-Me/Post Judges Meeting Q's

• Keep a Group-Me or What's App chat going with your judges. Teams will have hella questions even after the judges meeting, so keep a way to relay messages between teams and judges during this period.
• Make a google form or some similar system where teams can submit their personal, private questions regarding judging that they may not wish to share with the other teams during the Judges Meeting.

Rubric Creation

• I am not a huge fan of checklist type rubrics where every category is divided into many subcategories specifically detailing point allocations and deductions. Though it is awesome and helpful to teams, I feel that's what the Pre Competition Google Hangout judges meeting is for. Keeping a rubric that is too specific, it seems at times, can cause teams to become too complacent with their and treat the rubric like a checklist which in a sense restricts creativity both at the competition and in the circuit in general. By saying this, I definitely am not implying that there shouldn't be standardization in judging and point allocation. I'm just pushing for more holistic, open-ended rubrics.

• Deductions - Some competitions keep a deductions category in their rubric for things such as vardi malfunctions or prop malfunctions (i.e. saap breaking). I personally am not a huge fan of this. Instead of keeping a category for vardi/prop deductions, what we tried at Bruin last year was to emphasize the "Presentation" category which encompassed both of these things. Teams would be rewarded, as opposed to docked, if everything in terms of their presentation, which includes vardis, props, music,etc. was ok. Having a category where teams can earn points, as opposed to lose points, for these things may push teams to be more creative and risky in their routine and ideas, which may involve vardis and/or props.

Rubric Scoring!!

• Bruin has for the past several years used an aspect of Faizan's WBBC approach with normalizing scores. Normalizing scores provides teams with a much easier way to follow their scores in relation to the other teams from that competition. I can gladly send/upload the template of the Bruin Judging Excel sheets if anybody would like to mess around with these see how they were created!

• Judges: Please, if possible, judge with large spreads. After closely following judging sheets at Bruin for the past several years, it seems that the judges who have scored with a standard deviation of around 8-10 points seem to be the most precise in getting the order placings relatively correct.
• Dropping Scores - Teams will always want to ensure that subjectivity and biases aren't playing a role in the judging process. Of course all competitions, including Bruin, will try to select judges who are honest and who will judge the competition without taking into consideration any personal affiliations and relationships with the competing teams. Still, however, it's pretty damn important to keep some protocols that protect the teams. For this, at Bruin, we drop the Lowest Normalized Score from each team (assuming there are 6 judges). Dropping normalized scores (as opposed to raw scores) ensures that a judge who, for example, may on average score lower or higher than the other judges on the panel won't have his or her scores discarded solely for the reason of them scoring at a different mean or average. It also gives teams a peace of mind that there won't be a judge that is out to get them or that may not be a fan of their particular style of bhangra. In a way it protects both judges and teams, and keeps the competition fair!

• Side note: At Bruin, after analyzing the scores over the last two years, dropping the lowest scores has 1) Not Changed placings either year 2) Only changed one overall ranking: in 2015, the team who placed 7th and the team who placed 8th would have been switched around.

Competition Day

• Have judges watch videos before the competition. We typically show 2-3 videos of strong, competitive performances to the judges less than an hour before showtime, and have the judges fill out a rubric to those videos, so there is no inherent bias towards the teams that perform earlier in the show.
• Organizers Inputting Scores - make sure to have several people working on this. It's easy to make mistakes when inputting numbers into excel!

Deliberation

• Agree with Arnab! Seems like many competitions are taking on the top 5 teams potentially placing approach.
• For the past 4 years at Bruin, only the top 5 teams in terms of final score are considered for placings. However, 1st and 2nd ranking teams (in terms of final scores) are guaranteed placings, and 1st ranking team (in terms of points) is guaranteed either First or Second Place. Here's the breakdown -

Breakdown:

1st Place in Points – Can move down 1 spot (2nd place) or remain at 1st​

2nd Place in Points – Can move down 1 spot (3rd place) or up 1 spot (1st place). ​

3rd Place in Points – Can move up 1 spot (2nd) or move out of Placings (4th/5th)​

4th/5th Place in Points – Can move up to 3rd place only, or remain at 4th/5th​

Special Case:​

If 2nd in points team awarded 1st place in deliberations, 1st place team in points automatically moves to 2nd. ​

3rd place in points, now, can’t move up to 2nd.​

3rd Place will still be deliberated between 3rd, 4th, 5th placing teams in terms of points​

PowerPoint Breakdown for Bruin XVIII Judging, for those interested:
https://www.dropbox.com/s/u3kmi0im6wqbcr1/BBXVIII%20Judging%20Breakdown.pptx?dl=0
(I believe it was sent to teams in March or April before Bruin)


Also feel free to add, modify, critique any of those judging methods and ideas!

 

[ManrajBajwa]

Great Points, Arnab! I agree with so many of your points on effective judging strategies. It would be great to see many more competitions begin to implement these things in the near future!
I would just like to add to and reiterate several of the points I have noticed to be effective at many competitions and especially at Bruin, where I've had the chance to run judging for the past four years.


Selection of Judges

• Choose Judges that understand what your competition is about and what your committee wants/expects from its teams in terms of performance.
• We at Bruin are huge fans of large judging panels (6-7 judges). Larger sample size, assuming that the judges judge to your competitions rubric and beliefs, should ideally provide more precision and accuracy in competition results.
• Look to diversify your judges lineup (i.e. male & female representation, younger & older judges, etc.). Teams can get some great, well-rounded feedback from a more diverse panel, I've noticed.

Pre-Comp Judges Meeting

• Typically hold a Google Hangout Judges meeting at least 1.5-2 months before the competition date.
• Pro tip- If your university/school/college email is associated with google, use that email to create a Google Hangout with 15 users!
• Give teams an ample amount of time to prepare a routine to cater to your competition, rubric, and judges (whether they all be traditional, modern, mixed, etc.)
• I know some competitions have a free Q&A sessions while others have teams submit Q's before hand and have judges answer them at a later date, but personally I'm not a big fan of the latter. What has seemed to be effective at the Bruin pre-judges meeting and also at Blowout's Pre-Judges meeting from this year is Structuring the meeting by having judges go through each judging category (i.e. Creativity, Synchronization, etc.) by describing their expectations for a team that specific category, and what they expect from a first place winning team at that competition in the context of that category. This will provide teams a clear understanding of what each judge expects.

Judges Group-Me/Post Judges Meeting Q's

• Keep a Group-Me or What's App chat going with your judges. Teams will have hella questions even after the judges meeting, so keep a way to relay messages between teams and judges during this period.
• Make a google form or some similar system where teams can submit their personal, private questions regarding judging that they may not wish to share with the other teams during the Judges Meeting.

Rubric Creation

• I am not a huge fan of checklist type rubrics where every category is divided into many subcategories specifically detailing point allocations and deductions. Though it is awesome and helpful to teams, I feel that's what the Pre Competition Google Hangout judges meeting is for. Keeping a rubric that is too specific, it seems at times, can cause teams to become too complacent with their and treat the rubric like a checklist which in a sense restricts creativity both at the competition and in the circuit in general. By saying this, I definitely am not implying that there shouldn't be standardization in judging and point allocation. I'm just pushing for more holistic, open-ended rubrics.

• Deductions - Some competitions keep a deductions category in their rubric for things such as vardi malfunctions or prop malfunctions (i.e. saap breaking). I personally am not a huge fan of this. Instead of keeping a category for vardi/prop deductions, what we tried at Bruin last year was to emphasize the "Presentation" category which encompassed both of these things. Teams would be rewarded, as opposed to docked, if everything in terms of their presentation, which includes vardis, props, music,etc. was ok. Having a category where teams can earn points, as opposed to lose points, for these things may push teams to be more creative and risky in their routine and ideas, which may involve vardis and/or props.

Rubric Scoring!!

• Bruin has for the past several years used an aspect of Faizan's WBBC approach with normalizing scores. Normalizing scores provides teams with a much easier way to follow their scores in relation to the other teams from that competition. I can gladly send/upload the template of the Bruin Judging Excel sheets if anybody would like to mess around with these see how they were created!

• Judges: Please, if possible, judge with large spreads. After closely following judging sheets at Bruin for the past several years, it seems that the judges who have scored with a standard deviation of around 8-10 points seem to be the most precise in getting the order placings relatively correct.
• Dropping Scores - Teams will always want to ensure that subjectivity and biases aren't playing a role in the judging process. Of course all competitions, including Bruin, will try to select judges who are honest and who will judge the competition without taking into consideration any personal affiliations and relationships with the competing teams. Still, however, it's pretty damn important to keep some protocols that protect the teams. For this, at Bruin, we drop the Lowest Normalized Score from each team (assuming there are 6 judges). Dropping normalized scores (as opposed to raw scores) ensures that a judge who, for example, may on average score lower or higher than the other judges on the panel won't have his or her scores discarded solely for the reason of them scoring at a different mean or average. It also gives teams a peace of mind that there won't be a judge that is out to get them or that may not be a fan of their particular style of bhangra. In a way it protects both judges and teams, and keeps the competition fair!

• Side note: At Bruin, after analyzing the scores over the last two years, dropping the lowest scores has 1) Not Changed placings either year 2) Only changed one overall ranking: in 2015, the team who placed 7th and the team who placed 8th would have been switched around.

Competition Day

• Have judges watch videos before the competition. We typically show 2-3 videos of strong, competitive performances to the judges less than an hour before showtime, and have the judges fill out a rubric to those videos, so there is no inherent bias towards the teams that perform earlier in the show.
• Organizers Inputting Scores - make sure to have several people working on this. It's easy to make mistakes when inputting numbers into excel!

Deliberation

• Agree with Arnab! Seems like many competitions are taking on the top 5 teams potentially placing approach.
• For the past 4 years at Bruin, only the top 5 teams in terms of final score are considered for placings. However, 1st and 2nd ranking teams (in terms of final scores) are guaranteed placings, and 1st ranking team (in terms of points) is guaranteed either First or Second Place. Here's the breakdown -

Breakdown:
1st Place in Points – Can move down 1 spot (2nd place) or remain at 1st
2nd Place in Points – Can move down 1 spot (3rd place) or up 1 spot (1st place).
3rd Place in Points – Can move up 1 spot (2nd) or move out of Placings (4th/5th)
4th/5th Place in Points – Can move up to 3rd place only, or remain at 4th/5th


Special Case:
If 2nd in points team awarded 1st place in deliberations, 1st place team in points automatically moves to 2nd.
3rd place in points, now, can’t move up to 2nd.
3rd Place will still be deliberated between 3rd, 4th, 5th placing teams in terms of points



PowerPoint Breakdown for Bruin XVIII Judging, for those interested:
https://www.dropbox.com/s/u3kmi0im6wqbcr1/BBXVIII%20Judging%20Breakdown.pptx?dl=0
(I believe it was sent to teams in March or April before Bruin)


Also feel free to add, modify, critique any of those judging methods and ideas!

Thank you for sharing

 

[arnab]

Great Points, Arnab! I agree with so many of your points on effective judging strategies. It would be great to see many more competitions begin to implement these things in the near future!
I would just like to add to and reiterate several of the points I have noticed to be effective at many competitions and especially at Bruin, where I've had the chance to run judging for the past four years.


Selection of Judges

• Choose Judges that understand what your competition is about and what your committee wants/expects from its teams in terms of performance.
• We at Bruin are huge fans of large judging panels (6-7 judges). Larger sample size, assuming that the judges judge to your competitions rubric and beliefs, should ideally provide more precision and accuracy in competition results.
• Look to diversify your judges lineup (i.e. male & female representation, younger & older judges, etc.). Teams can get some great, well-rounded feedback from a more diverse panel, I've noticed.

Pre-Comp Judges Meeting

• Typically hold a Google Hangout Judges meeting at least 1.5-2 months before the competition date.
• Pro tip- If your university/school/college email is associated with google, use that email to create a Google Hangout with 15 users!
• Give teams an ample amount of time to prepare a routine to cater to your competition, rubric, and judges (whether they all be traditional, modern, mixed, etc.)
• I know some competitions have a free Q&A sessions while others have teams submit Q's before hand and have judges answer them at a later date, but personally I'm not a big fan of the latter. What has seemed to be effective at the Bruin pre-judges meeting and also at Blowout's Pre-Judges meeting from this year is Structuring the meeting by having judges go through each judging category (i.e. Creativity, Synchronization, etc.) by describing their expectations for a team that specific category, and what they expect from a first place winning team at that competition in the context of that category. This will provide teams a clear understanding of what each judge expects.

Judges Group-Me/Post Judges Meeting Q's

• Keep a Group-Me or What's App chat going with your judges. Teams will have hella questions even after the judges meeting, so keep a way to relay messages between teams and judges during this period.
• Make a google form or some similar system where teams can submit their personal, private questions regarding judging that they may not wish to share with the other teams during the Judges Meeting.

Rubric Creation

• I am not a huge fan of checklist type rubrics where every category is divided into many subcategories specifically detailing point allocations and deductions. Though it is awesome and helpful to teams, I feel that's what the Pre Competition Google Hangout judges meeting is for. Keeping a rubric that is too specific, it seems at times, can cause teams to become too complacent with their and treat the rubric like a checklist which in a sense restricts creativity both at the competition and in the circuit in general. By saying this, I definitely am not implying that there shouldn't be standardization in judging and point allocation. I'm just pushing for more holistic, open-ended rubrics.

• Deductions - Some competitions keep a deductions category in their rubric for things such as vardi malfunctions or prop malfunctions (i.e. saap breaking). I personally am not a huge fan of this. Instead of keeping a category for vardi/prop deductions, what we tried at Bruin last year was to emphasize the "Presentation" category which encompassed both of these things. Teams would be rewarded, as opposed to docked, if everything in terms of their presentation, which includes vardis, props, music,etc. was ok. Having a category where teams can earn points, as opposed to lose points, for these things may push teams to be more creative and risky in their routine and ideas, which may involve vardis and/or props.

Rubric Scoring!!

• Bruin has for the past several years used an aspect of Faizan's WBBC approach with normalizing scores. Normalizing scores provides teams with a much easier way to follow their scores in relation to the other teams from that competition. I can gladly send/upload the template of the Bruin Judging Excel sheets if anybody would like to mess around with these see how they were created!

• Judges: Please, if possible, judge with large spreads. After closely following judging sheets at Bruin for the past several years, it seems that the judges who have scored with a standard deviation of around 8-10 points seem to be the most precise in getting the order placings relatively correct.
• Dropping Scores - Teams will always want to ensure that subjectivity and biases aren't playing a role in the judging process. Of course all competitions, including Bruin, will try to select judges who are honest and who will judge the competition without taking into consideration any personal affiliations and relationships with the competing teams. Still, however, it's pretty damn important to keep some protocols that protect the teams. For this, at Bruin, we drop the Lowest Normalized Score from each team (assuming there are 6 judges). Dropping normalized scores (as opposed to raw scores) ensures that a judge who, for example, may on average score lower or higher than the other judges on the panel won't have his or her scores discarded solely for the reason of them scoring at a different mean or average. It also gives teams a peace of mind that there won't be a judge that is out to get them or that may not be a fan of their particular style of bhangra. In a way it protects both judges and teams, and keeps the competition fair!

• Side note: At Bruin, after analyzing the scores over the last two years, dropping the lowest scores has 1) Not Changed placings either year 2) Only changed one overall ranking: in 2015, the team who placed 7th and the team who placed 8th would have been switched around.

Competition Day

• Have judges watch videos before the competition. We typically show 2-3 videos of strong, competitive performances to the judges less than an hour before showtime, and have the judges fill out a rubric to those videos, so there is no inherent bias towards the teams that perform earlier in the show.
• Organizers Inputting Scores - make sure to have several people working on this. It's easy to make mistakes when inputting numbers into excel!

Deliberation

• Agree with Arnab! Seems like many competitions are taking on the top 5 teams potentially placing approach.
• For the past 4 years at Bruin, only the top 5 teams in terms of final score are considered for placings. However, 1st and 2nd ranking teams (in terms of final scores) are guaranteed placings, and 1st ranking team (in terms of points) is guaranteed either First or Second Place. Here's the breakdown -

Breakdown:
1st Place in Points – Can move down 1 spot (2nd place) or remain at 1st
2nd Place in Points – Can move down 1 spot (3rd place) or up 1 spot (1st place).
3rd Place in Points – Can move up 1 spot (2nd) or move out of Placings (4th/5th)
4th/5th Place in Points – Can move up to 3rd place only, or remain at 4th/5th


Special Case:
If 2nd in points team awarded 1st place in deliberations, 1st place team in points automatically moves to 2nd.
3rd place in points, now, can’t move up to 2nd.
3rd Place will still be deliberated between 3rd, 4th, 5th placing teams in terms of points



PowerPoint Breakdown for Bruin XVIII Judging, for those interested:
https://www.dropbox.com/s/u3kmi0im6wqbcr1/BBXVIII%20Judging%20Breakdown.pptx?dl=0
(I believe it was sent to teams in March or April before Bruin)


Also feel free to add, modify, critique any of those judging methods and ideas!

Awesome stuff man, especially on the rubric and judges meetings. I'd love to see the excel sheet you've been using at Bruin, and I'm sure others would as well! Do you mind uploading?

 

[J Wong]

How do you feel about just ranking teams? Like a rubric could exist with certain things the competition wants to see, but in terms of the scores it could simply be: given the above criteria, please rank the following teams. It gives you the ability to modify how you felt the team should place and doesn't restrain your ability to judge, it also has the added benefit of not being complicated

 

[vaibhav]

Awesome stuff man, especially on the rubric and judges meetings. I'd love to see the excel sheet you've been using at Bruin, and I'm sure others would as well! Do you mind uploading?

Of course! Here is the link to the Bruin Bhangra XVII Judging Template:
https://www.dropbox.com/s/kyehryv4ma3snva/Bruin Bhangra Judging Template.xlsx?dl=0

 

[arnab]

How do you feel about just ranking teams? Like a rubric could exist with certain things the competition wants to see, but in terms of the scores it could simply be: given the above criteria, please rank the following teams. It gives you the ability to modify how you felt the team should place and doesn't restrain your ability to judge, it also has the added benefit of not being complicated

I've thought about that before. An intriguing idea for the sake of simplicity, but gets a lot of pushback because 1) doesn't capture the magnitude of difference between teams (i.e. all the scores 1-8 are distributed exactly evenly) and 2) not transparent at all so teams have a harder time understanding the logic behind placings. This is kind of what happens at comps where the rubrics are just "guidelines" and placings are decided entirely via deliberation, and I find that it introduces a lot of subjectivity into the process.

 

[hsdeol]

random qu- is the (perceived or real) issue with judging placings related or feedback related?

i havent been to a comp for a while, but when ive watched vids from shows its usually pretty obvious which teams were in contention to place/"win"


edit- props to the guys and gals that have been judging the past few years. i think your judgement has been pretty solid

 

[arnab]

random qu- is the (perceived or real) issue with judging placings related or feedback related?

i havent been to a comp for a while, but when ive watched vids from shows its usually pretty obvious which teams were in contention to place/"win"


edit- props to the guys and gals that have been judging the past few years. i think your judgement has been pretty solid

My impression is that it's mostly feedback related. I keep hearing that teams want more transparency in the process, a need that I think most judges are sensitive to.

That being said, I can think of at least a few competitions where the judging decisions have been debated/argued by teams. I've been a part of some of those panels, and it's never easy.

There's also the opinion that judges have the responsibility to drive more innovation/creativity/diversity in the circuit by rewarding those performances with placings. The idea is that if 5-6 teams place everywhere, then everyone will just try to look like those teams. Personally, I do believe there's too much homogeneity in the circuit, but I strongly disagree with the idea that it's a judge's role to fight homogeneity. Our role is to interpret performances through the eyes of a rubric as objectively as possible. If the circuit wants more diverse performances, then I suggest committees aggressively increase the role of creativity in rubrics going forward. Therefore, technique/execution are weighed relatively less, but if you want creativity that's what you gotta do.