Publishing Judges' Scoresheets

[scaplash]

I'm not sure if this has been said in the past, but I think as a community we could all benefit from publishing the judging scores/comments for the following reasons:


1. Transparency. We might not always agree with the placings of varying competitions because judging has an inherent subjective component. By publishing scores we can see what the judges thought was clean vs. what we thought was clean. Increased transparency leads to increased accountability.


2. Circuit Development. One thing that I feel like happens a lot among teams is that when we see a placing set, or a team that places consistently we try to emulate them-which is a natural thing to do. But I think the problem with that is that often times we end up emulating the wrong part of that set. For example, lets say example team A has just placed at 3 comps in a row. Example team B sees the video and sees-wow, look at those gimmicks-that's probably why they placed so much-Example team B focuses on gimmicks, and maybe not as much on execution, thinking that is their golden ticket, and if repeated in other teams, we have a circuit saturated in gimmicky teams.
I understand a lot of experienced captains probably won't agree with this point- because spending time with judges at judges meetings or competing consistently teaches you that no one component is needed to win a comp, its a balance of form, execution and arguably entertainment. But for up and coming teams with less competition experience, this is a lesson that is often hard learned. Seeing rubrics shows people why a team did well in that judges' eyes

3. Constructive Discussions. By seeing scores, hopefully we can have real discussions on a teams performance, instead taking shots at the judges or the competition. The whole point of this forum is to connect teams and dancers for the betterment of the circuit.


Kind of a long post-my bad, but I think that if you are running a comp or competing in a comp, we need to make it a standard that scores are published. Judges can remain anonymous (not that is matters either way-people will find out). But if a competition stands by its placings, then it should stand by the scoresheets/comments.

What do you guys think?

-Sonny

 

[hsdeol]

i'm for this, not for the same reasons, but seems fair. also judges should not be anonymous.

honestly tho, more teams should post their videos or send them out to btf users or on fb to get honest constructive opinions. for whatever reason, a lot of teams refrain from posting their vid after a comp. there are a lot of competitions, melas, on-campus shows, local gigs that host newer teams and videos are never released.

i encourage teams to release vids for feedback. if you dont feel comfortable posting here, then send it out in messages. (but really, if you dont feel comfortable posting here, you prob wont be happy about your judging scores getting released either).




**edit: i recognize that judging is pretty difficult, and i wouldnt expect a judge to be able to breakdown an entire set with just one viewing. releasing scores (without anonymity) would to be to allow teams to gauge the overall vibe a judging panel prefers.

 

[arnab]

So, I've judged a few competitions now, and I will agree that the comments on the score sheet definitely do give the teams some idea of what our impression of the set was in the moment. The one caveat though (which is why I think some committees don't publicly release the sheets), is the role of deliberation in the placing process. If deliberation does play a role, then many feel like releasing the sheets opens up a can of worms - so maybe a summary of the deliberation process might also be something you'd want to include.


That being said, I think at the end of the day, teams just really want to improve. Releasing the sheets might be a good step, but I don't think the teams get THAT much out of it.... I know personally my notes are all shorthand and just points for me to remember to mention in the captains' meeting later. I know that in most of the panels I've been part of, the judges have all offered to follow up with teams to discuss their performances beyond what can be said in a 5-10 min captain's meeting. My advice to captains is that if judges offer that kind of meeting, take them up on it. In my opinion, that's the kind of experience where you're really going to get detailed, granular feedback...more so than my chicken scratch on a rubric.

 

[Basim]

I am sure this topic has been discussed before, but it's always good to hear fresh perspectives in modern times.

I'm all for competitions releasing judging scoresheets as I believe it adds a lot of transparency and judges are essentially held accountable.

Some of most prominent and well run competitions, that really pushed the envelope of executing flawlessly and going out of their way to cater toward teams, compiled & released scoresheets/provided analysis after the competition took place. I know not everyone was competing or part of the "bhangra scene" at the time, so below are a few links

Examples:
Elite 8: http://bhangrateamsforum.com/discuss/main-bhangra-discussion/elite-8-bhangra-invitational-2015-judges'-scores-other-notes/

World's Best Bhangra Crew (WBBC): http://bhangrateamsforum.com/discuss/main-bhangra-discussion/$$$$$$$$$$$$-comprehensive-judging-system-$$$$$$$$$$$$$$$$/

Another nicely tabulated scores presented publicly was with the competition My Life is Bhangra (MLIB): http://bhangrateamsforum.com/discuss/main-bhangra-discussion/my-life-is-bhangra-judging-workbook/

Really worth a look through if you haven't seen these topics before.

~ Basim

 

[mafzal]

+1 As a judge I'm all for this, assuming the teams are cool with publishing their scores. I think it is important to release the notes and feedback/comments along with the scores. The scores don't say much without the explanation as to why points were awarded or docked

 

[Saab]

Have been thinking on this more in the past few weeks - would also like to see some of the deliberation of the judges for placings released. I realize that this may be TOO much, but something along the lines of a short summary for the judges about the competition and feedback would be appreciated. It would be something more substantial than the raw score and few comments on the sheet, and would give a team a contextual placing based on where they rank in the show.

 

[amancheema]

would also like to see some of the deliberation of the judges for placings released

allowing deliberation is absolute lunacy.


if you win the most points, you should win the competition. end of story.


if judges want to "deliberate" after points have been calculated, then the competition should either get rid of the numerical rubric or it should get rid of the judges.

 

[mrchicity]

would also like to see some of the deliberation of the judges for placings released

allowing deliberation is absolute lunacy.


if you win the most points, you should win the competition. end of story.


if judges want to "deliberate" after points have been calculated, then the competition should either get rid of the numerical rubric or it should get rid of the judges.

You're applying objective scores to subjective criteria. It's not always perfect. I'm all for doing away with rubrics though, the necessity of 5 bullshit points to something like an Elite 8 gimmick was always lost on me.

 

[Saab]

would also like to see some of the deliberation of the judges for placings released

allowing deliberation is absolute lunacy.


if you win the most points, you should win the competition. end of story.


if judges want to "deliberate" after points have been calculated, then the competition should either get rid of the numerical rubric or it should get rid of the judges.

You're applying objective scores to subjective criteria. It's not always perfect. I'm all for doing away with rubrics though, the necessity of 5 bullshit points to something like an Elite 8 gimmick was always lost on me.

I mean, I hate the way modern rubrics are constructed, but until someone utilizes a rank based system that's more feedback oriented while putting people in the right spots, I think deliberation still needs to happen. Rubrics just don't capture nuance well.

 

[amancheema]

what's the point of having an objective standard if subjective opinions matter?


north american competitions are market-driven. teams apply to competitions that they feel comfortable going to (folk, modern, gimmick, whatever), but in return, teams expect judges to assign them objective scores.


you all dance/coach... so imagine your team gets the highest marks one night and then is told at the judges meeting that a subjective deliberation bumped you down to 3rd place. now what?

 

[Mr. Mankiran]

what's the point of having an objective standard if subjective opinions matter?


north american competitions are market-driven. teams apply to competitions that they feel comfortable going to (folk, modern, gimmick, whatever), but in return, teams expect judges to assign them objective scores.


you all dance/coach... so imagine your team gets the highest marks one night and then is told at the judges meeting that a subjective deliberation bumped you down to 3rd place. now what?

I agree with you to an extent. The problem is that numbers are so arbitrary. You might score a team a 90, then a team performs later in the show that absolutely blows them out of the water and it may influence you to think that 90 team was ACTUALLY an 80. We don't have formal evaluators who have been trained to judge this style of dance, most of them were former dancers with some success who have now decided to start judging.


I complain, like many of us do, about judging but I think a deliberation allows them to come together and a least make the right choice. There hasn't been any outstanding, questionable placing at a big comp this season, imo.

 

[Basim]

I agree with you to an extent. The problem is that numbers are so arbitrary. You might score a team a 90, then a team performs later in the show that absolutely blows them out of the water and it may influence you to think that 90 team was ACTUALLY an 80.

^ This is a great point. The reason that I would prefer judges to "talk" at the end is to reaffirm scoring after having watched ALL performances. It's so hard to blindly give points in subjective criteria when you haven't even seen all teams.

If judges haven't seen all performances, they are indirectly forced into factoring "show order" into their distribution of points.

I personally like it when you use points to narrow down to say the top 3 teams, then use deliberation to rank the last few teams after a 10-15 minute discussion amongst the judges. Usually competitions have some sort of entertainment for the audience while the judges deliberate.

~ Basim

 

[scaplash]

This might be a subtle point, but in judging, one of my problems is the time between performances. What makes judging even harder, on top of all of the different things you are looking for is the time to write down thoughts between performances. While you can certainly write things down during the performance, I think the judging system can benefit with more time between performances (even 2-3 minutes) where judges can step back, think about the performance as a whole, and write down even 1 sentence that can be unique and specific to the performance.

Also, I am not sure how a lot of other people judge, but something I do personally, that I think could be adopted in rubrics is to have the rubric split up into "Segments". So for example it would be:
Segment 1:
Synchronization:___/10
Execution:__/10
Formations:___/10
Segment 2:
Synchronization:___/10
Execution:__/10
Formations:___/10
I think doing it like this makes it easier to judge, because as you are watching the set, you can grade each individual segment and then average out the scores for the final scores on synchronization, etc. It's also easier for teams to learn from judges because they can get an aspect of what segments they executed well and which ones they didn't.
I know the idea of judging deliberation can be tricky, but I think its important so that judges can qualitatively determine which team did the best. I think the judging deliberation falls through when comp organizers seem to influence it based on "comp style". Like if team A executes the best, and Team B doesn't execute as well, but is more modern and the comp wants to be seen as a modern comp and then places that team over Team A to stick to its "identity". I don't think deliberation should be used for that.
-Sonny

 

[scaplash]

what's the point of having an objective standard if subjective opinions matter?


north american competitions are market-driven. teams apply to competitions that they feel comfortable going to (folk, modern, gimmick, whatever), but in return, teams expect judges to assign them objective scores.


you all dance/coach... so imagine your team gets the highest marks one night and then is told at the judges meeting that a subjective deliberation bumped you down to 3rd place. now what?

I complain, like many of us do, about judging but I think a deliberation allows them to come together and a least make the right choice. There hasn't been any outstanding, questionable placing at a big comp this season, imo.

^While I am leaning towards agreeing, its hard to say when not a lot of the videos have been put up yet.

 

[mafzal]

Responding to Sonny's recent post

1. More time between performances for judges to gather thoughts and write comments - YES, please! I am always scrambling to get my thoughts onto paper and compile some concluding comments. Typically I'm writing during the team's performance, after they leave stage, and all the way through the next team's intro video.


2. Evaluating segment by segment - I like this idea and will experiment with it at home


3. Deliberation - I agree.

 

[Cherag]

1. Transparency. We might not always agree with the placings of varying competitions because judging has an inherent subjective component. By publishing scores we can see what the judges thought was clean vs. what we thought was clean. Increased transparency leads to increased accountability.

If a newer team is okay with people knowing points wise they placed last, it can be demoralizing. I understand when it comes to the discussion of the top 3 or 4 and how the discussion was weighed out in terms of deliberation but then again I think it matters more so for those teams to understand, and quite frankly it doesn't do anything for the circuit. As long as you tell those who are involved, I think that things are okay. Teams don't not go to competitions because of the way individuals judge. I encourage teams to talk to an organizer and ask them exactly what they want to see in a performance, even before talking to judges, this way you know what you're getting into, and judges then are responsible for carrying out the vision of the organizers as an extension of the competition.

2. Circuit Development. One thing that I feel like happens a lot among teams is that when we see a placing set, or a team that places consistently we try to emulate them-which is a natural thing to do. But I think the problem with that is that often times we end up emulating the wrong part of that set. For example, lets say example team A has just placed at 3 comps in a row. Example team B sees the video and sees-wow, look at those gimmicks-that's probably why they placed so much-Example team B focuses on gimmicks, and maybe not as much on execution, thinking that is their golden ticket, and if repeated in other teams, we have a circuit saturated in gimmicky teams.
I understand a lot of experienced captains probably won't agree with this point- because spending time with judges at judges meetings or competing consistently teaches you that no one component is needed to win a comp, its a balance of form, execution and arguably entertainment. But for up and coming teams with less competition experience, this is a lesson that is often hard learned. Seeing rubrics shows people why a team did well in that judges' eyes

Emulate greatness. Kobe emulated Jordan, he practiced the same killer instinct that MJ exhibited and it made him on of the greatest of all time, but will he ever be better at a a legacy standpoint than Michael? No. MJ did it first, and a lot of older school kats can respect how important your identity and originality is and their teams were successful because they didn't go out and jack what other teams did, and try to make it your own. Steal things that will help with your progression in growth as a dancer, and distribute knowledge of how to make things easy when it comes to the fundamentals while executing at a high level. You jack something someone else does, I personally wouldn't respect you identity as much, and I know other friends and members of the circuit who date back to when I started dancing in 2004 would say the same thing. Be yourself, don't follow suit because a lot of teams are following suit and not stepping out of the box because it's easier not to. Just doing what others are doing and it being attractive among the younger generation of dancers, is a huge reason why the circuit has diminished in the rate of development as it once had.

"Success humbles the great man, astonishes the common man, and puffs up the little man. " - Pick the man you want to be, and be yourself not someone else.

3. Constructive Discussions. By seeing scores, hopefully we can have real discussions on a teams performance, instead taking shots at the judges or the competition. The whole point of this forum is to connect teams and dancers for the betterment of the circuit.

You don't need to see scores to have good discussions about team performances, a blank rubric will do, just to know what you're looking at. And those who care to get better or learn about teams will reach out. I reach out to teams all of the time and have others hit me up too, but I want people to work for it. Hold the hand but also let the child fall, guidance should be provided by those who can, but don't give answers to people. I've said it in the past, with the influx of mixing knowledge that is already out there, video to learn aspects of dance from good dancers and teams, teams shouldn't need much more to get good.

Individuals in the circuit are connected already, but to a fault at times, the competitive nature is gone because people run forward to congratulate those who won, rather than look back at your team who failed. I know it may sound harsh but when in aspects of competition, people need to drop the buddy buddy-ness at time and understand the goal at hand. This doesn't apply to those who dance strictly for fun and do it because they love it, because I am one of those people, but I also know the time and effort it takes to get somewhere which is why I want those I dance with to feel accomplished, and that doesn't mean you need to place to do so.

In terms of deliberation, I think it can be helpful in standardizing thought and understanding on how heavily the rubric was reflect through a judges eyes holding to what the organization is look for and what the teams have been told beforehand. Although I believe deliberation isn't needed if all judges see the competition strictly through the rubric and have the same defined understanding of what is expected. I don't think this should happen across the board for competitions, live and traditionally focused competitions yes, because it clarifies guidelines for teams.

The job of a judge is hard and not easy, but you must yourself be good at the craft, and want to influence and help others to get better. I would love to judge anonymously, because I just care about helping teams and keeping it fair for them and don't care if they know it's me, but those who don't get they want care to know.

My two cents,
Cherag

 

[Nimit]

Solid thread. I think aman brings up a decent point about judging deliberation. I think the concept makes sense if you had two judges. Then show order or whatever can skew someone's score and it is important to have a look back. Yet when comps have 5 or so judges, the law of averages should be good enough to replace the deliberation process. There are enough people on the panel that each opinion does not carry too much weight and that bias in any way will be limited. I think when you have enough people judging a competition, you might as well just do placings based on raw scores. Deliberation in itself kind of ruins the point of having separate judges and should probably only be there in case of a tie.


Also going back to the initial thread, people should post judging rubrics. I went back and checked the WBBC and Elite 8 judging rubrics and I thought they were not only cool to look at, but also pretty informative as to how judges interpreted the set. Kind of gives you a benchmark as to what level of synch/execution/whatever earns you points.


As for the problem about last place teams getting roasted on BTF, I get why that would be an issue. We've all performed somewhere or another and been like, "I hope no one ever sees that video." But a simple check box asking if it is ok for the competition to release your scoresheet would be an easy solution.