Elite 8 Bhangra Invitational 2012: Judging Package

[gurteg]

i think going first in the show really effects scoring, had we had gone last and shuffled around the other teams the outcome may have been different. As a community we should try to find a way to eliminate this from happening, i do not have a solution i just think that we should try to come up with one. Every comp has a team that has to go first, its NEVER fun being that team lol. goodnight. please enjoy our performance!

The way I judge (non-bhangra, non-dancing) competitions is to give the first group 50/100 and then then judge according to that standard. Its tricky, and many people do not like getting 22/100 but I figure a normalization is required somewhere..

This is a genius idea and should be implemented. However take into account that if the first team is the "best" team then the highest score potentially would be just a 50/100. Like you said teams don't like receiving these scores and would start complaining about such low scores when their sets in fact hit the hardest. If every judge/competition were to start practicing this, then that wouldn't be such a big deal. It standardizes the rest of the competition, especially when judges aren't allowed to go back and alter scores for previous teams.

 

[Arjun Barua]

i think going first in the show really effects scoring, had we had gone last and shuffled around the other teams the outcome may have been different. As a community we should try to find a way to eliminate this from happening, i do not have a solution i just think that we should try to come up with one. Every comp has a team that has to go first, its NEVER fun being that team lol. goodnight. please enjoy our performance!

The way I judge (non-bhangra, non-dancing) competitions is to give the first group 50/100 and then then judge according to that standard. Its tricky, and many people do not like getting 22/100 but I figure a normalization is required somewhere..

This is a genius idea and should be implemented. However take into account that if the first team is the "best" team then the highest score potentially would be just a 50/100. Like you said teams don't like receiving these scores and would start complaining about such low scores when their sets in fact hit the hardest. If every judge/competition were to start practicing this, then that wouldn't be such a big deal. It standardizes the rest of the competition, especially when judges aren't allowed to go back and alter scores for previous teams.

Yeah, I'm in favor of this idea. I remember going through the very same thought process when I was judging Bhangre Di Mehfil.

 

[Saleem]

I agree with normalization but this is the wrong way to do it.

Example:
"Creativity". First team is the MOST creative. Ending range of scores for all teams may be 10 - 50 (40 points)
"Traditionality" First team is average traditionality. Ending range of scores for all teams may be 20 - 100 (80 points).

You've just made traditionality more important than creativity based on the character of the first team. That spread of points is very important.

 

[jfried]

I think he was talking about this strictly in terms of overall point totals, but you make a good point, Saleem. Might be worth starting a bit higher too, to account for situations like what Saleem described. 75?

 

[arnab]

Welllllll then what happens when the first team is the worst?

 

[rajkaran]

I agree with normalization but this is the wrong way to do it.

Example:
"Creativity". First team is the MOST creative. Ending range of scores for all teams may be 10 - 50 (40 points)
"Traditionality" First team is average traditionality. Ending range of scores for all teams may be 20 - 100 (80 points).

You've just made traditionality more important than creativity based on the character of the first team. That spread of points is very important.

good catch!


What if we had it so that each category was worth the same amount of points (10 pts traditionality, 10 pts synch, 10 points, grace, etc), then normalized it (base everything on the first team getting 50/100), THEN at the end, during deliberation, each category can be weighed according to a different percentage. Would that work?

 

[yraparla]

We had this exact discussion last year....although it seemed to be primarily Sue, Saleem, and I at the time. We got labeled 'nerds' but it's good to see others realize that it's a problem and needs fixin'


The issue with doing things properly statistically is that it may involve some obfuscation of scores, and you can't trust committees to implement it properly (Adding up number is one thing, normalizing and applying statistical methods increases chances of mistakes).


The three issues are:
Mean (average score)
Variance (spread of the scores)
Range (distance between highest and lowest score)


You can't simply scale/normalize because if one judges scores a bunch of teams together and then one team farther ahead, then scaling will give that first team undue influence based on one judge (and vice versa). At the base level, you have put the mean on 50, then adjust the scores to have the std deviation be the same. Outliers will still have increased influence (as they should). It's not perfect, and I wrote up something last year that did better but I can't remember what it was. I'll try to dig it up the excel sheet for it


We also need to stop dropping highest lowest scores. Although it's not listed in the E8 release, I actually did ask the committee to give us the undropped scores so we could consider that datapoint as well (UNC and NYPD don't change and I think JJ and AEG swap? That's what I remember). Usually one judge has a higher average score and one judge has the lowest average score (pashi and benny in this case) and they get discounted unfairly.

 

[Cherag]

The worst thing about all of this is that number one spot. It is really a shame to see teams getting their name picked out of a hat, or whatever the process is at any competition, and see people going nuts over the "sweet spot" in the show, depending on factors like when intermission is, who's before us, who's after us, etc. Teams shouldn't place if they aren't the best team there, this is in no reference to Elite but in general, your performance is always the one you prepared, and likewise other teams worry about themselves in the same way. I have seen teams perform worse because of their spot in the show, the predisposed mentality that going first in the show means no placing, can really affect a teams performance psychologically. This goes for every competition. For example, had JJ won, it would of been like "Hey they went first in the show, maybe we can pull it off too", when teams look back at it who stand in that number one spot for their competition whether it's first second or third place. Regardless I feel like this would have to happen a few times for X team to think like this. I felt like JJ, not placing-wise but performance-wise could have done better had they gone later in the show because they wouldn't be put in that number one spot position and mentality, along with other teams like AEG who were third. It is shameful to not see that each team is judged on it's own basis.

 

[dheerja]

The worst thing about all of this is that number one spot. It is really a shame to see teams getting their name picked out of a hat, or whatever the process is at any competition, and see people going nuts over the "sweet spot" in the show, depending on factors like when intermission is, who's before us, who's after us, etc. Teams shouldn't place if they aren't the best team there, this is in no reference to Elite but in general, your performance is always the one you prepared, and likewise other teams worry about themselves in the same way. I have seen teams perform worse because of their spot in the show, the predisposed mentality that going first in the show means no placing, can really affect a teams performance psychologically. This goes for every competition. For example, had JJ won, it would of been like "Hey they went first in the show, maybe we can pull it off too", when teams look back at it who stand in that number one spot for their competition whether it's first second or third place. Regardless I feel like this would have to happen a few times for X team to think like this. I felt like JJ, not placing-wise but performance-wise could have done better had they gone later in the show because they wouldn't be put in that number one spot position and mentality, along with other teams like AEG who were third. It is shameful to not see that each team is judged on it's own basis.

Totally agree. I remember we went first in the show at Bruin 2 years ago, and after months of preparing for the ultimate goal of killing it at Bruin, pulling that "1" paper out of the hat killed us. We felt so deflated, especially since it's so hard to keep your ranking at competitions that have such strong lineups like Bruin and Elite. I've seen teams that go first do well at smaller competitions (CMU placed at VPD recently) - but at these big name shows it really hurts you. The crowd isn't even seated, and it's so hard to shake the mentality that you're at a severe disadvantage. The best thing I've seen shows do is put 2 exhibition acts in the beginning to get the show started as much as possible.

 

[siddyp]

what if you were to SCORE each team based on what is your gold standard and then judge/deliberate after all teams are done being scored?

heres something that may have occurred at elite:

(from the averages) JJ is a bit off from UNC and NYPD though it seems a consensus that all three should have been very close.

If we look at the order of the show, JJ first ripped it, then 5 teams went that did not achieve scores as high as JJ, then BOOM: NYPD and UNC. Judges may have viewed that these 2 teams performances blew the previous 2 or 3 out of the water, thus acheiving significantly higher scores than JJ. this could be attributed to the fact that judges were in fact, basing their judging on the performances they just watched.

so why not score EACH team to what you believe to be your gold standard? this way, every team is scored against the SAME criteria in the judges mind. this eliminates scoring teams according to performances that you just watched. when it comes to deciding who wins/places, then you begin comparing/deliberating and in the end, judging who should win/place.

thoughts?

 

[yraparla]

at Bhangre di Mehfil, all the judges followed this (As Barua mentioned)


Question, should judges be allowed to change scores later? Suppose after team 4 they want to change around score for team 1. Should this be allowed. I'm personally not in favor of this because I think then it becomes overly contextual again. Best would be the committee taking the sheets after they're scored, typing it up and handing them back to the judges for reference during the next team. 5 min is plenty of time to type up that many numbers.

 

[Saab]

Get rid of points. They don't mean anything but setting a standard from which you can justify your comments...whereas, I would argue, teams would prefer feedback more than points.

Honestly, the usage of points takes away from the captains meeting discussion. Rather than focus on things during the performance, judges are more worried about points(my personal feelings) with traditional rubrics. It would be better rather to either bracket scores in a certain range, or further utilize a packet that is more based on relativity amongst teams(what your judging).


Think about it...first team is at a more disadvantage with the point system because you have to give point allocation, but there is no concrete standard which every other team performing can enjoy. We shouldn't focus on changing scores, per-say, but relative rankings

 

[MPElive]

I don't think judges should be changing the scores at all which will defeat the purpose of marking the team fairly. Personally, I start off at 75 as my median and mark the team accordingly.

I have seen teams go 1st or 2nd and set the bar high where no team is able to match the performance till the last two teams. In my opinion, teams going early in the show tend to give up before stepping on the stage and not give their 100%.

I am a fan of discussing the winners between top 4 teams to finalize the placings because scores can throw off the rankings of how the teams place.

 

[hsdeol]

With all due respect, and I think many people agree with me on this one, the circuit needs a new wave of qualified judges to enter the scene ASAP. That's all I'm going to say about the judging packet and "deliberation".

i completely agree with you. i feel that a number of dancers just recently crossed over from the active side of dancing to the retired side, and i hope to see a lot of them judging soon. perhaps these new eligible judges can mix with the judges that are currently out there to create a new balance of judging at comps.

 

[oJaLa]

With all due respect, and I think many people agree with me on this one, the circuit needs a new wave of qualified judges to enter the scene ASAP. That's all I'm going to say about the judging packet and "deliberation".

i completely agree with you. i feel that a number of dancers just recently crossed over from the active side of dancing to the retired side, and i hope to see a lot of them judging soon. perhaps these new eligible judges can mix with the judges that are currently out there to create a new balance of judging at comps.

I also think there needs to be a retirement period of maybe one season where the person does not dance before they become a judge. Too many times have I seen judges make comebacks on stage as dancers shortly after they judge a comp. I really don't think you should be judging if you hope to continue dancing.

 

[Karan V]

I think what idols does is take the score sheets after each team has performed. And for show order, thats determined by whoever submits there post registration package(IDs, consents, mix) in first. I think so..

 

[Gaurav Sidhu]

lol if you want proper bhangra judges on the scene then say by by to all this modern bhangra....itll be pure traditional with a hit of creativity...these judges have to be placed there as they like and know modern bhangra..if you put real bhangra judges in the scene say bye bye to all these hardcore gimmicks and say hellow to raw folk bhangra ..just my opinion

 

[Saab]

I disagree. In theory, numbers are the simplest way of breaking down each performance. If a team goes first or last, it doesn't really matter because numbers are impartial. If the criterion is cleanliness out of 10 points, then think of it as percentages. Was 80% of the routine clean with 20% of mistakes? Because each individual criterion is its own x%, it shouldn't matter what the order is because 100% is perfection and Perfection exactly equals perfection. If you go based on those standards, the system works fine. The "gold" standard is still imperfect because that is a subjective level based on expectations. 100% is objective...

Thats great, but numbers don't really achieve anything in the placing area. There can be an alternate system that does the same exact thing and take into account the judging aspects in a rubric. Bigger issue is that, with the difference of judges and their own tastes, it's is more important for judges to come to a consensus when deciding who wins. Its why there having a discussion. If you didn't talk about the results with other judges, and went into the judges meeting with a team, how can you speak about placings critically and explain how your positioning?

I think judging would do better to shift to a more non-numerical perspective. It would benefit the teams more if there was a criterion that focused more on the judges feedback in relation to the scoring than a number format(and I know you might argue the number format does, but it gives judges a backdoor in pointing to the numbers). This way we can evaluate judges better, as they are marked by their observations in terms of dancing.

 

[footballforlife]

We had this exact discussion last year....although it seemed to be primarily Sue, Saleem, and I at the time. We got labeled 'nerds' but it's good to see others realize that it's a problem and needs fixin'


The issue with doing things properly statistically is that it may involve some obfuscation of scores, and you can't trust committees to implement it properly (Adding up number is one thing, normalizing and applying statistical methods increases chances of mistakes).


The three issues are:
Mean (average score)
Variance (spread of the scores)
Range (distance between highest and lowest score)


You can't simply scale/normalize because if one judges scores a bunch of teams together and then one team farther ahead, then scaling will give that first team undue influence based on one judge (and vice versa). At the base level, you have put the mean on 50, then adjust the scores to have the std deviation be the same. Outliers will still have increased influence (as they should). It's not perfect, and I wrote up something last year that did better but I can't remember what it was. I'll try to dig it up the excel sheet for it


We also need to stop dropping highest lowest scores. Although it's not listed in the E8 release, I actually did ask the committee to give us the undropped scores so we could consider that datapoint as well (UNC and NYPD don't change and I think JJ and AEG swap? That's what I remember). Usually one judge has a higher average score and one judge has the lowest average score (pashi and benny in this case) and they get discounted unfairly.

Swi is right. The biggest thing to do is to stop dropping scores. Its confusing why that is done especially when there are only 5 judges, then the averages come from only 3. Data is data. Keep it and use it. Secondly, as Saleem pointed out (first), we cannot just normalize scores. But that is a first step to try to eliminate the 'going first' factor. I recall in last year's thread there was a method presented using sum of variances of judges points to determine placings. That also has some problems but does a better job than a simple normalization.

At the end of the day I think, points should just be a guide for judges to determine what the top 3 should be, although Elite 8 gets an easy out by only having to choose a winner and nothing else (if you consider that easy).

 

[Arjun Barua]

I'm about to go off on a tangent here, but bear with me. I just thought of a new approach to judging:


Specialized judging.


That is to say, each judge would be asked to focus on 1-3 different aspects of the judging rubric, and judge only those criteria across all teams. This would allow each judge to focus his/her attention on only what is necessary, instead of trying to follow every single facet of every team's performance, allowing him/her ample time to take notes and process everything.


Another reason I like this is because each judge is naturally specialized in different areas already. Some, such as myself, are traditional-minded, and the first thing they look for is proper form and grace. Some are creative-minded and like to see unique formations and choreography. Some value uniformity above everything.


So why not play to these strengths and have the judges judge what they know best?