$$$$$$$ COMPREHENSIVE JUDGING SYSTEM $$$$$$$$

[faizan]

Hello,

It was a pleasure serving as judging coordinator and creative director of WBBC this past weekend. Thank you to all the teams for taking your time and effort to help WBBC succeed. Without the teams, we are nothing.

With the wisdom of my years on the circuit and on the WBBC board, I have created something I call the Comprehensive Judging System (CJS). This in my mind should be the new standard in judging for the circuit. The CJS consists of a base of the following elements:

1. Judges who are familiar with the circuit and who understand the genre of the competition they will judge at;
2. Multi-level or multi-view judging (i.e., Upper and Lower Level) with separate rubrics;
3. Judge's video chat sessions at least 6 weeks before the competition;
4. Team's and Judge's feedback on the rubric(s) (no rubric should be the sole work of a single author or committee);
5. Prompt analytics and feedback for teams (3 days after show maximum);
6. CJS Analysis

I implemented all six of these elements at WBBC this year.

I have attached the CJS analysis for WBBC 2014 BELOW. (Authors: Faizan Ghaznavi and Kinnell Shah, in consultation with Yasaswi Raparla). Kinnell and Swi are both Bhangra Geniuses, thank you for your input and assistance. see also: the wbbc 2014 upper and lower level rubrics below.

I hope this system helps move the circuit forward and serves as a turning point for the circuit's judging (which has
lagged far behind the quality of dancing).

Thank you,

Faizan Ghaznavi
Founding Member of Anakh-E-Gabroo
Founder of BX Bhangra
Founding member of NYC Jugni
Judging Coordinator and
Creative Director for World's Best Bhangra Crew

 

[yraparla]

Excellent work Faizan and Kinnell. Truly impressed with the effort you all put in.


My only critique for the doc would be to include a transposed graph that has all the judges scores centered around the same average. This is different from normalization in that the relative points between teams is untouched and we can extrapolate certain information that way. Normalization transforms the matrix and while incredibly useful, provides a different set of information.


I would also include definitions of the rubric categories, or simply include the actual rubric. For instance the category 'choreography' means something specific to me, however my notion of what it entails does not match up with the scores between the teams. This is most likely because the rubric definition explicitly mentions something that I don't naturally account under that category. Others may have similar questions with other categories as well.

 

[J-Skeet]

wow. you guys def get a lot of respect for all of the hard work. Never seen anything like this as far as post-comp follow ups. with that said...I like deliberation.

 

[mrchicity]

Idea for future judging: Allow judges to use their phones to record comments and feedback while a team is performing and provide teams with said recordings later on. One of the difficult parts of judging that most people don't understand is having to watch a performance and try to write down comments while still trying to watch and then allocating points on a rubric before all of it is taken away by a competition organizer as the next act is coming up. Having judges record feedback with a routine in progress will allow them to give transparent critiques without the pressure of looking down to write and potentially missing parts of a performance, as well as allow them more time to deliberate on how to score a team (when you're rushed it's easy to put down a quick number without much care). Put these files on a CD after the comp and give them to teams, they will benefit from having better feedback than "good jugni" or "red dancer +".

 

[hardeep_singh]

Idea for future judging: Allow judges to use their phones to record comments and feedback while a team is performing and provide teams with said recordings later on. One of the difficult parts of judging that most people don't understand is having to watch a performance and try to write down comments while still trying to watch and then allocating points on a rubric before all of it is taken away by a competition organizer as the next act is coming up. Having judges record feedback with a routine in progress will allow them to give transparent critiques without the pressure of looking down to write and potentially missing parts of a performance, as well as allow them more time to deliberate on how to score a team (when you're rushed it's easy to put down a quick number without much care). Put these files on a CD after the comp and give them to teams, they will benefit from having better feedback than "good jugni" or "red dancer +".

i've suggested it in the past, you could mic up all the judges and have the commentary synced to a camera recording from judges perspective, then you get a video with audio from each of the judges

 

[mrchicity]

Also about point number 2: If you have multi-level judging, do you take into account how the scores may be influenced by seating? How can you expect judges who are sitting at the front to score formations as accurately as judges who are sitting at the back and can appreciate the full effect of formation mechanisms?

 

[yraparla]

Also about point number 2: If you have multi-level judging, do you take into account how the scores may be influenced by seating? How can you expect judges who are sitting at the front to score formations as accurately as judges who are sitting at the back and can appreciate the full effect of formation mechanisms?

the rubric accounts for this. Lower level judges do not judge formations but do judge synchronization and energy. Upper level judges focused on formations, choreography and general creative elements

 

[mrchicity]

Also about point number 2: If you have multi-level judging, do you take into account how the scores may be influenced by seating? How can you expect judges who are sitting at the front to score formations as accurately as judges who are sitting at the back and can appreciate the full effect of formation mechanisms?

the rubric accounts for this. Lower level judges do not judge formations but do judge synchronization and energy. Upper level judges focused on formations, choreography and general creative elements

Fair enough, thanks for the clarification.

 

[faizan]

I have attached the upper and lower level rubrics to the original post above.


Also see below for the technical rubric, thanks.

 

[faizan]

See the slide that ranks the teams by rubric category. You can infer a lot from it in terms of why teams lost points or gained them.

 

[SGupta7]

This is truly impressive. I remember seeing Faizan, Kinnell, and Arjun working on this in the lobby of the hotel on Friday night and when they invited me and some other members of my team to take a look I was blown away by the amount of effort they had put into giving teams the most comprehensive judging packet i'll probably ever see. It's great to be able to compare yourself against other teams so you don't have to wonder where you stacked up against the competition. Hopefully other competitions will be able to build off of this system and provide even more in-depth analysis of performances in the future.

 

[shreeps]

Idea for future judging: Allow judges to use their phones to record comments and feedback while a team is performing and provide teams with said recordings later on. One of the difficult parts of judging that most people don't understand is having to watch a performance and try to write down comments while still trying to watch and then allocating points on a rubric before all of it is taken away by a competition organizer as the next act is coming up. Having judges record feedback with a routine in progress will allow them to give transparent critiques without the pressure of looking down to write and potentially missing parts of a performance, as well as allow them more time to deliberate on how to score a team (when you're rushed it's easy to put down a quick number without much care). Put these files on a CD after the comp and give them to teams, they will benefit from having better feedback than "good jugni" or "red dancer +".

i've suggested it in the past, you could mic up all the judges and have the commentary synced to a camera recording from judges perspective, then you get a video with audio from each of the judges

This is a great idea. Back in middle school/high school when I went to band competitions, they did this, and since you could hear the music in the background along with the judges' comments, it was helpful for knowing exactly where and why we lost points. It's not too difficult to do and would be immensely helpful to teams.

 

[jattkharku]

Not to rain on this parade here, but this really doesn't mean anything. Judging will always be inconsistent. You just put it all in a pdf. You had six judges and only one of the six had Cornell winning. All the others had Cornell at third. Cornell won. Not trying to disrespect Cornell, just saying if anything, this proves how terrible judging can be in bhangra.

 

[Basim]

Being a fan of analysis and numbers (especially in an easy to read/concise manner), I was thoroughly impressed by this. Excellent work Faizan, Kinnell, Swi, Arjun and whoever else might have been involved.


Small typo on page 19:
"We can then normalize the all the normalized scores of the judges of a said team to see how opinions and preferences varied between judges and what categories each judge liked for what team more than other judges."

I think it was supposed to say: We can then normalize all of the...

I would also include definitions of the rubric categories, or simply include the actual rubric. For instance the category 'choreography' means something specific to me, however my notion of what it entails does not match up with the scores between the teams. This is most likely because the rubric definition explicitly mentions something that I don't naturally account under that category. Others may have similar questions with other categories as well.

This is an excellent point brought up by Swi and I was going to mention the same thing. I'm not sure if judged where told what exactly goes into each of the categories (in essence, further defining each category), but these general words could mean different things to different people.


Lastly, on a personal level, I have the belief that judges deliberating to shift overall placings by 1 or 2 spots max should be a part of the entire process. It's so hard to gauge what one judge saw more in team over another based on just numbers. Discussion is needed to really combine thoughts and possibly share what might have been missed (or incorrectly interpreted) by one judge over another.


The entire judging process along with setting criteria for a rurbic is always a difficult hurdle for competitions and it's always good to share what a successful competition like WBBC has done. Much appreciated friends.


~ Basim

 

[faizan]

Basim, see the rubrics in my original post for criteria.

Wbbc had deliberation between the top two teams if separated by 15 or less points.

 

[shreeps]

Not to rain on this parade here, but this really doesn't mean anything. Judging will always be inconsistent. You just put it all in a pdf. You had six judges and only one of the six had Cornell winning. All the others had Cornell at third. Cornell won. Not trying to disrespect Cornell, just saying if anything, this proves how terrible judging can be in bhangra.

True, judges will always be inconsistent, but I think Faizan and whoever he worked with have put together a pretty decent system for generating rubrics and keeping teams and judges on the same page. Maybe it worked well this time, maybe not, but I think a lot of the core ideas he had and implemented (very early Google Hangout with judges to discuss what they're looking for, discussions with teams regarding the structure and content of the rubric and the extent to which deliberation plays a part in placings, etc.) are refreshing and should set the foundation for even more improvements.


I also like that WBBC worked so hard to get scores and rubrics to teams immediately. I've been to competitions where we never got our specific scores back, where we didn't see the judges' rubrics (and all the comments they wrote), etc. And regardless of how helpful the extensive analysis of the judges' scores that he and Kinnell (and Swi) put together was, it could be the start of a new standard for competitions to try to provide as much detail regarding judging decisions as possible.


I'm not saying that any of the ideas are perfect, or that your point is incorrect, but I think that Faizan and company have done some pretty awesome work here which has potential to affect the circuit in a positive way.

 

[Bballer]

Cool idea regarding the judging system however I found one small technicality in the rubric weird. It only may be a couple of points but why are points being docked for phumans coming off. I understand if you take points off in a situation like this in the video below at 7:55 but other than that it should not be a technicality on a rubric imo.


BC Cultural Bhangra Academy @ Jashan 2011

 

[jattkharku]

I'm not saying that any of the ideas are perfect, or that your point is incorrect, but I think that Faizan and company have done some pretty awesome work here which has potential to affect the circuit in a positive way.

Not to offend you or Faizan or anyone else, but this is 37 pages of the same scores just organized differently into charts and graphs and matchups etc.

Same judge gave Vancity and AG 8th and 7th respectively. I strongly disagree with that verdict.

WOW! Didn't even notice that. Something is fishy there. VanCity and AG are two of the better teams in the circuit and definitely this competition. No way they were 7th and 8th.

 

[Sue Sylvester]

This is truly impressive. I remember seeing Faizan, Kinnell, and Arjun working on this in the lobby of the hotel on Friday night and when they invited me and some other members of my team to take a look I was blown away by the amount of effort they had put into giving teams the most comprehensive judging packet i'll probably ever see. It's great to be able to compare yourself against other teams so you don't have to wonder where you stacked up against the competition. Hopefully other competitions will be able to build off of this system and provide even more in-depth analysis of performances in the future.

Thanks, Shawn. It was great for Faizan and I to be able to give you, Arjun and others a sneak preview of the packet. Our goal was to have this ready for the judges meeting on Saturday, but unfortunately, we were limited by the speed of our printer and opted to email this out instead.

Not to rain on this parade here, but this really doesn't mean anything. Judging will always be inconsistent. You just put it all in a pdf. You had six judges and only one of the six had Cornell winning. All the others had Cornell at third. Cornell won. Not trying to disrespect Cornell, just saying if anything, this proves how terrible judging can be in bhangra.

Not to rain on your parade here, but the judging packet wasn't meant to find the cure to cancer or teach you how to dougie. It is simply a way to visualize the scores and attempt to understand the data, nothing more. Sorry to disappoint you and get your expectations up.

Ideally, we would love to have all the judges rank the teams similarly and accurately that rewards excellence in bhangra, whatever that may mean. However, in reality, it may get a little messy, even when the rubric is the same for all the judges. However, WBBC had 2 sets of rubrics. Let me give you an example of an extreme that may help you understand why this matters. Bhangra Idols employs a judging system where each judge is given a separate category to judge all the teams. One judge will evaluate formations for all teams, another judge for energy, another judge for choreography, another judge for LED gimmicks (I kid!)..etc. It would be amazing if the team that wins had the highest score in each category, but it's possible that teams that may excel in one category be deficient in another. It might not be common, but it's possible that the team with the highest overall score may not have had the highest score in any category, simply high enough scores in everything in order to have the highest accumulative score. WBBC may not have had 7 different rubrics, but they had 2 distinct rubrics that could have factored into this. With a multiple rubric judging system, a single judge does not decide placings, rather they are a part of the whole judging machine that crank out the placings.

Finally, the point of this packet is certainly not to defend placings. It's simply to show what happened. You claim that this packet means nothing, but I find it amusing that you would have never come your conclusion about Cornell without this packet making that information so freaking easy to access. Judging is not perfect and Faizan took a risk with his judging rubrics in attempt to improve the system. Just a couple of years ago, competitions were throwing out highest and lowest scores until it was proven to be stupid. This doesn't fix judging, this provides us with some information that can be potentially referenced in efforts to better rubrics, the categories, the points..etc.

 

[Sue Sylvester]

I also don't understand how you can dock a team 10 points for "exposing legs," unless 5 dancers unintentionally dropped their chadran off. I feel that if that many points can be taking off for costume failure, you should be taking off at least double for purposely misusing a prop, and for choosing songs that lyrically do not align with Bhangra boliyian norms.

I'm going to assume that you might have skimmed the packet. Many people falsely bring up concern regarding the technical penalties because they assume they are subtracted from each judge. Technical penalties are subtracted from overall total. The technical rubric specifies that 2 pts would be deducted for each dancer that has shown leg. To each 6 judges, this infraction of a dancer exposing themselves to an audience costs the team 0.34 of a point. 0.34 of a point per judge doesn't sound too harsh to me for something that may make most audience members cringe in seeing and most dancers regard as a mistake that decreases the aesthetic value of the performance. The penalty is completely objective (points were also taken off breaking saaps or dropping them accidentally, which constitutes misusing props, unless you think throwing props is misusing them... Song selection can be viewed as subjective at times) and often based on the team's ability to tie their chaadre properly to prevent exposing themselves and dancing within their means to prevent that, upholding a certain presentation on stage as a team. I don't think this is so ridiculous for teams vying for the title of World's Best Bhangra Crew.