Quicklinks

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 
 
 

Judging Process

The judging process can be downloaded as a PDF here.

Background

  • Australian Event Awards Pty Ltd, ABN 85 154 434 358 (Australian Event Awards), is the owner of the Australian Event Awards (The Event Awards) – the pinnacle awards program for the Australian Events Industry.
  • The Managing Director of Australian Event Awards Pty Ltd appoints and maintains the appointment of a Management Company to provide project management services for The Event Awards. The role of the Management Company is to manage all matters in relation to entrants, entries, payments and administration of the Event Awards on behalf of Australian Event Awards Pty Ltd.
  • The current Management Company is The Epic Team Pty Limited. All enquiries in regards to entry related matters should be directed to The Epic Team at [email protected], or in writing addressed to The Epic Team, Suite 1A1, 410 Elizabeth St, Surry Hills, NSW 2010.
  • Entries into The Event Awards for 2017 are open to events staged in Australia and achievements by Australians during the period from 1 July 2016 to 30 June 2017.
  • Entries for The Event Awards for 2017 open on 3 May, 2017 and close on 17 July, 2017.
  • This document outlines the Judging process which will be used for the 2017 Awards and should be read in conjunction with the Governance & General Rules document.
  • This document will be reviewed prior to the 2018 Awards.

The Judging Panel

  • Entries are judged and awards conferred by the members of the Independent Industry Judging Panel (IIJP).
  • The Managing Director of the Australian Event Awards Pty Ltd appoints and maintains the appointment of at least two Co-Chairs of the Independent Industry Judging Panel.
  • No member, director or employee of Australian Event Awards Pty Ltd or the Management Company may be a Co-Chair or member of the Independent Industry Judging Panel.
  • The Role of the Co-Chairs of the IIJP is to:
    • Determine the Categories, Eligibility Requirements and Judging Criteria of The Event Awards before the opening of entries each year
    • Determine the judging process to be applied to The Event Awards before the opening of entries each year
    • Address any questions regarding eligibility from entrants during the entry submission process
    • Appoint all other members of the Independent Industry Judging Panel
    • Inform and brief all judges on the judging process, criteria and expected outcomes
    • Rule on any real or perceived conflicts of interest in the judging panel
    • Lead the final judging day
    • Ensure the integrity and industry relevance of the Judging Process
    • Provide input and advice on the direction of The Event Awards generally
  • The IIJP is divided into two sub-panels:
    • a Finalist Selection Panel (FSP) and
    • a Winner Selection Panel (WSP)
  • Members of the IIJP can be appointed as a member of either sub-panel or both sub-panels by the Co-Chairs of the IIJP.
  • The Co-Chairs of the Judging Panel may, at any stage, appoint a person to act as a fact-checker to verify claims made in entries, and, following this, to advise the judges on either the FSP or the WSP as to his or her findings.
  • If there are particular issues in the public domain relating to an event or achievement which is featured in an Event Awards entry, this may be drawn to the attention of the IIJP.

Judging Process

Part 1: Finalist Selection

Scoring

  • Each entry is assigned to at least two members of the FSP (FSP Judges) for scoring.
  • Each FSP Judge checks the entries assigned to them and confirms that he/she does not have any real or perceived conflicts of interest in relation to the assigned entries.
  • If a member of the FSP identifies any real or perceived conflicts of interest in relation to the assigned entries, the Judge must immediately notify the Management Company. The Management Company will then bring the conflict to the attention of the Co-Chairs of Judging who will evaluate the existence and seriousness of the conflict and rule that:
    1. no action is required OR
    2. the single entry should be reallocated to another FSP judge OR
    3. the conflicted FSP judge should be removed from judging any entries in that category
  • Members of the FSP are provided with a scoring guide such as the one below for the purposes of calibration:

Score Range (0-7)

Guide

1-2

Demonstrates little or no understanding of the criteria or addresses it at an unsatisfactory level

3-4

Demonstrates satisfactory understanding of the criteria and addresses it at a satisfactory level

5-7

Demonstrates a good working understanding of the criteria and demonstrates a high level of achievement in addressing it

  • Each FSP Judge assesses the entries assigned to them, scoring each entry against the criteria for its Category and providing comments to assist entrants with future entries.
  • As an FSP Judge assesses further entries, they may wish to amend scores from previously-judged entries for consistency. This can be done at any time up until the close of the judging period.
  • Judges may use industry knowledge of the event or achievement when scoring each entry.

Finalist Threshold Score

  • The Finalist Threshold Score is set by the Co-Chairs of the IIJP to establish a quality standard that entries must achieve to be eligible to become a Finalist
  • To be eligible to become a Finalist, an entry must achieve a weighted average raw score that is at least equal to the Finalist Threshold Score.
  • The weighted average raw score is calculated by:
    • adding the raw score from each judge for each criterion
    • multiplying by the weighting of each criterion
    • adding the resulting weighted score for each criterion
    • then dividing by the number of judges for that entry

resulting in a single weighted average raw score for that entry. An example of this is illustrated in the table below:

Criteria

Judge 1 Raw Score

Judge 2 Raw Score

A 40%

3

4

B 20%

4

5

C 15%

6

5

D 15%

2

3

E 10%

4

3

Weighted Average Raw Score = 3.85

  • Entries do not have to meet the Finalist Threshold Score in order to become State or Territory Finalists. Likewise all entries (excluding National Events - those staged in four or more states or territories) accrue points towards the State or Territory Award regardless of whether they have met the Finalist Threshold Score.

Score Moderation

  • Scores for all entries are moderated using an academic scaling algorithm to remove any bias that may have been introduced by the use of different judges across the pool.
  • The result is a score out of 100 (Moderated Score) for each entry.

Automatic Finalists

  • The Entries that receive the top four Moderated Scores in each category are Automatic Finalists, providing they also meet the Finalist Threshold Score.

Finalist Review

  • Each member of the FSP is advised of the Automatic Finalists and members are invited to nominate any entries that are not Automatic Finalists that they believe, against the published Category Criteria, should be considered by the Winner Selection Panel (WSP).
  • Such a nomination must be supported by a statement from the nominating FSP Judge that:
    • gives reasons for the nomination against the Category Criteria; and
    • puts forward the position that, in relation to the Category Criteria, the entry cannot be reasonably distinguished from the Automatic Finalists on the basis of quality.
  • To make such a nomination, the entry must:
    • Have been scored by the nominating judge and received a score from that judge equal to or greater than the Finalist Threshold Score
    • Meet the Finalist Threshold Score
  • Entries nominated for review (Review Entries) are made available to a Review Panel made up of all judges that scored that Category (other than the nominating judge), and the Co-Chairs of the IIJP. The nominating judge may be asked to participate as an advisor to the Review Panel.
  • In any Category where there is a Review Entry, Members of the Review Panel have the opportunity to re-read the Automatic Finalist entries together with any Review Entries. Members of the review panel may, in relation to each Review Entry:
    • support the nominating judge’s view that, in relation to the criteria, the entry should be considered by the WSP
    • reject the nominating judge’s view that, in relation to the criteria, the entry should be considered by the WSP
    • abstain from voting on the nomination (this is compulsory if the member of the Review Panel is conflicted in relation to any of the entries nominated for review or any of the Automatic Finalists for the Category)
  • If any Review Entry receives:
    • at least one vote of support or nominations from two judges; and
    • no votes of rejection
    the Review Entry will become a Review Finalist for the Category.

Finalist Announcement

  • At the end of the Finalist Review, the Automatic Finalists together with any Review Finalists for each Category will become the Category Finalists for the Category and the list of Category Finalists will be released.
  • From this point, no distinction will be drawn between Automatic Finalists and Review Finalists.

Part 2: Winner Selection for all categories except Headline Categories

Category Allocation

  • Each member of the Winner Selection Panel (WSP) is allocated a number of Categories to review and coordinate.
  • Each member of the WSP checks the Category Finalists in the allocated Categories and confirms that he/she does not have any real or perceived conflicts of interest in relation to the entries.
  • If a member of the WSP identifies any real or perceived conflicts of interest in relation to any Category Finalists in an allocated Category, the Judge must immediately notify the Management Company. The Management Company will then bring the conflict to the attention of the Co-Chairs of Judging who will evaluate the existence and seriousness of the conflict and rule that:
    1. no action is required OR
    2. the conflicted judge should be removed from judging any entries in that category
  • Once the confirmation above is made, the member of the WSP becomes the Category Coordinator for each category they've been assigned.
  • This means they are responsible for undertaking detailed review of the Finalist entries in each of their categories prior to Final Judging Day. They will represent the categories assigned to them and, for each category, lead the process described below.
  • Members of the WSP may use industry knowledge of the event or achievement when scoring each entry.
  • Comments made in each Finalist entry by the FSP Judges may be provided to the WSP Category Coordinator.

Final Judging Day

  • All members of the WSP attend the final judging day.
  • Where a member of the WSP becomes unavailable for the final judging day, the Management Company must assign any Categories for which the unavailable member was Coordinator to another member of the WSP or to one of the Co-Chairs of the IIJP. This person then becomes the Category Coordinator for the assigned Category.

Sub-Panel Review

  • The WSP breaks into sub-panels.
  • Each sub-panel is responsible for the in-depth review of every Finalist in the categories allocated to each member of the sub-panel
  • For each Category represented in the sub-panel, the following process is followed:
    • The Category Coordinator introduces the Category Finalists
    • Each member of the sub-panel confirms that he/she does not have any real or perceived conflicts of interest in relation to any of the Category Finalists
    • Where a member of the sub-panel has any real or perceived conflicts of interest in relation to any of the Category Finalists, the sub-panel member must remove themselves from all discussion and voting regarding that Category
    • The Category Coordinator leads a discussion of the Category Finalists with regard to the Criteria.
    • The sub-panel may contact a person nominated by the entrant if they feel further information or clarification is required.
    • The sub-panel decides on a recommendation for the Winner, based on the Category Criteria that will be presented to the full panel.

Full Panel Review

  • The full panel convenes to consider the recommendations of each sub-panel and to vote on the Winner of each Category.
  • The final consideration of each Category proceeds as follows:
    • The Category Coordinator introduces the Category Finalists
    • Each member of the full panel confirms that he/she does not have any real or perceived conflicts of interest in relation to the Category Finalists
    • Where a member of the full panel has any real or perceived conflicts of interest in relation to any of the Category Finalists, the full panel member must remove themselves from all discussion and voting regarding the Category
    • The Category Coordinator presents the sub-panel’s assessment of the Finalists, and the recommendation for the Winner of that Category
    • If requested by any member of the WSP or the Co-Chairs of the IIJP, discussion may take place, covering any questions or concerns that the WSP may have regarding the Category Finalists.
    • The Winning Entry is then selected from the Category Finalists by secret ballot in which each member of the WSP casts a single vote nominating their preference for the winner against the Category Criteria.
    • Votes are sealed in an envelope to be counted at a later date using the 'first past the post' method. The results are not made known to the members of the IIJP.
    • Should there be a tie, the entry with the highest Moderated Score will be declared the winner.

Part 3: Winner Selection for the Headline Categories

Lifetime Achievement Award - One Step Process

  • All members of the WSP will have had the opportunity to review entries via the judges' Portal prior to the judging day.
  • On final judging day, a short discussion may take place covering any questions or concerns that any member of the WSP may have regarding the entries.
  • The Co-Chairs of the IIJP ask for a vote by show of hands on the question, “can you separate the entrants to find one winner”. If a majority cannot, then a vote is held for “can you separate the entrants to find one or two winners.” Again, if a majority cannot then “one, two or three winners” and so on until a majority of the panel (50%+1) agree on how many awards will be made. During this process there should be no discussions over who would be the winner(s), the decision is how many winners there should be.
  • Once the majority have expressed the number of winners that they can separate to, everyone votes knowing they are choosing (say) 2 winners (or whatever the vote in the point above decides).
  • Written votes are cast and nominees are numbered preferentially. If it is decided that two winners will be named then votes are numbered 1,1,2,3,4, etc. For three winners 1,1,1,2,3,4, etc.
  • Ballots are sealed in an envelope to be counted by the Management Company at a later date. The winner is declared by preferential voting (as described below) for a single winner. For multiple winners, the following preferential voting method will be used to ensure equal weighting for each vote:
    • Each judge’s ballot paper is identified by a unique code (not linked to their name so that anonymity is maintained)
    • Multiple copies of the ballot papers (two for two winners, three for three winners etc) are created for each judge code, each showing only the first preference from that judge
    • The original ballot papers are set aside, and the copied vote papers are distributed amongst candidates.
    • The lowest candidate is eliminated the same way as explained below (under Preferential Voting)
    • When a vote paper (showing only the first preference) is eliminated, it is replaced by that judge’s original ballot paper (showing all remaining preferences) and the preferences are distributed as listed
    • Should another vote paper from the same judge be eliminated, it is removed from the pool, because their remaining preferences on the original ballot paper are already elsewhere in the pool and therefore still carried through
    • Should the winner not wish to accept the Award (ie they did not have prior knowledge of the nomination and do not wish to be recognised in this way) then all votes will be recounted with that person’s preferences redistributed.
  • For the avoidance of doubt, there are no finalists for this Category.

State or Territory Award - One Step Process

  • Categories and Criteria remain the same; entrants enter in the normal manner and are judged according to the above process
  • National Events (those staged in four or more states or territories) are excluded from this category.
  • The States and Territories of Australia in which each eligible event in the Best Event Categories occurred is determined and points are allocated to those States and Territories as follows:
    • If the entry is not a Finalist or National Winner: 1 point
    • If the Entry is a Finalist but not the National Winner: 2 points
    • If the Entry is the National Winner: 3 points
  • Every entry that is not a National Event in the Best Event Category accrues points towards their State or Territory regardless of whether or not they meet the Finalist Threshold Score.
  • The total number of points for each State and Territory is calculated and any State or Territory which does not have at 7 points including at least one Finalist and one National Winner is eliminated
  • The total number of points for each of the remaining States and Territories is divided by the population of that State or Territory to yield a State or Territory Score.
  • The State or Territory with the highest State or Territory Score wins the State or Territory Award.
  • For the avoidance of doubt, there are no finalists for this Category.

Australia's Favourite Event - Public Vote Process

  • Australia's Favourite Event is the people's choice Australian Event Award. It is chosen by members of the public from the public events that are finalists in any Best Event Category using the following process:
    • Categories and Criteria remain the same; entrants enter in the normal manner and finalists are selected using the above process
    • Any public events which are finalists in any of the Best Event Categories become the Nominees for Australia's Favourite Event
    • At the point when Finalists are announced, the Nominees for Australia's Favourite Event are advised of:
      • their finalist status
      • the day on which public voting for Australia's Favourite Event commences
      • the day on which the top 4 events nominated for Australia's Favourite Event will be announced as Finalists
    • On the nominated day, on-line voting for Australia's Favourite Event will commence on the Australian Event Awards website
    • Any person with an email address can submit a single vote for the Nominee(s) of their choice. The use of voting robots is strictly forbidden and may, at the discretion of the Management Company result in all votes for the Nominee being cancelled.
    • On the nominated day, the top 4 Australia's Favourite Events will be announced and these events will become the Finalists for Australia's Favourite Event.
    • Voting will continue for the Finalists until the close of voting which will be at a time set by the Management Company on the day of the Australian Event Awards Ceremony.
    • At the close of voting, the event with the largest number of votes wins the Australian Event Award for Australia's Favourite Event

Part 5: State and Territory Finalists and Winners

  • Categories and Criteria remain the same; entrants enter in the normal manner and are judged according to the above process
  • All entries in the Best Event Categories are ranked according to Moderated Score. Meeting the Finalist Threshold Score is not a requirement for becoming a State or Territory Finalist or Winner.
  • For each category, the two highest ranked events that occurred in each state or territory is awarded the State/Territory Finalist status
  • On the Awards night, after the National Finalists and Winners are announced in each category, the State and Territory Winners will be announced and displayed on screen. Certificates for State and Territory Finalists and Winners will be available for collection during the evening

Unforseen Circumstances

  • Should a ruling be required on a matter not covered in this document, the Co-Chairs of the IIJP will make a unanimous ruling on that matter
  • The Co-Chairs of the IIJP may take advice from judges or others as they see fit
  • Generally, resolutions to unforeseen circumstances are documented in the form of a revision to this document ahead of The Event Awards in the following year

Preferential Voting

Instant Runoff Voting (IRV) - a form of preferential voting, is used to decide the outcome of the Lifetime Achievement Award.

Instant Runoff Voting Explained

  • Each voter ranks the list of candidates in order of preference. The voter marks a '1' beside the most preferred candidate, a '2' beside the second-most preferred, and so forth, in ascending order. The mechanics of the process are the same regardless of how many candidates the voter ranks, and how many are left unranked.
  • In the initial count, the first preference of each voter is counted and used to order the candidates. Each first preference counts as one vote for the appropriate candidate. Once all the first preferences are counted, if one candidate holds a majority, that candidate wins. Otherwise the candidate who holds the fewest first preferences is eliminated. If there is an exact tie for last place in numbers of votes, tie-breaking rules determine which candidate to eliminate (see below).
  • The second preferences of ballots assigned to eliminated candidates are recounted and assigned to one of the remaining candidates, potentially producing a different candidate ordering. The process then repeats until one candidate achieves a majority. Ballots that 'exhaust' all their preferences (all its ranked candidates are eliminated) are discarded.

Tie-Breaking Rules

  • In the event that two or more candidates have the lowest number of votes, the following procedures take place:
    • If the tied candidates combined have fewer votes than the next highest candidate, the entire tied set are eliminated at once. Their preferences are then redistributed.
    • If the tied candidates combined do not have fewer votes than the next highest candidate, then the candidate with the least number of first preference votes is eliminated. If there is a tie in first preference votes then the candidate with the least number of second preferences is eliminated. This process continues until eighth preferences have been counted
  • If a tie still exists, the candidate with the lowest Moderated Score is eliminated

Appeals

  • The Event Awards does not have an appeals process. All decisions by the IIJP are final but the IIJP welcomes feedback and encourages constructive criticism aimed at improving The Event Awards.