The Angoff Analysis Tool
A free spreadsheet to set cutscores that are legally defensible, using the modified-Angoff method with Beuk compromise and inter-rater reliability
FAIR & DEFENSIBLE
Methods like the Angoff approach are necessary to be defensible or achieve NCCA/ANSI Accreditation and follow AERA/APA/NCME guidelines
BETTER DECISIONS
Setting a better cutscore means that decisions made with the test will be more accurate
IMPROVE EFFICIENCY
Leverage remote raters; interactively discuss Angoff results; improve reliability
Using the Angoff Analysis Tool
The Angoff Analysis Tool supports best practices in implementing the modified-Angoff approach
- Utilize two rounds of ratings (Delphi method) to improve inter-rater reliability and validity
- Perform a “reality check” with the Beuk Compromise
- Estimate mean and SD of the exam forms using classical item statistics (P and Rpbis)
- Estimate a cutscore with the Hofstee, a completely separate method, as a parallel approach
- Evaluate pass rates with competing methods to consider candidate impact
Best of all, this is available for free: simply fill out the form on the right and you can download the spreadsheet.
Need a consultant to run the standard-setting study in a way that meets accreditation standards? Contact us and set up a time to talk to one of our experts.
I’m new to this. Tell me more.
What is standard setting?
Standard setting is the process of setting a cutscore or passing point on an exam, typically to indicate some sort of mastery. For example, by passing the test you might be considered certified to work in a profession or eligible to interview for a certain job. Obviously, there are important stakes associated with such exams, so it is not defensible to pick an arbitrary score as the cutscore. For example, you can’t just pick 70% because that’s what was considered passing on your 5th grade spelling tests. If you are making a criterion-referenced interpretation of the scores, you need a criterion-referenced method of setting the cutscore.
What is a criterion?
Here, it means some well-defined area of knowledge that is being assessed, such as knowledge of accounting practices or essential aspects of being a nurse. Unfortunately, criterion has another meaning in assessment too; it sometimes refers to an external variable used to provide corroborative evidence for test score interpretations, historically known as criterion-related validity.
How can I set a defensible cutscore?
There are several methods available, including modified-Angoff, Bookmark, Hofstee, Body Of Work, Borderline, and Contrasting Groups. This spreadsheet helps you implement the most common method, modified-Angoff, with or without the common add-on of the Beuk Compromise. It also supports the Hofstee method.
If you are a credentialing organization, using methods like these is a requirement of NCCA and ANSI accreditation.
How can I implement the Angoff method?
A step-by-step overview is described in this blog post. You will need a pool of high-quality items, preferably ones that are already pretested. You will then need a panel of at least 6 subject matter experts (SMEs). Need an enterprise-class item banker platform to implement an Angoff study as part of the item authoring/review process? Contact us about FastTest.
