Angoff Analysis Tool: Defensible Standard Setting
A free spreadsheet to set cutscores that are legally defensible, using modified-Angoff standard-setting with the Beuk compromise & inter-rater reliability; no, you can’t just pick a nice round score like 70%!
Prefer to have an expert handle the process? Get in touch.
Fair and Defensible
Methods like the Angoff approach are necessary to be defensible or achieve NCCA/ANSI Accreditation and follow AERA/APA/NCME guidelines
Backed by Science
The tool implements methods from scientific journals, focused on strict methodology and complex data science
Better Decisions
Setting a better cutscore means that decisions made with the test will be more accurate
Streamline the Process
Leverage remote raters; interactively discuss Angoff results; improve reliability
Setting Cutscores with the Angoff Analysis Tool
The Angoff Analysis Tool supports best practices and provides a calculator in implementing the modified-Angoff approach
- Utilize two rounds of ratings (Delphi method) to improve inter-rater reliability and validity
- Perform a “reality check” with the Beuk Compromise
- Estimate mean and SD of the exam forms using classical item statistics (P and Rpbis)
- Estimate a cutscore with the Hofstee, a completely separate method, as a parallel approach
- Evaluate pass rates with competing methods to consider candidate impact
Best of all, this is available for free: simply fill out the form on the right and you can download the spreadsheet.
Need a consultant to run the standard-setting study in a way that meets accreditation standards? Contact us and set up a time to talk to one of our experts.
What is Standard Setting?
Standard setting is the process of setting a cutscore or passing point on an exam, typically to indicate some sort of mastery. For example, by passing the test you might be considered certified to work in a profession or eligible to interview for a certain job. Obviously, there are important stakes associated with such exams, so it is not defensible to pick an arbitrary score as the cutscore. For example, you can’t just pick 70% because that’s what was considered passing on your 5th grade spelling tests. If you are making a criterion-referenced interpretation of the scores, you need a criterion-referenced method of setting the cutscore.
I’m new to this. Tell me more!
How can I implement Angoff standard-setting?
A step-by-step overview is described in this blog post. You will need a pool of high-quality items, preferably ones that are already pretested. You will then need a panel of at least 6 subject matter experts (SMEs). Need an enterprise-class item banker platform to implement an Angoff study as part of the item authoring/review process? Contact us about FastTest.
What is a criterion?
Here, it means some well-defined area of knowledge that is being assessed, such as knowledge of accounting practices or essential aspects of being a nurse. Unfortunately, criterion has another meaning in assessment too; it sometimes refers to an external variable used to provide corroborative evidence for test score interpretations, historically known as criterion-related validity.
Are there other methods to set a cutscore?
There are several types of studies you can conduct, including modified-Angoff, Bookmark, Hofstee, Body Of Work, Borderline, and Contrasting Groups. This spreadsheet helps you implement the most common method, modified-Angoff, with or without the common add-on of the Beuk Compromise. It also supports the Hofstee method. Learn more about the different approaches. If you are a credentialing organization, using methods like these is a requirement of NCCA and ANSI accreditation.
Get in touch with one of our consultants to learn more.
Request a Consultation