The Angoff Analysis Tool

A free spreadsheet to set cutscores that are legally defensible, using modified-Angoff standard-setting with the Beuk compromise & inter-rater reliability; no, you can’t just pick 70%!

FAIR & DEFENSIBLE

Methods like the Angoff approach are necessary to be defensible or achieve NCCA/ANSI Accreditation and follow AERA/APA/NCME guidelines

BETTER DECISIONS

Setting a better cutscore means that decisions made with the test will be more accurate; calculate inter-rater reliability

IMPROVE EFFICIENCY

Leverage remote raters; interactively discuss Angoff results; improve reliability

Using the Angoff Analysis Tool

The Angoff Analysis Tool supports best practices and provides a calculator in implementing the modified-Angoff approach

Utilize two rounds of ratings (Delphi method) to improve inter-rater reliability and validity
Perform a “reality check” with the Beuk Compromise
Estimate mean and SD of the exam forms using classical item statistics (P and Rpbis)
Estimate a cutscore with the Hofstee, a completely separate method, as a parallel approach
Evaluate pass rates with competing methods to consider candidate impact

Best of all, this is available for free: simply fill out the form on the right and you can download the spreadsheet.

Need a consultant to run the standard-setting study in a way that meets accreditation standards? Contact us and set up a time to talk to one of our experts.

Get in touch

I’m new to this. Tell me more.

What is standard setting?

Standard setting is the process of setting a cutscore or passing point on an exam, typically to indicate some sort of mastery. For example, by passing the test you might be considered certified to work in a profession or eligible to interview for a certain job. Obviously, there are important stakes associated with such exams, so it is not defensible to pick an arbitrary score as the cutscore. For example, you can’t just pick 70% because that’s what was considered passing on your 5th grade spelling tests. If you are making a criterion-referenced interpretation of the scores, you need a criterion-referenced method of setting the cutscore.

What is a criterion?

Here, it means some well-defined area of knowledge that is being assessed, such as knowledge of accounting practices or essential aspects of being a nurse. Unfortunately, criterion has another meaning in assessment too; it sometimes refers to an external variable used to provide corroborative evidence for test score interpretations, historically known as criterion-related validity.

How can I set a defensible cutscore?

There are several types of studies you can conduct, including modified-Angoff, Bookmark, Hofstee, Body Of Work, Borderline, and Contrasting Groups. This spreadsheet helps you implement the most common method, modified-Angoff, with or without the common add-on of the Beuk Compromise. It also supports the Hofstee method. Learn more about the different approaches.

If you are a credentialing organization, using methods like these is a requirement of NCCA and ANSI accreditation.

How can I implement Angoff standard-setting?

A step-by-step overview is described in this blog post. You will need a pool of high-quality items, preferably ones that are already pretested. You will then need a panel of at least 6 subject matter experts (SMEs). Need an enterprise-class item banker platform to implement an Angoff study as part of the item authoring/review process? Contact us about FastTest.