Have you heard about standard setting approaches such as the Hofstee method, or perhaps the Angoff, Ebel, Nedelsky, or Bookmark methods? There are certainly various ways to set a defensible cutscore or a professional credentialing or pre-employment test. Today, we are going to discuss the Hofstee method.
Why Standard Setting?
Certification organizations that care about the quality of their examinations need to follow best practices and international standards for test development, such as the Standards laid out by the National Commission for Certifying Agencies (NCCA). One component of that is standard setting, also known as cutscore studies. One of the most common and respected approaches for that is the modified-Angoff methodology.
However, the Angoff approach has one flaw: the subject matter experts (SMEs) tend to expect too much out of minimally competent candidates, and sometimes set a cutscore so high that even they themselves would not pass the exam. There are several reasons this can occur. For example, raters might think “I would expect anyone that worked for me to know how to do this” and not consider the fact that people who work for them might have 10 years of experience while test candidates could be fresh out of training/school and have the topic only touched on for 5 minutes. SMEs often forget what it was like to be a much younger and inexperienced version of themselves.
For this reason, several compromise methods have been suggested to compare the Angoff-recommended cutscore with a “reality check” of actual score performance on the exam, allowing the SMEs to make a more informed decision when setting the official cutscore of the exam. I like to use the Beuk method and the Hofstee method.
The Hofstee Method
One method of adjusting the cutscore based on raters’ impressions of the difficulty of the test and possible pass rates is the Hofstee method (Mills & Melican, 1987; Cizek, 2006; Burr et al., 2016). This method requires the raters to estimate four values:
- The minimum acceptable failure rate
- The maximum acceptable failure rate
- The minimum cutscore, even if all examinees failed
- The maximum cutscore, even if all examinees passed
The first two values are failure rates, and are therefore between 0% and 100%, with 100% indicating a test that is too difficult for anyone to pass. The latter two values are on the raw score scale, and therefore range between 0 and the number of items in the test, again with a higher value indicating a more difficult cutscore to achieve.
These values are paired, and the line that passes through the two points estimated. The intersection of this line with the failure rate function, is the recommendation of the adjusted cutscore.
How can I use the Hofstee Method?
Unlike the Beuk, the Hofstee method does not utilize the Angoff ratings, so it represents a completely independent reality check. In fact, it is sometimes used as a standalone cutscore setting method itself, but because it does not involve rating of every single item, I recommend it be used in concert with the Angoff and Beuk approaches.
How can you perform all the calculations that go into the Hofstee method? Well, you don’t need to program it all from scratch. Just head over to our Angoff Analysis Tool page and download a copy for yourself.
Nathan Thompson, PhD, is CEO and Co-Founder of Assessment Systems Corporation (ASC). He is a psychometrician, software developer, author, and researcher, and evangelist for AI and automation. His mission is to elevate the profession of psychometrics by using software to automate psychometric work like item review, job analysis, and Angoff studies, so we can focus on more innovative work. His core goal is to improve assessment throughout the world.
Nate was originally trained as a psychometrician, with an honors degree at Luther College with a triple major of Math/Psych/Latin, and then a PhD in Psychometrics at the University of Minnesota. He then worked multiple roles in the testing industry, including item writer, test development manager, essay test marker, consulting psychometrician, software developer, project manager, and business leader. He is also cofounder and Membership Director at the International Association for Computerized Adaptive Testing (iacat.org). He’s published 100+ papers and presentations, but his favorite remains https://scholarworks.umass.edu/pare/vol16/iss1/1/.
