high jump adaptive testing 2

Computerized adaptive testing ( CAT, adaptive testing, computer-adaptive testing, and adaptive assessment) is an AI-based technology that has existed since the 1980s. But, most assessments in the world still don’t capitalize on the benefits of adaptive testing.

There is, of course, a  large body of scientific research on the topic over the past 40+ years, but many organizations are not aware of the benefits of adaptive testing, do not have the expertise, or think it is simply too complex and expensive (spoiler alert: it isn’t!).

What is adaptive testing?

Adaptive testing is the delivery of a test to an examinee using an algorithm that adapts the difficulty of the test to their ability, as well as adapting the number of items used (no need to waste their time).  It is sort of like the High Jump in Track & Field.  You start the bar in a middling-low position.  If you make it, the bar is raised.  This continues until you fail, and then it is dropped a little.  Or if you fail the first one, it can be dropped until you succeed.

For more info, visit this post.

Benefits of adaptive testing

As you might imagine, by making the test more intelligent, adaptive testing provides a wide range of advantages.  Some of the well-known benefits of adaptive testing, recognized by scholarly psychometric research, are listed below.

Shorter tests

Research has found that adaptive tests produce anywhere from a 50% to 90% reduction in test length.  This is no surprise.  Suppose you have a pool of 100 items.  A top student is practically guaranteed to get the easiest 70 correct; only the hardest 30 will make them think.  Vice versa for a low student.  Middle-ability students do no need the super-hard or the super-easy items.

Why does this matter?  Primarily, it can greatly reduce costs.  Suppose you are delivering 100,000 exams per year in testing centers, and you are paying $30/hour.  If you can cut your exam from 2 hours to 1 hour, you just saved $3,000,000.  Yes, there will be increased costs from the use of adaptive testing, but you will likely save money in the end.

For the K12 assessment, you aren’t paying for seat time, but there is the opportunity cost of lost instruction time.  If students are taking formative assessments 3 times per year to check on progress, and you can reduce each by 20 minutes, that is 1 hour; if there are 500,000 students in your State, then you just saved 500,000 hours of learning.

More precise scores

CAT will make tests more accurate, in general.  It does this by designing the algorithms specifically around how to get more accurate scores without wasting examinee time.

More control of score precision (accuracy)

CAT ensures that all students will have the same accuracy, making the test much fairer.  Traditional tests measure the middle students well but not the top or bottom students.  Is it better than A) students see the same items but can have drastically different accuracy of scores, or B) have equivalent accuracy of scores, but see different items?

Better test security

Since all students are essentially getting an assessment that is tailored to them, there is better test security than everyone seeing the same 100 items.  Item exposure is greatly reduced; note, however, that this introduces its own challenges, and adaptive test algorithms have considerations of their own item exposure.

A better experience for examinees, with reduced fatigue

Adaptive tests will tend to be less frustrating for examinees on all ranges of ability.  Moreover, by implementing variable-length stopping rules (e.g., once we know you are a top student, we don’t give you the 70 easy items), reduces fatigue.

Increased examinee motivation

Since examinees only see items relevant to them, this provides an appropriate challenge.  Low-ability examinees will feel more comfortable and get many more items correct than with a linear test.  High-ability students will get the difficult items that make them think.

Frequent retesting is possible

The whole “unique form” idea applies to the same student taking the same exam twice.  Suppose you take the test in September, at the beginning of a school year, and take the same one again in November to check your learning.  You’ve likely learned quite a bit and are higher on the ability range; you’ll get more difficult items, and therefore a new test.  If it was a linear test, you might see the same exact test.

This allows adaptive tests to be used quite frequently for formative assessments in schools; it is an ideal fit.

Individual pacing of tests

Examinees can move at their own speed.  Some might move quickly and be done in only 30 items.  Others might waver, also seeing 30 items but taking more time.  Still, others might see 60 items.  The algorithms can be designed to maximize the process.

Want to learn more?

Interested in learning more about the benefits of adaptive testing?  If you want a full book, I recommend Computerized Adaptive Testing: A Primer by Howard Wainer.  Prefer a short article for now?  Here’s my favorite.

Want to implement the benefits of adaptive testing?

The first step is to perform simulation studies to evaluate the potential benefits for your organization.  We can help you with those, or recommend software if you prefer to do it on your own.  Ready to develop and deliver your own adaptive tests?  Sign up for a free account on our platform!

The following two tabs change content below.

Nathan Thompson, PhD

Nathan Thompson earned his PhD in Psychometrics from the University of Minnesota, with a focus on computerized adaptive testing. His undergraduate degree was from Luther College with a triple major of Mathematics, Psychology, and Latin. He is primarily interested in the use of AI and software automation to augment and replace the work done by psychometricians, which has provided extensive experience in software design and programming. Dr. Thompson has published over 100 journal articles and conference presentations, but his favorite remains https://pareonline.net/getvn.asp?v=16&n=1.

Latest posts by Nathan Thompson, PhD (see all)