What are cognitive diagnostic models?
Cognitive diagnostic models are an area of psychometric research that has seen substantial growth in the past decade, though the mathematics behind them, dating back to MacReady and Dayton (1977). The reason that they have been receiving more attention is that in many assessment situations, a simple overall score does not serve our purposes and we want a finer evaluation of the examinee’s skills or traits. For example, the purpose of formative assessment in education is to provide feedback to students on their strengths and weaknesses, so an accurate map of these is essential. In contrast, a professional certification/licensure test focuses on a single overall score with a pass/fail decision.
What are cognitive diagnostic models?
The predominant psychometric paradigm since the 1980s is item response theory (IRT), which is also known as latent trait theory. Cognitive diagnostic models are part of a different paradigm known as latent class theory. Instead of assuming that we are measuring a single neatly unidimensional factor, latent class theory instead tries to assign examinees into more qualitative groups by determining whether they categorized along a number of axes.
What this means is that the final “score” we hope to obtain on each examinee is not a single number, but a profile of which axes they have and which they do not. The axes could be a number of different psychoeducational constructs, but are often used to represent cognitive skills examinees have learned. Because we are trying to diagnose strengths vs. weaknesses, we call it a cognitive diagnostic model.
Example: Fractions
A classic example you might see in the literature is a formative assessment on dealing with fractions in mathematics. Suppose you are designing such a test, and the curriculum includes these teaching points, which are fairly distinct skills or pieces of knowledge.
- Find the lowest common denominator
- Add fractions
- Subtract fractions
- Multiply fractions
- Divide fractions
- Convert mixed number to improper fraction
Now suppose this is one of the questions on the test.
What is 2 3/4 + 1 1/2?
This item utilizes skills 1, 2, and 6. We can apply a similar mapping to all items, and obtain a table. Researchers call this the “Q Matrix.” Our example item is Item 1 here. You’d create your own items and tag appropriately.
| Item | Find the lowest common denominator | Add fractions | Subtract fractions | Multiply fractions | Divide fractions | Convert mixed number to improper fraction |
|---|---|---|---|---|---|---|
| Item 1 | X | X | X | |||
| Item 2 | X | X | ||||
| Item 3 | X | X | ||||
| Item 4 | X | X |
So how do we obtain the examinee’s skill profile?
This is where the fun starts. I used the plural cognitive diagnostic models because there are a number of available models. Just like in item response theory we have the Rasch, 2 parameter, 3 parameter, generalized partial credit, and more. Choice of model is up to the researcher and depends on the characteristics of the test.
The simplest model is the DINA model, which has two parameters per item. The slippage parameter s refers to the probability that a student will get the item wrong if they do have the skills. The guessing parameter g refers to the probability a student will get the item right if they do not have the skills.
The mathematical calculations for determining the skill profile are complex, and are based on maximum likelihood. To determine the skill profile, we need to first find all possible profiles, calculate the likelihood of each (based on item parameters and the examinee response vector), then select the profile with the highest likelihood.
Calculations of item parameters are an order of magnitude greater complexity. Again, compare to item response theory: brute force calculation of theta with maximum likelihood is complex, but can still be done using Excel formulas. Item parameter estimation for IRT with marginal maximum likelihood can only be done by specialized software like Xcalibre. For CDMs, item parameter estimation can be done in software like MPlus or R (see this article).
In addition to providing the most likely skill profile for each examinee, the CDMs can also provide the probability that a given examinee has mastered each skill. This is what can be extremely useful in certain contexts, like formative assessment.
How can I implement cognitive diagnostic models?
The first step is to analyze your data to evaluate how well CDMs work by estimating one or more of the models. As mentioned, this can be done in software like MPlus or R. Actually publishing a real assessment that scores examinees with CDMs is a greater hurdle.
Most tests that use cognitive diagnostic models are proprietary. That is, a large K12 education company might offer a bank of prefabricated formative assessments for students in grades 3-12. That, of course, is what most schools need, because they don’t have a PhD psychometrician on staff to develop new assessments with CDMs. And the testing company likely has several on staff.
On the other hand, if you want to develop your own assessments that leverage CDMs, your options are quite limited. I recommend our FastTest platform for test development, delivery, and analytics.
This is cool! I want to learn more!
I like this article by Alan Huebner, which talks about adaptive testing with the DINA model, but has a very informative introduction on CDMs.
Jonathan Templin, a professor at the University of Iowa, is one of the foremost experts on the topic. Here is his website. Lots of fantastic resources.
This article has an introduction to different CDM models, and guidelines on estimating parameters in R.
Nathan Thompson, PhD
Latest posts by Nathan Thompson, PhD (see all)
- What is a T score? - April 15, 2024
- Item Review Workflow for Exam Development - April 8, 2024
- Likert Scale Items - February 9, 2024
