Harpp and Hogan (1993) suggested a response similarity index defined as
EEIC denote the number of exact errors in common or identically wrong,
EIC is the number of errors in common.
This is calculated for all pairs of examinees that the researcher wishes to compare.
One advantage of this approach is that it extremely simple to interpret: if examinee A and B each get 10 items wrong, 5 of which are in common, and gave the same answer on 4 of those 5, then the index is simply 4/5 = 0.80. A value of 1.0 would therefore be perfect “cheating” – on all items that both examinees answered incorrectly, they happened to select the same distractor.
The authors suggest utilizing a flag cutoff of with the following reasoning (Harpp & Hogan, 1993, p. 307):
The choice of 0.75 is derived empirically because pairs with less than this fraction were not found to sit adjacent to one another while pairs with greater than this ratio almost always were seated adjacently.
The cutoff can differ from dataset to dataset, so SIFT allows you to specify the cutoff you wish to use for flagging pairs of examinees. However, because this cutoff is completely arbitrary, a very high value (e.g., 0.95) is recommended by as this index can easily lead to many flaggings, especially if the test is short. False positives are likely, and this index should be used with great caution. Wesolowsky (unpublished PowerPoint presentation) called this method “better but not good.”
Latest posts by nthompson (see all)
- What is a question bank? - December 9, 2020
- Psychometrician and Psychometrist: What’s the difference? - November 29, 2020
- ASC to speak at ATP’s first EdTech and Computational Psychometrics Summit - November 17, 2020