This exam cheating index (collusion detection) simply calculates the number of errors in common between a given pair of examinees.  For example, two examinees got 80/100 correct, meaning 20 errors, and they answered all of the same questions wrongly, the EIC would be 20. If they both scored 80/100 but had only 10 wrong questions in common, the EIC would be 10.  There is no probabilistic evaluation that can be used to flag examinees, as with more advanced indices. In fact, it is used inside some other indices, such as Harpp & Hogan.  However, this index could be of good use from a descriptive or investigative perspective.

Note that EIC is not standardized in any way, so its range and relevant flag cutoff will depend on the number of items in your test, and how much your examinee responses vary.  For a 100-item test, you might want to set the flag at 10 items.  But for a 30-item test, this is obviously irrelevant, and you might want to set it at 5 (because most examinees will probably not even get more than 10 errors).

Learn more about applying EIC with SIFT, a free software program for exam cheating detection and other assessment issues.

Harpp and Hogan (1993) suggested a response similarity index defined as

Response Similarity Index Explanation

EEIC denote the number of exact errors in common or identically wrong, EIC is the number of errors in common.

This is calculated for all pairs of examinees that the researcher wishes to compare. 

One advantage of this approach is that it is extremely simple to interpret: if examinee A and B each get 10 items wrong, 5 of which are in common, and gave the same answer on 4 of those 5, then the index is simply 4/5 = 0.80.  A value of 1.0 would therefore be perfect “cheating” – on all items that both examinees answered incorrectly, they happened to select the same distractor.

The authors suggest utilizing a flag cutoff of with the following reasoning (Harpp & Hogan, 1993, p. 307):

The choice of 0.75 is derived empirically because pairs with less than this fraction were not found to sit adjacent to one another while pairs with greater than this ratio almost always were seated adjacently.

The cutoff can differ from dataset to dataset, so SIFT allows you to specify the cutoff you wish to use for flagging pairs of examinees.  However, because this cutoff is completely arbitrary, a very high value (e.g., 0.95) is recommended by as this index can easily lead to many flaggings, especially if the test is short.  False positives are likely, and this index should be used with great caution.  Wesolowsky (unpublished PowerPoint presentation) called this method “better but not good.”

You may also be interested in the revised version of this index produced by Harpp, Hogan, and Jennings in 1996.

The Frary, Tideman, and Watts (1977) g₂ index is a collusion (cheating) detection index, which is a standardization that evaluates a number of common responses between two examinees in the typical standardized format: observed common responses minus the expectation of common responses, divided by the expected standard deviation of common responses.  It compares all pairs of examinees twice: evaluating examinee copying off b and vice versa.

Frary, Tideman, and Watts (1977) g2 Index

The g₂ collusion index starts by finding the probability, for each item, that the Copier would choose (based on their ability) the answer that the Source actually chose.  The sum of these probabilities than the expected number of equivalent responses.  We can then compare this to the actual observed number of equivalent responses and standardize that difference with the standard deviation.  A very positive value could be possibly indicative of copying.

 

Where

C_ab = Observed number of common responses (e.g., both examinees selected answer D)

k = number of items i

U_ia = Random variable for examinee a’s response to item i

X_ia = Observed response of examinee b to item i.

Frary et al. estimated P using classical test theory, and the definitions are provided in the original paper, while a slightly more clear definitions are provided in Khalid, Mehmood, and Rehman (2011).

The g₂ approach produces two half-matrices, which SIFT presents as a single matrix separated by a blank diagonal.  That is, the lower half of the matrix evaluates whether examinee a copied off b, and the upper half whether b copied off a.  More specifically, the row number is the copier and the column number is the source.  So Row1/Column2 evaluates whether 1 copied off 2, while Row2/Column1 evaluates whether 2 copied off 1.

For g₂ and Wollack’s (1997) ω, the flagging procedure counts all values in the matrix greater than the critical value, so it is possible – likely actually – that each pair will be flagged twice.  So the numbers in those flag total columns will be greater than those in the unidirectional indices.

How to interpret?  This collusion index is standardized onto a z-metric, and therefore can easily be converted to the probability you wish to use.  A standardized value of 3.09 is default for g₂, ω, and Z_jk because this translates to a probability of 0.001.  A value beyond 3.09 then represents an event that is expected to be very rare under the assumption of no collusion.

Want to implement this statistic? Download the SIFT software for free.

Wesolowsky’s (2000) index is a collusion detection index, designed to look for exam cheating by finding similar response vectors amongst examinees. It is in the same family as g₂ and Wollack’s ω.  Like those, it creates a standardized statistic by evaluating the difference between observed and expected common responses and dividing by a standard error.  It is more similar to the g₂ index in that it is based on classical test theory rather than item response theory.  This has the advantage of being conceptually simpler as well as more feasible for small samples (it is well-known that IRT requires minimum sample sizes of 100 to 1000 depending on the model).  However, this of course means that it lacks the conceptual, theoretical, and mathematical appropriateness of IRT, which is the dominant psychometric paradigm for large-scale tests for good reason.

Wesolowsky defined his collusion detection index as

where

Here, the expected number of common responses  is equal to the joint probability of each examinee (j and k) getting item i correct, plus both getting it incorrect with the same distractor t selected.  This is calculated as a single probability for each item then summed across items.  The probability for each item is then of course multiplied by one minus itself to create a binomial variance.

The major difference between this and g₂ is that g₂ estimated the probability using a piecewise linear function that grossly approximated an item response function from IRT.  Wesolowsky utilized a curvilinear function he called “iso-contours” which is better in that it is curvilinear, but it is still not on par with the item response function in terms of conceptual appropriateness.  The iso-contours are described by a parameter Wesolowsky referred to as a (completely unrelated to the IRT discrimination parameter), which must be estimated by bisection approximation.

How to interpret?  This index is standardized onto a z-metric, and therefore can easily be converted to the probability you wish to use.  A standardized value of 3.09 is default for g₂, ω, and Z_jk because this translates to a probability of 0.001.  A value beyond 3.09 then represents an event that is expected to be very rare under the assumption of no collusion.

Want to calculate this index? Download the free program SIFT.

The Holland K index and variants are probability-based indices for psychometric forensics, like the Bellezza & Bellezza indices, but make use of conditional information in their calculations. All three estimate the probability of observing  w_ij  or more identical incorrect responses (that is, EEIC, exact errors in common) between a pair of examinees in a directional fashion. This is defined as

.

Here, W_s is the number of items answered incorrectly by the source, W_cs is the EEIC, and P_r is the probability of the source and copier having the same incorrect response to an item.  So, if the source had 20 items incorrect and the suspected copier had the same answer for 18 of them, we are calculating the probability of having 18 EEIC (the right side), then multiplying it by the number of ways there can be 18 EEICs in a set of 20 items (the middle).  Finally, we do the same for 19 and 20 EEIC and sum up our three values.  In this example, that would likely be summing three very small values because P_r is being taken to large powers and it is a probability such as 0.4.  Such a situation would be very unlikely, so we’d expect a K index value of 0.000012.

If there were no cheating, the copier might have only 3 EEIC with the source, and we’d be summing from 3 up to 20, with the earlier values being relatively large. We’d likely then end up with a value of 0.5 or more.

The key number here is the P_r. The three variants of the K index differ in how it is calculated. Each of them starts by creating a raw frequency distribution of EEIC for a given source to determine an expected probability at a given “score group” r defined by the number of incorrect responses. 

Here, MW refers to the mean number of EEIC for the score group and W_s is still the number of incorrect responses for the source.

The K index (Holland, 1996) uses this raw value. The K₁ index applies linear regression to smooth the distribution, and the K₂ index applies a quadratic regression to smooth it (Sotaridona & Meijer, 2002); because the regression-predicted value is then used, the notation becomes M-hat.  Since these three then only differ by the amount of smoothing used in an intermediate calculation, the results will be extremely close to one another. This frequency distribution could be calculated based on only examinees in the same location, however, SIFT uses all examinees in the data set, as this would create a more conceptually appealing null distribution.

 

S₁ and S₂ apply the same framework of the raw frequency distribution of EEIC, but apply it to a different probability calculation instead of using a Poisson model:

.

S2 is often glossed over in publications as being similar, but it is much more complex.  It contains the Poisson model but calculates the probability of the observed EEIC plus a weighted expectation of observed correct responses in common. This makes much more logical sense because many of the responses that a copier would copy from a smarter student will, in fact, be correct. 

All the other K variants ignore this since it is so much harder to disentangle this from an examinee knowing the correct answer. Sotaridona and Meijer (2003), as well as Sotaridona’s original dissertation, provide treatment on how this number is estimated and then integrated into the Poisson calculations.