Posts on psychometrics: The Science of Assessment

The COVID-19 Pandemic drastically changing all aspects of our world, and one of the most impacted areas is educational assessment and other types of assessment. Many organizations still delivered tests with methodologies from 50 years ago, such as putting 200 examinees in a large room with desks, paper exams, and a pencil. COVID-19 is forcing many organizations to pivot, which provides an opportunity to modernize assessments. But how can we maintain assessment security – and therefore validity – through these changes? Here are some suggestions, all of which can be easily implemented on ASC’s industry-leading assessment platforms. Get started by signing up for a free account at http://assess.ai.

True item banking with content access

A good assessment starts with good items. While learning management systems (LMSs) and other not-true-assessment platforms include some item authoring functionality, they usually don’t meet the basic requirements for real item banking. Here are some examples

  • Items are reusable
  • Item version tracking
  • User edit tracking and audits
  • Author content controls
  • Store metadata such as item response theory statistics
  • Track item usage across tests
  • Item review workflow

Role based access

All users should be limited by roles, such as Item Author, Item Review, Test Publisher, and Examinee Manager. So, for example, someone in charge of managing the roster of examinees/students might never see any test questions.

Data Forensics

There are many ways you can analyze your test results to look for possible security/validity threats. Our SIFT software provides a free platform to get started.

Randomization

When tests are delivered online, you should have the option to randomize item order and also answer order. When printing to paper, there should be an option to randomize the order.

Linear on the fly testing (LOFT)

LOFT will create a uniquely randomized test for each examinee. For example, you might have a pool of 300 items spread across 4 domains, and set each examinee will receive 100 items with 25 from each domain. This greatly increases security.

Computerized adaptive testing (CAT)

CAT takes the personalization even further, and adapts the difficulty of the exam and the number of items seen to each student, based on certain psychometric goals and algorithms. This makes the test extremely secure.

Lockdown browser

Want to ensure that the student can’t surf for answers, or take screenshots of items? You need a lockdown browser. ASC’s assessment platforms, Assess.ai and FastTest, both come with this out of the box and at no extra cost.

Examinee test codes

Want to make sure the right person takes the right exam? Generate unique one-time passwords to be given out by a proctor after identity verification.

Proctor codes

Want an extra layer on the test kickoff procedure? After a student has their identity verified, and enters their code, then the proctor needs to also enter a different password that is unique to them on that day.

Date/Time Windows

Want to prevent examinees from logging in early or late? Set up a specific time window, such as 9-12AM on Friday.

AI-based proctoring

This level of proctoring is relatively inexpensive and scalable, and does a great job of validating the results of an individual examinee. However, it does nothing to protect the intellectual property of your exam questions. If an examinee steals all the questions, you won’t know until later.

Live Online Proctoring

If you can’t do in-person test centers because of COVID, this is the next best option. Live proctors can check in the candidate, verify identity, and implement all the other things above. In addition, they can verify the examinee’s environment and stop the exam if they see the examinee stealing questions or other major issues. MonitorEDU is a great example of this.

How can I start?

Need a hand implementing some of these measures? Or just want to talk about the possibilities? Email ASC at solutions@assess.com.

This collusion detection (test cheating) index simply calculates the number of responses in common between a given pair of examinees.  For example, both answered ‘B’ to a certain item regardless of whether it was correct or incorrect.  There is no probabilistic evaluation that can be used to flag examinees.  However, it could be of good use from a descriptive or investigative perspective.  It has a major flaw in that we expect it to be very high for high-ability examinees.  If two smart examinees both get 99/100 correct, the minimum RIC they could have is 98/100.  Even if they have never met each other and have no possibility of collusion or cheating.

Note that RIC is not standardized in any way, so its range and relevant flag cutoff will depend on the number of items in your test, and how much your examinee responses vary.  For a 100-item test, you might want to set the flag at 90 items.  But for a 50-item test, this is obviously irrelevant, and you might want to set it at 45.

This extremely basic collusion detection index simply calculates the number of responses in common between a given pair of examinees.  For example, suppose two examinees got 80/100 correct on a test. Of the 20 each got wrong, they had 10 in common. Of those, they gave the same wrong answer on 5 items. This means that the EEIC would be 5. Why does this index provide evidence of collusion detection? Well, if you and I both get 20 items wrong on a test (same score), that’s not going to raise any eyebrows. But what if we get the same 20 items wrong? A little more concerning. What if we gave the same exact wrong answers on all of those 20? Definitely cause for concern!

There is no probabilistic evaluation that can be used to flag examinees.  However, it could be of good use from a descriptive or investigative perspective. Because it is of limited use by itself, it was incorporated into more advanced indices, such as Harpp, Hogan, and Jennings (1996).

Note that because EEIC is not standardized in any way, so its range and relevant flag cutoff will depend on the number of items in your test, and how much your examinee responses vary.  For a 100-item test, you might want to set the flag at 10 items.  But for a 20-item test, this is obviously irrelevant, and you might want to set it at 5 (because most examinees will probably not even get more than 10 errors).

EEIC is easy to calculate, but you can download the SIFT software for free.

This exam cheating index (collusion detection) simply calculates the number of errors in common between a given pair of examinees.  For example, two examinees got 80/100 correct, meaning 20 errors, and they answered all of the same questions wrongly, the EIC would be 20. If they both scored 80/100 but had only 10 wrong questions in common, the EIC would be 10.  There is no probabilistic evaluation that can be used to flag examinees, as with more advanced indices. In fact, it is used inside some other indices, such as Harpp & Hogan.  However, this index could be of good use from a descriptive or investigative perspective.

Note that EIC is not standardized in any way, so its range and relevant flag cutoff will depend on the number of items in your test, and how much your examinee responses vary.  For a 100-item test, you might want to set the flag at 10 items.  But for a 30-item test, this is obviously irrelevant, and you might want to set it at 5 (because most examinees will probably not even get more than 10 errors).

Learn more about applying EIC with SIFT, a free software program for exam cheating detection and other assessment issues.

Harpp, Hogan, and Jennings (1996) revised their Response Similarity Index somewhat from Harpp and Hogan (1993). This produced a new equation for a statistic to detect collusion and other forms of exam cheating:

where

EEIC denote the number of exact errors in common or identically wrong,

D is the number of items with a different response.

Note that D is calculated across all items, not just incorrect responses, so it is possible (and likely) that D>EEIC.  Therefore, the authors suggest utilizing a flag cutoff of 1.0 (Harpp, Hogan, & Jennings, 1996):

Analyses of well over 100 examinations during the past six years have shown that when this number is ~1.0 or higher, there is a powerful indication of cheating.  In virtually all cases to date where the exam has ~30 or more questions, has a class average <80% and where the minimum number of EEIC is 6, this parameter has been nearly 100% accurate in finding highly suspicious pairs.

However, Nelson (2006) has evaluated this index in comparison to Wesolowsky’s (2000) index and strongly recommends against using the HHJ.  It is notable that neither makes any attempt to evaluate probabilities or standardize.  Cizek (1999) notes that both Harpp-Hogan methods do not even receive attention in the psychometric literature.

This approach has very limited ability to detect cheating when the source has a high ability level. While individual classroom instructors might find the EEIC/D straightforward and useful, there are much better indices for use in large-scale, highstakes examinations.

Harpp and Hogan (1993) suggested a response similarity index defined as   

                                                                            

where

EEIC denote the number of exact errors in common or identically wrong,

EIC is the number of errors in common.

This is calculated for all pairs of examinees that the researcher wishes to compare. 

One advantage of this approach is that it extremely simple to interpret: if examinee A and B each get 10 items wrong, 5 of which are in common, and gave the same answer on 4 of those 5, then the index is simply 4/5 = 0.80.  A value of 1.0 would therefore be perfect “cheating” – on all items that both examinees answered incorrectly, they happened to select the same distractor.

The authors suggest utilizing a flag cutoff of with the following reasoning (Harpp & Hogan, 1993, p. 307):

The choice of 0.75 is derived empirically because pairs with less than this fraction were not found to sit adjacent to one another while pairs with greater than this ratio almost always were seated adjacently.

The cutoff can differ from dataset to dataset, so SIFT allows you to specify the cutoff you wish to use for flagging pairs of examinees.  However, because this cutoff is completely arbitrary, a very high value (e.g., 0.95) is recommended by as this index can easily lead to many flaggings, especially if the test is short.  False positives are likely, and this index should be used with great caution.  Wesolowsky (unpublished PowerPoint presentation) called this method “better but not good.”

This index evaluates error similarity analysis (ESA), namely estimating the probability that a given pair of examinees would have the same exact errors in common (EEIC), given the total number of errors they have in common (EIC) and the aggregated probability P of selecting the same distractor.  Bellezza and Bellezza utilize the notation of k=EEIC and N=EIC, and calculate the probability

Note that this is summed from k to N; the example in the original article is that a pair of examinees had N=20 and k=18, so the equation above is calculated three times (k=18, 19, 20) to estimate the probability of having 18 or more EEIC out of 20 EIC.  For readers of the Cizek (1999) book, note that N and k are presented correctly in the equation but their definitions in the text are transposed.

The calculation of P is left to the researcher to some extent.  Published resources on the topic note that if examinees always selected randomly amongst distractors, the probability of an examinee selecting a given distractor is 1/d, where d is the number of incorrect answers, usually one less than the total number of possible responses.  Two examinees randomly selecting the same distractor would be (1/d)(1/d).  Summing across d distractors by multiplying by d, the calculation of P would be

That is, for a four-option multiple choice item, d=3 and P=0.3333.  For a five-option item, d=4 and P=0.25.

However, examinees most certainly do not select randomly amongst distractors. Suppose a four-option multiple-choice item was answered correctly by 50% (0.50) of the sample.  The first distractor might be chosen by 0.30 of the sample, the second by 0.15, and the third by 0.05.  SIFT calculates these probabilities and uses the observed values to provide a more realistic estimate of P

SIFT therefore calculates this error similarity analysis index using the observed probabilities and also the random-selection assumption method, labeling them as B&B Obs and B&B Ran, respectively.  The indices are calculated all possible pairs of examinees or all pairs in the same location, depending on the option selected in SIFT. 

How to interpret this index?  It is estimating a probability, so a smaller number means that the event can be expected to be very rare under the assumption of no collusion (that is, independent test taking).  So a very small number is flagged as possible collusion.  SIFT defaults to 0.001.  As mentioned earlier, implementation of a Bonferroni correction might be prudent.

The software program Scrutiny! also calculates this ESA index.  However, it utilizes a normal approximation rather than exact calculations, and details are not given regarding the calculation of P, so its results will not agree exactly with SIFT.

Cizek (1999) notes:

Scrutiny! uses an approach to identifying copying called “error similarity analysis” or ESA—a method which, unfortunately, has not received strong recommendation in the professional literature. One review (Frary, 1993) concluded that the ESA method: 1) fails to utilize information from correct response similarity; 2) fails to consider total test performance of examinees; and 3) does not take into account the attractiveness of wrong options selected in common. Bay (1994) and Chason (1997) found that ESA was the least effective index for detecting copying of the three methods they compared.

Want to implement this statistic? Download the SIFT software for free.

The Frary, Tideman, and Watts (1977) g2 index is a collusion (cheating) detection index, which is a standardization that evaluates number of common responses between two examinees in the typical standardized format: observed common responses minus the expectation of common responses, divided by the expected standard deviation of common responses.  It compares all pairs of examinees twice: evaluating examinee a copying off b and vice versa.

The g2 collusion index starts by finding the probability, for each item, that the Copier would choose (based on their ability) the answer that the Source actually chose.  The sum of these probabilities then the expected number of equivalent responses.  We can then compare this to the actual observed number of equivalent responses and standardize that difference with the standard deviation.  A very positive value could be possibly indicative of copying.

Where

Cab = Observed number of common responses (e.g., both examinees selected answer D)

k = number of items i

Uia = Random variable for examinee a’s response to item i

Xia = Observed response of examinee b to item i.

Frary et al. estimated P using classical test theory, and the definitions are provided in the original paper, while a slightly more clear definitions are provided in Khalid, Mehmood, and Rehman (2011).

The g2 approach produces two half-matrices, which SIFT presents as a single matrix separated by a blank diagonal.  That is, the lower half of the matrix evaluates whether examinee a copied off b, and the upper half whether b copied off a.  More specifically, the row number is the copier and the column number is the source.  So Row1/Column2 evaluates whether 1 copied off 2, while Row2/Column1 evaluates whether 2 copied off 1.

For g2 and Wollack’s (1997) ω, the flagging procedure counts all values in the matrix greater than the critical value, so it is possible – likely actually – that each pair will be flagged twice.  So the numbers in those flag total columns will be greater than those in the unidirectional indices.

How to interpret?  This collusion index is standardized onto a z-metric, and therefore can easily be converted to the probability you wish to use.  A standardized value of 3.09 is default for g2, ω, and Zjk because this translates to a probability of 0.001.  A value beyond 3.09 then represents an event that is expected to be very rare under the assumption of no collusion.

Want to implement this statistic? Download the SIFT software for free.

Wollack (1997) adapted the standardized collusion index of Frary, Tidemann, and Watts (1977) g2 to item response theory (IRT) and produced the Wollack Omega (ω) index.  It is clear that the graphics in the original article by Frary et al. (1977) were crude classical approximations of an item response function, so Wollack replaced the probability calculations from the classical approximations with those from IRT.  The probabilities could be calculated with any IRT model.  Wollack suggested Bock’s Nominal Response Model since it is appropriate for multiple choice data, but that model is rarely used in practice and very few IRT software packages support it.  SIFT instead supports the use of dichotomous models: 1-parameter, 2-parameter, 3-parameter, and Rasch.

Because of the use of IRT, implementation of ω requires additional input.  You must include the IRT item parameters in the control tab as well as examinee theta values in the examinee tab.  If any of that input is missing, the omega output will not be produced.

The ω index is defined as

Where P is the probability of an examinee with θa selecting the response that examinee b selected, and cab is the RIC.  That is, the probability that the copier, with θa, would select the responses that the source did; when summed, this can be interpreted as the expected RIC.  Note that this uses all responses, not just errors.

How to interpret?  The value will be higher when the copier had more responses in common with the source than we’d expect from a person of that (probably lower) ability.  This index is standardized onto a z-metric, and therefore can easily be converted to the probability you wish to use.  A standardized value of of 3.09 is default for g2, ω, and Zjk because this translates to a probability of 0.001.  A value beyond 3.09 then represents an event that is expected to be very rare under the assumption of no collusion.

Interested in applying the Wollack Omega index to your data? Download the SIFT software for free.

Wesolowsky’s (2000) index is a collusion detection index, designed to look for exam cheating by finding similar response vectors amongst examinees. It is in the same family as g2 and Wollack’s ω.  Like those, it creates a standardized statistic by evaluating the difference between observed and expected common responses and dividing by a standard error.  It is more similar to the g2 index in that it is based on classical test theory rather than item response theory.  This has the advantage of being conceptually simpler as well as more feasible for small samples (it is well-known that IRT requires minimum sample sizes of 100 to 1000 depending on the model).  However, this of course means that it lacks the conceptual, theoretical, and mathematical appropriateness of IRT, which is the dominant psychometric paradigm for large-scale tests for good reason.

Wesolowsky defined his collusion detection index as

where

and

Here, the expected number of common responses  is equal to the joint probability of each examinee (j and k) getting item i correct, plus both getting it incorrect with the same distractor t selected.  This is calculated as a single probability for each item then summed across items.  The probability for each item is then of course multiplied by one minus itself to create a binomial variance.

The major difference between this and g2 is that g2 estimated the probability using a piecewise linear function that grossly approximated an item response function from IRT.  Wesolowsky utilized a curvilinear function he called “iso-contours” which is better in that it is curvilinear, but it is still not on par with the item response function in terms of conceptual appropriateness.  The iso-contours are described by a parameter Wesolowsky referred to as a (completely unrelated to the IRT discrimination parameter), which must be estimated by bisection approximation.

How to interpret?  This index is standardized onto a z-metric, and therefore can easily be converted to the probability you wish to use.  A standardized value of 3.09 is default for g2, ω, and Zjk because this translates to a probability of 0.001.  A value beyond 3.09 then represents an event that is expected to be very rare under the assumption of no collusion.

Want to calculate this index? Download the free program SIFT.