SIFT: A new tool for detection of test fraud
Content
Introduction
Test fraud is an extremely common occurrence. We’ve all seen articles about examinee cheating. However, there are very few defensible tools to help detect it. I once saw a webinar from an online testing provider that proudly touted their reports on test security… but it turned out that all they provided was a simple export of student answers that you could subjectively read and form conjectures.
The goal of SIFT is to provide a tool that implements real statistical indices from scientific research on statistical detection of test fraud. It’s user-friendly enough to be used by someone without a PhD in psychometrics or experience in data forensics. SIFT provides more collusion indices and analysis than any other software, making it the industry standard from the day of its release. The science behind SIFT is also implemented in our world-class online testing platform, FastTest which supports computerized adaptive testing known to increase test security.
Interested? Download a free trial version of SIFT!
What is Test Fraud?
As long as tests have been around, people have been trying to cheat them. Anytime there’s a system with stakes or incentives involved, people will try to game it. The root culprit is the system itself, not the test. Blaming the test is just shooting the messenger.
In most cases, the system serves a useful purpose. K-12 assessments provide information on curriculum and teachers, certification tests identify qualified professionals, and so on. To preserve the system’s integrity, we must minimize test fraud.
When it comes to test fraud, the old cliché is true: an ounce of prevention is worth a pound of cure. While I recommend implementing preventative measures to deter fraud, some cases will always occur. SIFT is designed to help find those cases. Additionally, the knowledge that such analysis is being conducted can deter some examinees.
How can SIFT help me with statistical detection of test fraud?
Like other psychometric software, SIFT does not interpret results for you. For example, software for item analysis like Iteman and Xcalibre
doesn’t tell you which items to retire or how to revise them—they provide output for practitioners to analyze. SIFT offers a wide range of outputs to help identify:
- Copying
- Proctor assistance
- Suspect test centers
- Brain dump usage
- Low examinee motivation
YOU decide what’s important for detecting test fraud and look for relevant evidence. More details are provided in the manual, but here’s a glimpse.
SIFT Test Security Data Forensics
SIFT calculates various indices to evaluate potential fraud:
- Collusion Indices: SIFT calculates these for each student pair, summarizing the number of flags.
First, there are a number of indices you can evaluate, as you see above. SIFT will calculate those collusion indices for each pair of students, and summarize the number of flags.
- Brain Dump Detection: Compare examinee responses with known brain dump content, especially content intentionally seeded by the organization.
A certification organization could use SIFT to look for evidence of brain dump makers and takers by evaluating similarity between examinee response vectors and answers from a brain dump site – especially if those were intentionally seeded by the organization! We also might want to find adjacent examinees or examinees in the same location that group together in the collusion index output. Unfortunately, these indices can differ substantially in their conclusions.
- Adjacent Examinee Analysis: Identify students in the same location with suspiciously similar responses.
Additionally, we can roll up many of these statistics to the group level. Below is an example that provides a portion of SIFT output regarding teachers. Note that Gutierrez has suspiciously high scores but without spending much more time. Cheating? Possibly. On the other hand, that is the smallest N, so perhaps the teacher just had a group of accelerated students. Worthington, on the other hand, also had high scores but had notably shorter times – perhaps the teacher was helping?
- Response Time Data: Evaluate time spent on questions to detect irregularities.
Finally, you might want to evaluate time data. SIFT provides this as well.
Group-Level Analysis
SIFT rolls up statistics to the group level. For example, one teacher might have suspiciously high scores without much time spent per question. Is it cheating? Possibly. But perhaps the teacher had a group of accelerated students. Another teacher may show high scores with notably shorter times—perhaps due to unauthorized assistance.
The Story of SIFT
I started developing SIFT in 2012. ASC previously sold a program called Scrutiny!, but we stopped due to compatibility issues with newer Windows versions. Despite that, we kept receiving inquiries.
Determined to create a better tool, I set out to develop SIFT. I aimed to include the analysis from Scrutiny! (the Bellezza & Bellezza index) and much more. After years of business hurdles and countless hours, SIFT was released in July 2016.
Version 1.0 of SIFT includes:
- 10 Collusion Indices (5 probabilistic, 5 descriptive)
- Response Time Analysis
- Group-Level Analysis
- Additional tools for detecting test fraud
While not exhaustive of all literature analyses, SIFT surpasses other options for practitioners.
Suggestions? We’d love to hear from you!
Nathan Thompson, PhD
Latest posts by Nathan Thompson, PhD (see all)
- SIFT: A new tool for detection of test fraud - February 4, 2025
- What is a KSAO, and how does it impact assessment? - January 24, 2025
- Computerized Adaptive Testing (CAT): An Introduction - January 21, 2025
