psychometric data forensics

An emerging sector in the field of psychometrics is the area devoted to analyzing test data to find cheaters and other illicit or invalid testing behavior.  There is a distinction between primary and secondary collusion, and there are specific collusion detection indices and methods to investigate aberrant testing behavior, such as

We lack a generally agreed-upon and marketable term for that sort of work, and I’d like to suggest that we use Psychometric Forensics.

While research on this topic is more than 50 years old, the modern era did not begin until Wollack published his paper on the Omega index in 1997. Since then, the sophistication and effectiveness of methodology in the field has multiplied, and many more publications focus on it than in the pre-Omega era. This is evidenced by not one but three recent books on the subject:

  1. Wollack, J., & Fremer, J. (2013).  Handbook of Test Security.
  2. Kingston, N., & Clark, A. (2014).  Test Fraud: Statistical Detection and Methodology.
  3. Cizek, G., & Wollack, J. (2016). Handbook of Quantitative Methods for Detecting Cheating on Tests.

In addition, there was the annual Conference on the Statistical Detection of Test Fraud, which recently changed its name to the Conference on Test Security (COTS). Assessment Systems regularly participates in COTS, including in 2015, 2016, and 2017, to stay at the forefront of the latest strategies and technologies for preventing cheating in assessments.

I’m excited about the advances being made, but one thing has always bugged me about this: the name, or lack thereof. There is no name that’s relatively accepted for the topic of statistical detection. Test Security is a good name for the broader topic, which also includes things like security policies for test centers. But not for the data side. You can see that the 2014 and 2016 books as well as the original conference title all have similarity, but don’t agree, which I think in part is because the names are too long and lack the modern marketing oomph. Just like the dreadful company names of the 1960s and 1970s… stuff like Consolidated International Business Logistics Incorporated. No wonder so many companies changed to acronyms in the 1980s.

As I thought about this, I began by laying down my requirements for what I would want in the name:

  1. Short enough to not be a mouthful – preferably two words max
  2. Descriptive enough to provide some initial idea what it is
  3. Not already taken elsewhere (the use of “certification management” in the testing industry is a great example of this failure)
  4. Wide enough to cover cases that are not fraud/malicious but still threats to validity

Statistical Detection of Test Fraud or configurations of it pass Requirements 2 and 3, but certainly fail 1 and 4. Conversely, Data Forensics or similar terms would pass Requirements 1 and 4, but fail 2 and 3 quite badly. I’m also guilty, by naming my software program Software for Investigating Fraud in Testing – mostly because I thought the acronym SIFT fit so well.

I’d like to suggest the term Psychometric Forensics. This meets all 4 requirements, at least within the testing industry (we’re still fighting that uphill battle to get the rest of the world to even understand what psychometrics is). A quick Google search does not find hits, and only finds two instances of psychometric data forensics, which are buried in technical reports that primarily use the term data forensics.

However, I’m not completely sold on it.  Any other suggestions?  I’d love to hear them!

The following two tabs change content below.
Avatar for Nathan Thompson, PhD

Nathan Thompson, PhD

Nathan Thompson earned his PhD in Psychometrics from the University of Minnesota, with a focus on computerized adaptive testing. His undergraduate degree was from Luther College with a triple major of Mathematics, Psychology, and Latin. He is primarily interested in the use of AI and software automation to augment and replace the work done by psychometricians, which has provided extensive experience in software design and programming. Dr. Thompson has published over 100 journal articles and conference presentations, but his favorite remains https://scholarworks.umass.edu/pare/vol16/iss1/1/ .
Avatar for Nathan Thompson, PhD

Latest posts by Nathan Thompson, PhD (see all)