Review of CopyDetect: R Package for Forensics

If you deal with data science on a daily basis, which psychometrics most definitely is, you’ve probably used R.  R is an environment that allows you to implement packages for many different types of analysis, which are built by a massive community of data scientists around the world.  These are completely self-policed, so there is a wide range of quality.  One package that is specific to psychometrics is called CopyDetect, and it’s designed particularly for psychometric forensics.  Here are some of my thoughts on CopyDetect and similar packages in R.

Note: This review is written in October 2017 for v1.2, and R packages are frequently updated by their authors, so some issues might be addressed in the future.

What is CopyDetect?

CopyDetect (documentation here) is a package, meaning that you must first install it and then integrate it into your command scripts, write and run all the code you need for importing and formatting your data, at which point you can call it using command code designed by the author.

An example in the documentation is something like you see below, which calls CopyDetect to compare the response strings of Examinee #30 to Examinee #70.

CopyDetect1(data=responses,item.par=est.ipar,pair=c(30,70))

This assumes, of course, at even a higher level than you are comfortable with writing code – most psychometricians are, but certainly not most professionals in the assessment field.

Advantages of CopyDetect

1. It is free.  Like many R packages, it enjoys attention simply because of the price tag.
2. It calculates well-respected, modern indices such as Wollack’s Omega and the K-variants.  In contrast, an old program called Scrutiny! calculated only one index, and it wasn’t a very good one.  Some other software only calculates really bad indices.
3. It’s not hard to get it running, at a basic level.  There have been other packages that I’ve simply had to give up on.
4. It is being updated.  Because it is maintained by a university professor with time to devote to it, some issues can be addressed.  Some packages have been abandoned, like many WordPress Plugins.
5. It makes psychometric forensics accessible to more organizations and psychometricians.

Disadvantages of CopyDetect

Here are some of the issues I have found.  Many thanks to a colleague (another psychometrician, who frequently speaks on forensics at conferences) who performed an independent evaluation and shared similar points.

1. It primarily uses dichotomous 0/1 data.  There is a “CopyDetect1” portion of the package that assumes multiple-choice items have been collapsed to 0/1.  However, most of the best collusion indices specifically focus on the probability of two examinees selecting the same incorrect distractor, and this info is ignored in dichotomous data because all incorrect answers have been collapsed to 0.  There is a “CopyDetect2” portion of the package, but neither I nor a colleague at another organization (who is an internationally recognized speaker on psychometric forensics) were able to even get it to run.
2. CopyDetect counts an examinee against itself in the baseline calculations, which of course is a minor effect in a large sample, but still an obvious error because an examinee can’t copy off themselves.
3. CopyDetect doubles all frequencies by looking at the entire matrix rather than a diagonal; if Examinees 1 and 2 had the same response on 10 items, it also adds in that “2 vs 1” had 10 items with the same response, stating the total number of same responses is 20 when it really is 10.
4. The documentation is lacking, though this is actually normal for R packages.  The vast majority of the useful help/examples for R are on websites like StackOverflow or Quora.  Very rarely have I found anything of use in the official documentation for any R package.  The example does not talk about how to run real data and evaluate results for cheating – the entire purpose of the software – and instead provides instructions on simulation studies to check the power/error of indices on fake data.
5. CopyDetect does not produce usable output.  Results are stored as an object, which is a sort of min-database inside R.  You then have to write more code to find, format, and print the results you might need.
6. CopyDetect does not recognize on its own that you might have more than 2 examinees, which of course is what always happens.  You have to write more code to perform all pairwise comparisons.
7. It can be incredibly slow.  This is also normal for a lot of R since 1) many packages are low quality and 2) R itself is not a compiled language (though many packages contain compiled code, and some packages are indeed incredibly fast at what they do).
8. It does not do roll-up analysis at the group level, which is incredibly important for psychometric forensics.

Summary

So what does this mean?  Well, I spent one evening looking into using CopyDetect for some of my consulting work, and immediately recognized the above issues.  That was enough to drop it and move on.

This isn’t the only experience I’ve had like this with R, either.  As noted in a scientific research report from the University of Minnesota, some of the item response theory packages in R aren’t the best.

Update: The author has released a new version (currently dated 2018.10.08), and some of these issues might now be addressed.

Nathan Thompson, PhD

Nathan Thompson, PhD, is CEO and Co-Founder of Assessment Systems Corporation (ASC). He is a psychometrician, software developer, author, and researcher, and evangelist for AI and automation. His mission is to elevate the profession of psychometrics by using software to automate psychometric work like item review, job analysis, and Angoff studies, so we can focus on more innovative work. His core goal is to improve assessment throughout the world.

Nate was originally trained as a psychometrician, with an honors degree at Luther College with a triple major of Math/Psych/Latin, and then a PhD in Psychometrics at the University of Minnesota. He then worked multiple roles in the testing industry, including item writer, test development manager, essay test marker, consulting psychometrician, software developer, project manager, and business leader. He is also cofounder and Membership Director at the International Association for Computerized Adaptive Testing (iacat.org). He’s published 100+ papers and presentations, but his favorite remains https://scholarworks.umass.edu/pare/vol16/iss1/1/.