In the past decade, terms like machine learning, artificial intelligence, and data mining are becoming greater buzzwords as computing power, APIs, and the massively increased availability of data enable new technologies like self-driving cars. However, we’ve been using methodologies like machine learning in psychometrics for decades. So much of the hype is just hype. So, what […]
An emerging sector in the field of psychometrics is the area devoted to analyzing test data to find cheaters and other illicit or invalid testing behavior. We lack a generally agreed-upon and marketable term for that sort of work, and I’d like to suggest that we use Psychometric Forensics. While research on this topic is more […]
Last week, I had the opportunity to attend the 2017 Conference on Test Security (COTS), hosted by the University of Wisconsin-Madison. If your organization has any concerns about test security (that is, you have any sort of real stakes tied to your test!), I recommend that you attend COTS. It has a great mix of […]
Cognitive diagnostic models are an area of psychometric research that has seen substantial growth in the past decade, though the mathematics behind them, dating back to MacReady and Dayton (1977). The reason that they have been receiving more attention is that in many assessment situations, a simple overall score does not serve our purposes and […]
Guttman errors are a concept derived from the Guttman Scaling approach to evaluating assessments. There are a number of ways that they can be used. Meijer (1994) suggests an evaluation of Guttman errors as a way to flag aberrant response data, such as cheating or low motivation. He quantified this with two different indices, G […]
What is a rubric? It’s a rule for converting unstructured responses on an assessment into structured data that we can use psychometrically. Why do we need rubrics? Measurement is a quantitative endeavor. In psychometrics, we are trying to measure things like knowledge, achievement, aptitude, or skills. So we need a way to convert qualitative data […]
Test security is an increasingly important topic. There are several causes, including globalization, technological enhancements, and the move to a gig-based economy driven by credentials. Any organization that sponsors assessments that have any stakes tied to them must be concerned with security, as the greater the stakes, the greater the incentive to cheat. And threats […]
Today I read an article in The Industrial-Organizational Psychologist (the colloquial journal published by the Society for Industrial Organizational Psychology) that really resonated with me. Has Industrial-Organizational Psychology Lost Its Way? -Deniz S. Ones, Robert B. Kaiser, Tomas Chamorro-Premuzic, Cicek Svensson Why? Because I think a lot of the points they are making are also true […]
A modified-Angoff method study is one of the most common ways to set a defensible cutscore on an exam. It therefore means that the pass/fail decisions made by the test are more trustworthy than if you picked a random number; if your doctor, lawyer, accountant, or other professional has passed an exam where the cutscore […]
I often hear this question about scaling, especially regarding the scaled scoring functionality found in software like FastTest and Xcalibre. The following is adapted from lecture notes I wrote while teaching a course in Measurement and Assessment at the University of Cincinnati. Test Scaling: Sort of a Tale of Two Cities Scaling at the test […]
