Incremental Validity
Incremental validity is an aspect of validity that refers to what an additional assessment or predictive variable can add to the information provided by existing assessments or variables. It refers to the amount of “bonus” predictive power by adding in another predictor. In many cases, it is on the same or similar trait, but often the most incremental validity comes from using a predictor/trait that is relatively unrelated to the original. See examples below.
Note that this is often discussed with respect to tests and assessment, but in many cases a predictor is not a test or assessment, as you will also see.
How is Incremental Validity Evaluated?
It is most often quantified with a linear regression model and correlations. However, any predictive modeling approach could work from support vector machines to neural networks.
Example of Incremental Validity: University Admissions
One of the most commonly used predictors for university admissions is an admissions test, or battery of tests. You might be required to take an assessment which includes an English/Verbal test, a Logic/Reasoning test, and a Quantitative/Math test. These might be used individually or aggregate to create a mathematical model, based on past data, that predicts your performance at university. (there are actually several variables for this, such as first year GPA, final GPA, and 4 year graduation rate, but that’s beyond the scope of this article)
Of course, the admissions exams scores are not the only point of information that the university has on students. It also has their high school GPA, perhaps an admissions essay which is graded by instructors, and so on. Incremental validity poses this question: if the admissions exam correlates 0.59 with first year GPA, what happens if we make it into a multiple regression/correlation with High School GPA (HGPA) as a second predictor? It might go up to, say, 0.64. There is an increment of 0.05. If the university has that data from students, they would be wasting it by not using it.
Of course, HGPA will correlate very highly with the admissions exam scores. So it will likely not add a lot of incremental validity. Perhaps the school finds that essays add a 0.09 increment to the predictive power, because it is more orthogonal to the admissions exam scores. Does it make sense to add that, given the additional expense of scoring thousands of essays? That’s a business decision for them.
Example of Incremental Validity: Pre-Employment Testing
Another common use case is that of pre-employment testing, where the purpose of the test is to predict criterion variables like job performance, tenure, 6-month termination rate, or counterproductive work behavior. You might start with a skills test; perhaps you are hiring accountants or bookkeepers and you give them a test on MS Excel. What additional predictive power would we get by also doing a quantitative reasoning test? Probably some, but that most likely correlates highly with MS Excel knowledge. So what about using a personality assessment like Conscientiousness? That would be more orthogonal. It’s up to the researcher to determine what the best predictors are. This topic, personnel selection, is one of the primary areas of Industrial/ Organizational Psychology.
Nathan Thompson, PhD
Latest posts by Nathan Thompson, PhD (see all)
- What is a T score? - April 15, 2024
- Item Review Workflow for Exam Development - April 8, 2024
- Likert Scale Items - February 9, 2024
