Make sure your exams are reliable and valid with psychometric analysis & reporting
How do you know if your test is high quality, or how to improve it? Our psychometric experts will analyze your data with classical test theory or item response theory to ensure that your tests meet professional standards, and you can flag poor items for improvement.
What is Psychometric Analysis?
Psychometric analysis refers to the use of data science to evaluate the results of a test, with the goal of supporting validity. There are multiple levels to be analyzed.
- Items – are any questions too hard, too easy, too confusing?
- Domains or Content Areas – Are certain parts of the test unreliable, or not correlating with the rest?
- Test – How does the overall reliability of the test measure up? How about the standard error of measurement?
- Cutscore – Is the cutscore set with an approved method like Angoff? What is the decision consistency?
- Drift – Are item statistics changing over time?
- Equating – How do you know that a score of X means the same thing it did 8 years ago?
Our psychometricians can help you answer these tough questions, and explain them in plain language.
Expert psychometricians to guide you
Our PhD psychometricians excel not only in utilizing best practices and leading-edge psychometric software to perform these complex quantitative analyses but providing the results with understandable explanations that allow you and your subject matter experts to truly increase the quality of your test by modifying the content appropriately.
From simple reports of classical test theory to complex equating & scaling studies with item response theory (IRT), our psychometricians will help you lay the quantitative foundation for accurate measurement. We can also help you achieve accreditation and perform exam design from scratch, including job task analysis and modified-Angoff standard setting studies.
We’re here for your questions
Our expert psychometricians will work with you to explain the complexities each step of the way.
Why do I need psychometric analysis of my assessments?
An absolutely essential step in developing a defensible test is statistical analysis of response data. This step provides detailed feedback on what is working well in the test, what is not, and why. This drives continual improvement, but also provides documentation of validity evidence.
What sort of work is involved in this process?
Our psychometricians will need response data from examinees as well as metadata on each item. Once you provide these, we clean the data then run it through our specially designed software. This software drafts the report for you in seconds, which means we can provide this service at a fraction of the cost that large vendors will charge you. Though of course, we first go through the results with a fine-toothed comb and write up our recommendations.
What sort of result do I get from this process?
Well, suppose that we analyze a test question whose correct answer is A, but find that only 40% of your candidates got it correct (item difficulty). Meanwhile, just as many examinees answered B – and it happened to be smarter examinees. This item would be flagged and discussed with your experts, who might point out something like “Only candidates who have completed a special training course would know how to answer this question.” Even though the answer is technically correct, the statistics have helped flag an issue with the item, and it might be retired or rewritten.
I know this stuff! Can we just do it ourselves?
Have psychometric talent in-house but want to make them more effective in conducting these analyses? Iteman and Xcalibre provide user-friendly interfaces and automatically create Microsoft Word reports with all the tables and graphs already embedded for you. Or, start using our best-in-class platform Assess.ai and have the reports ready-made for you!
Ready to start? Contact us to request a demo with one of our assessment experts.
Request a Consultation