We get asked all the time at Assessment Systems Corporation: What are the steps involved in certification test development? What are the best practices that lead to a sound exam that can meet accreditation standards? There are a number of key steps:
A best practice among certification programs is to understand the knowledge, skills, and attributes used in the performance of the job or role. A job task analysis, or role delineation, uses quantitative methods to gain insight into job competence. Subject matter expert (SME) involvement is critical to validity. The value of this analysis ranges from helping practitioners identify areas for development to benchmarking content for educational and training programs, field leadership for the certification organization, and promotion and elevation of the field.
Developing a blueprint, or content outline, for the test from job analysis results lends validity to test results. A best practice is to use the job analysis to derive which content areas should be on the test and how many items should be in each. Beyond test validity, the benefits of publishing the blueprint include transparency for all in the certification process and a guide for candidates to focus their test preparation.
A foundational principle of test development says that the effectiveness of the test is only as good as the items on it. Savvy certification programs have quality standards for their items covering type, content, format, and grammar. They ensure that each item meets these before inclusion on a test. Once items have been sampled, program managers consider whether their performance warrants use in future testing.
The passing standard, or cut score, is usually where the certification program’s test intersects with their decision whether to confer certification. Test takers whose scores meet or exceed the passing standard receive a pass result, and those whose performance fails to meet the standard receive a fail. The stakes mentioned above rest on where and in what way this this standard is set. Common methods for setting the passing standard involve SME input and previous item performance. Publishing the passing standard and method used to derive it is valuable to candidates for understanding how they performed on the test and to certifying organizations for deciding where to grant certification.
Validity and reliability of the test are key concepts in psychological testing. Validity speaks to how accurately the test measures real-world job or role performance. Savvy programs document evidence of this in the points already discussed. Once candidates have answered items, analysis of test and item performance begins.
Traditionally, programs look at statistics such as reliability (especially at the passing standard), distribution of scores, difficulty, and discrimination of the test and each item for a particular group of test takers. Certifying organizations use this analysis to finalize test results and report on the test’s performance. Leading organizations with high candidate volumes are doing additional analysis of results and developing innovative ways to compile and administer tests, such as Linear on-the-fly (LOFT) testing and computer adaptive testing (CAT).
The Certification Test Development Cycle Continues…
Certification test development is not a one-and-done situation. If you leave one form of the test out in the field, the questions and answers will soon become common knowledge. It’s essential to periodically update the exam by developing new items and then rotating them in by publishing new forms. And every 5-10 years (depending on the profession) the test needs a complete overhaul, with a new job analysis and standard setting. As with any high performing machine, you need oil changes and periodic maintenance!