A core of any decent assessment program are strong item banks. Item banks are a central repository of test questions, each stored with important metadata such as Author or Difficulty. They are designed to treat items are re-usable objects, which makes it easier to publish new exam forms. Of course, the storage of metadata is very useful as well, and provides validity documentation evidence. Perhaps most importantly, a true item banking system will make the process of developing new items more efficient (lower cost) and effective (higher quality).
1. Item writers are screened for expertise
Make sure the item writers (authors) that are recruited for the program will meet minimum levels of expertise. Often, this involves number of years of experience in the field. You also might want to ensure their demographics are sufficiently distributed, such as specialty area or geographic region.
2. Item writers are trained on best practices
Item writers must be trained on best practices in item writing, as well as any guidelines provided by the organization. A great example is this book from TIMSS. ASC has provided their guidelines for download here. This facilitates higher quality item banks.
3. Items go through review workflow to check best practices
After items are written, they should proceed through a standardized workflow and quality assurance. This is best practice in the development of any products. The field of software development uses a concept called the Kanban Board, which ASC has implemented in its item banking platform. Review steps can include: psychometrician, bias, language editing, and of course content.
4. Items are all linked to blueprint/standards
All items in the item banks should be appropriately categorized. This ensures that no items are measuring an unknown or unneeded concept. Items should be written to meet blueprints or standards.
5. Item banks are (preferably) piloted
Items are all written with good intent. However, we all know that some items are better than others. Items need to be given to some actual examinees, so we can obtain feedback, and also to obtain data for psychometric analysis. In many cases, they are piloted as unscored items before eventual use as “live” scored items. But this isn’t always possible.
6. Items are psychometrically analyzed
After items are piloted, you need to analyze them with classical test theory and/or item response theory to evaluate their performance. I like to say there are three possible choices after this evaluation: Retain, Revise, and Retire. Items that perform well are retained as-is. Those of moderate quality might be modified and re-piloted. Those that are unsalvageable are slated for early retirement.
How to accomplish all this?
This process can be extremely long, involved, and expensive. Many organizations hire in-house test development managers or psychometricians; those without that option will hire organizations such as ASC to serve as consultants. Regardless, it is important to have a software platform in place that can effectively manage this process. Such platform have been around since the 1980s, but many organizations still struggle by managing their item banks with Word, Powerpoint, and Email!
ASC provides an item banking platform for free, which is utilized by hundreds of organizations. Click below to sign up for your own account.
Latest posts by nthompson (see all)
- What is a question bank? - December 9, 2020
- Psychometrician and Psychometrist: What’s the difference? - November 29, 2020
- ASC to speak at ATP’s first EdTech and Computational Psychometrics Summit - November 17, 2020