Item-review-kanban

A core of any decent assessment program are strong item banks. Item banks are a central repository of test questions, each stored with important metadata such as Author or Difficulty. They are designed to treat items are re-usable objects, which makes it easier to publish new exam forms. Of course, the storage of metadata is very useful as well, and provides validity documentation evidence. Perhaps most importantly, a true item banking system will make the process of developing new items more efficient (lower cost) and effective (higher quality).

1. Item writers are screened for expertise

Make sure the item writers (authors) that are recruited for the program will meet minimum levels of expertise. Often, this involves number of years of experience in the field. You also might want to ensure their demographics are sufficiently distributed, such as specialty area or geographic region.

2. Item writers are trained on best practices

Item writers must be trained on best practices in item writing, as well as any guidelines provided by the organization. A great example is this book from TIMSS. ASC has provided their guidelines for download here. This facilitates higher quality item banks.

3. Items go through review workflow to check best practices

After items are written, they should proceed through a standardized workflow and quality assurance. This is best practice in the development of any products. The field of software development uses a concept called the Kanban Board, which ASC has implemented in its item banking platform. Review steps can include: psychometrician, bias, language editing, and of course content.

4. Items are all linked to blueprint/standards

All items in the item banks should be appropriately categorized. This ensures that no items are measuring an unknown or unneeded concept. Items should be written to meet blueprints or standards.

5. Item banks are (preferably) piloted

Items are all written with good intent. However, we all know that some items are better than others. Items need to be given to some actual examinees, so we can obtain feedback, and also to obtain data for psychometric analysis. In many cases, they are piloted as unscored items before eventual use as “live” scored items. But this isn’t always possible.

6. Items are psychometrically analyzed

After items are piloted, you need to analyze them with classical test theory and/or item response theory to evaluate their performance. I like to say there are three possible choices after this evaluation: Retain, Revise, and Retire. Items that perform well are retained as-is. Those of moderate quality might be modified and re-piloted. Those that are unsalvageable are slated for early retirement.

How to accomplish all this?

This process can be extremely long, involved, and expensive. Many organizations hire in-house test development managers or psychometricians; those without that option will hire organizations such as ASC to serve as consultants. Regardless, it is important to have a software platform in place that can effectively manage this process. Such platform have been around since the 1980s, but many organizations still struggle by managing their item banks with Word, Powerpoint, and Email!

ASC provides an item banking platform for free, which is utilized by hundreds of organizations. Click below to sign up for your own account.

The following two tabs change content below.

nthompson

Nathan Thompson earned his PhD in Psychometrics from the University of Minnesota, with a focus on computerized adaptive testing. His undergraduate degree was from Luther College with a triple major of Mathematics, Psychology, and Latin. He is primarily interested in the use of AI and software automation to augment and replace the work done by psychometricians, which has provided extensive experience in software design and programming. Dr. Thompson has published over 100 journal articles and conference presentations, but his favorite remains https://pareonline.net/getvn.asp?v=16&n=1.