Tag Archive for: adaptive testing

Do you conduct adaptive testing research? Perhaps a thesis or dissertation? Or maybe you have developed adaptive tests and have a technical report or validity study? I encourage you to check out the Journal of Computerized Adaptive Testing as a publication outlet for your adaptive testing research. JCAT is the official journal of the International Association for Computerized Adaptive Testing, a nonprofit organization dedicated to improving the science of assessments.

JCAT has an absolutely stellar board of editors and was founded to focus on improving the dissemination of research in adaptive testing. The IACAT website also contains a comprehensive bibliography of research in adaptive testing, across all journals and tech reports, for the past 50 years.  IACAT was founded at the 2009 conference on computerized adaptive testing, and has since held conferences every other year as well as hosting the JCAT journal.

Here are some of the potential research topics:

  • Item selection algorithms
  • Item exposure algorithms
  • Termination criteria
  • Cognitive diagnostic models
  • Simulation studies
  • Validation studies
  • Item response theory models
  • Multistage testing
  • Use of adaptive testing in new(er) situations, like patient reported outcomes
  • Design of actual adaptive assessments and their release into the wild

If you are not involved in CAT research but are interested, please visit the journal website to read the articles.  Access is free.  JCAT would also appreciate it if you would share this information to colleagues so that they might consider publication.

Estimated reading time: 6 minutes

Item banking refers to the purposeful creation of a database of assessment items to serve as a central repository of all test content, improving efficiency and quality. The term item refers to what many call questions; though their content need not be restricted as such and can include problems to solve or situations to evaluate in addition to straightforward questions. As a critical foundation to the test development cycle, item banking is the foundation for the development of valid, reliable content and defensible test forms.

Automated item banking systems, such as Assess.ai or FastTest, result in significantly reduced administrative time for developing/reviewing items and assembling/publishing tests.  Contact us to request a free account.

What is Item Banking?

While there are no absolute standards in creating and managing item banks, best practice guidelines are emerging. Here are the essentials your should be looking for:

   Items are reusable objects; when selecting an item banking platform it is important to ensure that items can be used more than once; ideally, item performance should be tracked not only within a test form but across test forms as well.

   Item history and usage are tracked; the usage of a given item, whether it is actively on a test form or dormant waiting to be assigned, should be easily accessible for test developers to assess, as the over-exposure of items can reduce the validity of a test form. As you deliver your items, their content is exposed to examinees. Upon exposure to many examinees, items can then be flagged for retirement or revision to reduce cheating or teaching to the test.

   Items can be sorted; as test developers select items for a test form, it is imperative that they can sort items based on their content area or other categorization methods, so as to select a sample of items that is representative of the full breadth of constructs we intend to measure.

   Item versions are tracked; as items appear on test forms, their content may be revised for clarity. Any such changes should be tracked and versions of the same item should have some link between them so that we can easily review the performance of earlier versions in conjunction with current versions.

   Review process workflow is tracked; as items are revised and versioned, it is imperative that the changes in content and the users who made these changes are tracked. In post-test assessment, there may be a need for further clarification, and the ability to pinpoint who took part in reviewing an item and expedite that process.

   Metadata is recorded; any relevant information about an item should be recorded and stored with the item. The most common applications for metadata that we see are author, source, description, content area, depth of knowledge, IRT parameters, and CTT statistics, but there are likely many data points specific to your organization that is worth storing.

Managing an Item Bank

Names are important. As you create or import your item banks it is important to identify each item with a unique, but recognizable name. Naming conventions should reflect your bank’s structure and should include numbers with leading zeros to support true numerical sorting.  You might want to also add additional pieces of information.  If importing, the system should be smart enough to recognize duplicates.

Search and filter. The system should also have a reliable sorting mechanism. 

Assess' item banking output

Prepare for the Future: Store Extensive Metadata

Metadata is valuable. As you create items, take the time to record simple metadata like author and source. Having this information can prove very useful once the original item writer has moved to another department, or left the organization. Later in your test development life cycle, as you deliver items, you have the ability to aggregate and record item statistics. Values like discrimination and difficulty are fundamental to creating better tests, driving reliability, and validity.

Statistics are used in the assembly of test forms while classical statistics can be used to estimate mean, standard deviation, reliability, standard error, and pass rate.Item response theory parameters can come in handy when calsulating test information and standard error functions. Data from both psychometric theories can be used to pre-equate multiple forms.

In the event that your organization decides to publish an adaptive test, utilizing CAT delivery, item parameters for each item will be essential. This is because they are used for inteligent selection of items and scoring examinees. Additionally, in the event that the integrity of your test or scoring mechanism is ever challenged, documentation of validity is essential to defensibility and the storage of metadata is one such vital piece of documentation.

Increase Content Quality: Track Workflow

Utilize a review workflow to increase quality. Using a standardized review process will ensure that all items are vetted in a similar matter. Have a step in the process for grammar, spelling, and syntax review, as well as content review by a subject matter expert. As an item progresses through the workflow, its development should be tracked, as workflow results also serve as validity documentation.

Accept comments and suggestions from a variety of sources. It is not uncommon for each item reviewer to view an item through their distinctive lens. Having a diverse group of item reviewers stands to benefit your test-takers, as they are likely to be diverse as well!

item banking review kanban

Keep Your Items Organized: Categorize Them

Identify items by content area. Creating a content hierarchy can also help you to organize your item bank and ensure that your test covers the relevant topics. Most often, we see content areas defined first by an analysis of the construct(s) being tested. In the event of a high school science test, this may include the evaluation of the content taught in class. A high-stakes certification exam, almost always includes a job-task analysis. Both methods produce what is called a test blueprint, indicating how important various content areas are to the demonstration of knowledge in the areas being assessed.

Once content areas are defined, we can assign items to levels or categories based on their content. As you are developing your test, and invariably referring back to your test blueprint, you can use this categorization to determine which items from each content area to select.

Why Item Banking?

There is no doubt that item banking is a key aspect of developing and maintaining quality assessments. Utilizing best practices, and caring for your items throughout the test development life cycle, will pay great dividends as it increases the reliability, validity, and defensibility of your assessment. Moreover, good item banking will make the job easier and more efficient thus reducing the cost of item development and test publishing.

Ready to improve assesment quality through item banking?

Click below to visit our Contact page, where you can request a demonstration or a free account (up to 500 items).

Adaptive testing in medicine is one of the newer applications of this modern psychometric paradigm.  A recent article in the The Journal of Bone & Joint Surgery, Volume 95, Issue 20 evaluates the use of computerized adaptive testing (CAT) approaches in assessment of psychological factors surrounding perceived disability and pain.  Researchers from the Partners health system report favorable findings, namely that the CAT assessments provided in the Patient Reported Outcomes Measurement Information System (PROMIS) correlate highly with older paper-based assessments that will typically take much longer to administer.

Such findings are typical in this sort of research.  For example, the MHCAT project, in which Assessment Systems was involved, published this article which notes that test length was reduced by 95% – yes, 95%! – while maintaining a correlation of the short CAT assessment with the original long assessment of 0.93.  Why would a practitioner keep using an outdated assessment that takes 20 times as long to complete?

The Bone & Joint article was also discussed in the Helio website on medical news.

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source

 Are states prepared?

Computerized Adaptive Testing (CAT) has been in the news for quite some time as a technology being adopted at the state, and district level to improve the academic assessment performance for K-12 students. Legislation such as NCLB, and a push for a common-core standard have put pressure on many states to find solutions to improve performance and accountability in the classroom. For many state agencies the solution is CAT testing. But are states prepared for this innovation?

In mid-November Governor Malloy of Connecticut appropriated $14 million dollars for the use of CAT testing in Connecticut K-12 schools. This funding will be used to administer CAT end of year tests in grades 3 through 8 and 11. The CAT tests will be aligned to Smarter Balanced Assessments standards used by several other state education agencies across the United States.

Although Connecticut’s commitment to CAT testing will help improve the education performance of Connecticut’s students, many are still concerned about the state’s shortfall in the amount of money being appropriated with computer and bandwidth upgrades. These upgrades are paramount for a successful adoption of CAT testing.  According to WTNH Hartford, the state was only able to distribute $10 million out of the $24 million appropriated for technology upgrades in schools. This poses a central issue, will CAT be implemented successfully?

CAT testing is the innovative technology our country needs to successfully improve our K-12 education. Connecticut’s plan for CAT testing is a great example of what many states will have to deal with very shortly if funding is appropriated for CAT testing.  For many states, significant state-wide upgrades on technology are inevitable for CAT testing to be successfully implemented.

It has already been apparent in many states currently using CAT testing that they are seeing improved performance. What is still not apparent is how CAT testing will be implemented successfully in states where significant technology upgrades are still needed.

Link to WTNH article:

http://www.wtnh.com/news/politics/malloy-14m-in-grant-money-for-schools

The Huffington Post recently published an article discussing the role that computerized adaptive testing (CAT) plays in K-12 educational assessment.  It presents some of the issues in the field of standardized testing, and then discusses how adaptive testing in education can address some of those issues.

This article is a refreshing departure from those typically seen in the mainstream media, which often complain about student assessment without taking the time to become informed on the topic, most notably understanding all the benefits that it provides and advantages over alternatives.  In fact, the article specifically notes that much of the debate going on just completely misses the point.  I recently read an article in a parenting magazine that was on the other end of the spectrum, bordering on fear-mongering sensationalism.

Some of the advantages of adaptive testing in education discussed:

  • CAT helps refocus learning on instruction
  • CAT can reduce the “corruptive pressures” and teaching to the test
  • CAT can enhance security: there are no bubble sheets, which completely eliminates the possibility of teacher modification, and can provide accommodations automatically, greatly reducing the chance of teachers cheating during the accommodation process (e.g., reading the test questions to a student)
  • CAT provides more accurate scores for high-ability and low ability students, and for disabled students
  • CAT tests can cost less and typically take much less time to administer (research suggests 50% less time testing)

Of course, CAT is not a panacea to all the problems facing student assessment, much less our educational system.  However, it has been shown to have numerous benefits, supported by a large body of scientific literature.  The SMARTER Balanced Assessment Consortium has the right idea in utilizing CAT technology to meet the assessment needs of its member states.  I am eagerly looking forward to seeing how those endeavors turn out.

Here is the full article URL: http://www.huffingtonpost.com/darrell-west/how-technology-can-stop-c_b_3784392.html

Interested in adaptive testing in education?

Sign up for our newsletter on the left, or contact us to talk with one of our assessment consultants.

Update on IACAT 2012 Conference in Sydney, Australia
The conference website has been updated with accepted presentations – go to the “Program” tab in the website (www.iacat.org) to view. Presenters from the United States, Netherlands, Saudi Arabia, South Africa, United Kingdom, Canada, China, Australia, Japan, India, Thailand, Singapore, Hong Kong, etc. who will discuss the latest research, theory and applications of CAT.  Register now (note that registrations include lunch and coffee/tea) and join us in Sydney, Australia in August!

Although deadline for proposals has passed, there is room for three more presentations. Please submit an abstract via the website.  Note that grants are available to graduate students, supporting both research and travel; please visit https://assess.com/xcart/pages.php?pageid=9 for more information.

FastTest Web updates
Every month sees advanced new functionality added to FastTest Web (www.fasttestweb.com), our industry-leading platform for item authoring, secure test delivery, and management of testing programs.  Recent additions include an asset manager module for multimedia, split-screen testlets (example: reading passage on left, questions on the right), and the ability to divide your test into multiple sections with separate time limits and other options.  Upcoming features include an interactive on-screen ruler (allowing students to measure objects) and an enhanced psychometric analysis report to evaluate essential aspects of the test (including score reliability, item difficulty, item discrimination, and distractor analysis).

ASC presents at 2012 SIOP Conference
ASC presented a mini-workshop on the development of computerized adaptive tests (CATs) at the Society for Industrial-Organizational Psychology (SIOP) conference.  We will also be presenting at the IACAT 2012 Conference in Sydney, Australia.

Connect with us on LinkedIn
Want to stay on top of advancements in psychometrics, test development, and leading-edge software?  Follow us on the ASC LinkedIn page.

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source