Posts

Some time ago, I received this question regarding interpreting IRT cutscores (item response theory):

In my examination system, we are currently labeling ‘FAIL’ for student’s mark with below 50% and ‘PASS’ for 50% and above.  I found that this amazing Xcalibre software can classify students’ achievement in 2 groups based on scores.  But, when I tried to run IRT EPC with my data (with cut point of 0.5 selected), it shows that students with 24/40 correct items were classified as ‘FAIL’. Because in CTT, 24/40 correctly answered items is equal to 60% (Pass).  I can’t find its interpretation in Guyer & Thompson (2013) User’s Manual for Xcalibre.  How exactly should I set my cut point to perform 2-group classification using IRT EPC in Xcalibre to make it about equal to 50% achievement in CTT?

In this context, EPC refers to expected percent/proportion correct.  IRT uses the test response function (TRF) to convert a theta score to an expectation of what percent of items in the pool that a student would answer correctly.  So this Xcalibre user is wondering how to set IRT cutscores on theta that meets their needs.

Setting IRT cutscores

The short answer, in this case, would be to evaluate the TRF and reverse-calculate the theta for the cutscore.  That is, find your desired cutscore on the y-axis, and determine the corresponding value of theta.  In the example below, I have found a % cutscore of 54 and found the corresponding theta of -0.13 or so.  In the case above, a theta=0.5 likely corresponded to a percent correct score of 60%-70%, so observed scores of 24/40 would indeed fail.

test response function

Of course, it is indefensible to set a cutscore to be arbitrary round numbers.  To be defensible, you need to set the cutscore with an accepted methodology such as Angoff, modified-Angoff, Nedelsky, Bookmark, or Contrasting Groups.

A nice example is a the modified-Angoff, which is used extremely often in certification and licensure situations.  More information is available on this method here.  The result of this method will typically be a specific cutscore, either on the raw or percent metric.  The TRF can be presented in both of those metrics, allowing the conversion on the right to be calculated easily.

Alternatively, some standard-setting methods can work directly on the IRT theta scale, including the Bookmark and Contrasting Groups approaches.

Interested in applying IRT to improve your assessments?  Download a free trial copy of Xcalibre here.  If you want to deliver online tests that are scored directly with IRT, in real time (including computerized adaptive testing), check out FastTest.

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source

ASC provides a number of psychometrics tutorial videos regarding our leading-edge software, to help customers better understand the use of the software as well as advanced concepts in psychometrics.  This, in turn, helps you develop higher quality tests and assessments, enhancing reliability and validity.

We have recently released two new videos: Item Response Theory with Xcalibre 4, and an Introduction to Item Writing.

Xcalibre is landmark software that not only analyzes data with item response theory, but automatically builds tech reports in MS Word with embedded graphics, tables, and narratives.  This saves many hours of drudgery work from psychometricians, which is expensive!

Videos are available on our our YouTube Channel.

Here are some brief updates and news for April 2012: new website and updates to Xcalibre 4, CATSim, and FastTest Web!

New ASC Website

To reflect ASC’s growing stature in the testing industry, we have released a new website that better reflects our new role as a comprehensive, world-class testing organization.  In addition to massively expanding our psychometric software capabilities, we now offer full consulting services, a network of secure testing centers, and software to completely manage the entire assessment process.  Check out the new site at www.assess.com!  We have also upgraded our new webstore (shopping cart system) and added a sleek, modern layout to be more customer-friendly; visit the store at https://assess.com/xcart/home.php. In addition, we are in the process of integrating Google Wallet functionality for enhanced credit card security and convenience.

ASC presents workshop to SADC in Botswana

ASC presented a workshop on item response theory (IRT) and computerized adaptive testing (CAT) to member countries of the Southern African Development Council.  The workshop took place March 4-9 in Botswana, and provided five days of background and hands-on instruction regarding the application of these modern methods to solving issues in educational assessment, with a focus on ASC’s landmark software, Iteman 4 and Xcalibre 4.  Such a combination of ASC’s consulting services and leading-edge software can help your organization move to the next level; contact solutions@54.89.150.95 to learn more.

FastTEST Web updates

FastTest Web (FTW), our platform for item authoring and internet-based assessment, is being upgraded almost continually to stay at the forefront of testing technology.  In March, we released functionality that allows tests to be published as a public link, ideal for low stakes exams or pilot testing, as an alternative to the high-security registration process currently in place.  This approach is integrated with new functionality for managing demographic information on examinees; you can request that examinees enter such information before starting the test.  Additionally, we rebuilt the item authoring tool to be more user-friendly and WYSIWYG.  As always, we’ve also released a number of small enhancements.  The next major release will include an advanced Asset Manager for multimedia and the capability for testlets, groups of items that share a reading passage or other common stimulus.

Xcalibre 4 updated

ASC’s cutting-edge software for item response theory has been substantially updated.  This is a free update for current license holders, and includes several bug fixes as well as improved functionality:

  • User can specify bounds for scaled scores
  • Lz person fit statistic now computed
  • Test response function (TRF) values are now output
  • Chi-square item fit statistics enhanced
  • Subgroup analysis based on DIF group membership
  • Rebuilt ranking of ability estimates to be much faster
  • Observed item response function (fit line) now includes multi-key items
  • Many more small improvements!

Current users can upgrade at our support page; new users can download a free version here.  Don’t forget that graduate students can obtain free licenses from the APM student grants program!

CATSim updated

CATSim, the software that provides the research necessary for computerized adaptive testing (CAT) has also been upgraded.  The update focuses primarily on bug fixes.  Current users can upgrade for free at our support page; new users can download a free version here.

ASC to present at 2012 SIOP Conference

ASC will present a mini-workshop on the development of computerized adaptive tests (CATs) at the Society for Industrial-Organizational Psychology (SIOP) conference.  The session is scheduled for 1:30 P.M. on Friday, April 27.  More information is available at the SIOP website.  Please join us and learn about our CAT solutions and assessments!

Xcalibre 4 updates
Xcalibre 4is the most user-friendly software available for item response theory (IRT) analysis.  An update has been recently released, which includes a number of bug fixes and enhancements, including an addition of distractor (quantile) plots into the output.  These plots provide an excellent method for evaluating multiple-choice distractors by combining IRT and classical test theory.  We have also added a new fit plot with standard errors to the report document and percentiles to the scores spreadsheet.  Current license holders can download the no-cost update from https://assess.com/xcart/pages.php?pageid=10.  If you do not yet have Xcalibre 4, a free demo version is available at https://assess.com/xcart/product.php?productid=569.

Iteman 4 updates
We have also released an update for Iteman 4, the leading software for analysis with classical test theory (CTT).  This update focuses on bug fixes and maintenance.  If you have a current license, you can download the update at https://assess.com/xcart/pages.php?pageid=10.  If you do not yet have Iteman 4, a free demo version is available at https://assess.com/xcart/product.php?productid=541.

Conferences page on ASC website updated for 2012
ASC provides a number of resources to professionals in the field of testing and psychometrics, including a list of major conferences in the field.  We have updated that list for 2012 at https://assess.com/conferences.php… see where you want to go!  ASC has plans to be at the SIOP, IACAT, and ICE conferences.

Connect with ASC on LinkedIn
LinkedIn is an essential social networking site for professionals of all fields.  It marks an important advancement in professional connectivity.  ASC has had a page on LinkedIn for a number of years; if you aren’t already following us, please visit http://www.linkedin.com/company/assessment-systems-corporation. We also update our news at the ASC blog.

Free training videos released
ASC hosts one or two workshops every year (one workshop was just recently held, January 23-25 in Brazil) and presents at a number of conferences, but we realize that not everyone is able to attend.  Therefore, ASC is publishing a set of tutorial videos on our software and important issues in psychometrics, like item response theory and computerized adaptive testing.  The videos are available for free at https://assess.com/tutorials.php.  Two have been completed so far: Running Iteman 4 and Interpreting Iteman 4 Output.

ASC to present at 2012 SIOP Conference
ASC will present a mini-workshop on the development of computerized adaptive tests (CATs) at the Society for Industrial-Organizational Psychology (SIOP) conference.  The session is scheduled for 1:30 PM on Friday, April 27.  More information is available at the SIOP website.