Posts

Background of the Certification/Recertification Report

The National Commission for Certifying Agencies (NCCA) is a group that accredits certification programs.  Basically, many of the professional associations that had certifications (e.g., Certified Professional Widgetmaker) banded together to form a super-association, which then established an accreditation arm to ensure that certifications were of high quality – as there are many professional certifications in the world of near-zero quality.  So this is a good thing.  Becoming accredited is a rigorous process, and the story doesn’t stop there.  Once you are accredited, you need to submit annual reports to NCCA as well as occasionally re-apply.  This is mentioned in NCCA Standard 24, and is also described in the NCCA webpage regarding annual renewals of accreditation. It requires information on certification and recertification volumes.

The certification program must demonstrate continued compliance to maintain accreditation.

Essential Elements:

  1. The certification program must annually complete and submit information requested of the certification agency and its programs for the previous reporting year.

There are a number of reports that are required, one of which is a summary of Certification/Recertification numbers.  These are currently submitted through an online system, but an example of a previous paper form can be found here, which clearly states some of the tables that you need to fill out.

In the past, you had to pay consultants or staff to manually compile this information.  Our certification management system does it automatically for you – for free.

Overview of the Cert/Recert Report

Our easily generated Certification/Recertification report provides a simple, clean overview of your certification program. The report includes important information required for maintaining accreditation, including the number of applicants, new certifications, and recertifications, as well as percentages signifying the success rate of new candidates and recertification candidates. The report is automatically produced within FastTest’s reporting module, saving your organization thousands of dollars in consultant fees.

Here is a sample report. Assume that this organization has one base level certification, Generalist, with 3 additional specialist areas where certification can also be earned.

Annual Certification/Recertification Report

Date run: 11/10/2016
Timeframe: 1/1/2012 – 12/31/2015

Program Applicants for first time certification First time certified Due for recertification Recertified Percent due that recertified
Generalist 4,562 2,899 653 287 44%
Specialist A 253 122 72 29 40%
Specialist B 114 67 24 7 36%
Specialist C 44 13 2 0 0%

Let’s examine the data for the Generalist program. Follow the table across the first data line:

Generalist is the name of the program with the data being analyzed. The following data all refers to candidates who were involved in the certification process at the Generalist level within our organization.

4,562 is the number of candidates who registered to be certified for the program within the timeframe indicated above the table. These candidates have never been certified in this program before.

2,899 is the number of candidates who successfully completed the certification process by receiving a passing score and meeting any other minimum requirements set forth by the certification program.

653 is the number of previously certified candidates whose certification expired within the timeframe indicated above the table.

287 is the number of previously certified candidates who successfully completed the recertification process.

44% is the percentage of candidates eligible for recertification within the indicated timeframe who successfully completed the recertification process. Another way to express this value would be 287/653; the number of who successfully completed recertification divided by the number of those due for recertification within the given timeframe.

The same format follows for each of the Specialist programs.

If you found this post interesting, you might also be interested in checking out this post on the NCCA Annual Statistics Report.  That report is another one of the requirements, but focuses on statistical and psychometric characteristics of your exams.

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source

Here are the September 2012 updates and news…

 

Xcalibre 4.1.8 update provides new functionality

A free update to Xcalibre 4.1.8 has been released.  Current users can download the update at our support page, https://assess.com/xcart/pages.php?pageid=8. If you do not yet use Xcalibre to produce your IRT reports, you can download a copy at https://assess.com/xcart/product.php?productid=415. The following improvements have been made:

  • Polytomous categories with zero endorsement can now automatically be collapsed
  • Data Matrix and Item Control Files can now include a header line of identification (helps you keep track of which column is which!)
  • Not-administered and/or omitted responses can be replaced with monte-carlo generated model-fitting data in the scored matrix file – useful for sparse data sets
  • The scored data matrix file is now saved in the same format as the input data matrix
  • Delimited scored matrix files now automatically include header line with same item names used in Item Control File
  • The level of precision in the output is now controllable
  • Revised manual

 

ASC now provides complete one-stop-shop for certification organizations

ASC provides FastTest Web (www.fasttestweb.com), a best-in-class software for item authoring, test development, and test delivery.  We are currently integrating a module for examinee registration, payment processing, and scheduling for our network of testing centers.  In combination with extensive consulting services from our PhD experts in psychometrics and accreditation, we now provide a comprehensive solution for certification organizations large and small.  If your organization delivers professional certifications, we encourage you to contact sales@fasttestweb.com!

 

New credential released: Certified Social Media Strategist

ASC has worked with the National Institute for Social Media (www.nismonline.org)  to develop a landmark credential, the Certified Social Media Strategist.  This is the first certification designed specifically for the explosively growing field of social media.  More information is available at nismonline.org.   If your organization is involved in any social media marketing, please forward this message to relevant personnel!

 

These are the September 2012 updates!

Update on IACAT 2012 Conference in Sydney, Australia
The conference website has been updated with accepted presentations – go to the “Program” tab in the website (www.iacat.org) to view. Presenters from the United States, Netherlands, Saudi Arabia, South Africa, United Kingdom, Canada, China, Australia, Japan, India, Thailand, Singapore, Hong Kong, etc. who will discuss the latest research, theory and applications of CAT.  Register now (note that registrations include lunch and coffee/tea) and join us in Sydney, Australia in August!

Although deadline for proposals has passed, there is room for three more presentations. Please submit an abstract via the website.  Note that grants are available to graduate students, supporting both research and travel; please visit https://assess.com/xcart/pages.php?pageid=9 for more information.

FastTest Web updates
Every month sees advanced new functionality added to FastTest Web (www.fasttestweb.com), our industry-leading platform for item authoring, secure test delivery, and management of testing programs.  Recent additions include an asset manager module for multimedia, split-screen testlets (example: reading passage on left, questions on the right), and the ability to divide your test into multiple sections with separate time limits and other options.  Upcoming features include an interactive on-screen ruler (allowing students to measure objects) and an enhanced psychometric analysis report to evaluate essential aspects of the test (including score reliability, item difficulty, item discrimination, and distractor analysis).

ASC presents at 2012 SIOP Conference
ASC presented a mini-workshop on the development of computerized adaptive tests (CATs) at the Society for Industrial-Organizational Psychology (SIOP) conference.  We will also be presenting at the IACAT 2012 Conference in Sydney, Australia.

Connect with us on LinkedIn
Want to stay on top of advancements in psychometrics, test development, and leading-edge software?  Follow us on the ASC LinkedIn page.

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source

Here are some brief updates and news for April 2012: new website and updates to Xcalibre 4, CATSim, and FastTest Web!

New ASC Website

To reflect ASC’s growing stature in the testing industry, we have released a new website that better reflects our new role as a comprehensive, world-class testing organization.  In addition to massively expanding our psychometric software capabilities, we now offer full consulting services, a network of secure testing centers, and software to completely manage the entire assessment process.  Check out the new site at www.assess.com!  We have also upgraded our new webstore (shopping cart system) and added a sleek, modern layout to be more customer-friendly; visit the store at https://assess.com/xcart/home.php. In addition, we are in the process of integrating Google Wallet functionality for enhanced credit card security and convenience.

ASC presents workshop to SADC in Botswana

ASC presented a workshop on item response theory (IRT) and computerized adaptive testing (CAT) to member countries of the Southern African Development Council.  The workshop took place March 4-9 in Botswana, and provided five days of background and hands-on instruction regarding the application of these modern methods to solving issues in educational assessment, with a focus on ASC’s landmark software, Iteman 4 and Xcalibre 4.  Such a combination of ASC’s consulting services and leading-edge software can help your organization move to the next level; contact solutions@54.89.150.95 to learn more.

FastTEST Web updates

FastTest Web (FTW), our platform for item authoring and internet-based assessment, is being upgraded almost continually to stay at the forefront of testing technology.  In March, we released functionality that allows tests to be published as a public link, ideal for low stakes exams or pilot testing, as an alternative to the high-security registration process currently in place.  This approach is integrated with new functionality for managing demographic information on examinees; you can request that examinees enter such information before starting the test.  Additionally, we rebuilt the item authoring tool to be more user-friendly and WYSIWYG.  As always, we’ve also released a number of small enhancements.  The next major release will include an advanced Asset Manager for multimedia and the capability for testlets, groups of items that share a reading passage or other common stimulus.

Xcalibre 4 updated

ASC’s cutting-edge software for item response theory has been substantially updated.  This is a free update for current license holders, and includes several bug fixes as well as improved functionality:

  • User can specify bounds for scaled scores
  • Lz person fit statistic now computed
  • Test response function (TRF) values are now output
  • Chi-square item fit statistics enhanced
  • Subgroup analysis based on DIF group membership
  • Rebuilt ranking of ability estimates to be much faster
  • Observed item response function (fit line) now includes multi-key items
  • Many more small improvements!

Current users can upgrade at our support page; new users can download a free version here.  Don’t forget that graduate students can obtain free licenses from the APM student grants program!

CATSim updated

CATSim, the software that provides the research necessary for computerized adaptive testing (CAT) has also been upgraded.  The update focuses primarily on bug fixes.  Current users can upgrade for free at our support page; new users can download a free version here.

ASC to present at 2012 SIOP Conference

ASC will present a mini-workshop on the development of computerized adaptive tests (CATs) at the Society for Industrial-Organizational Psychology (SIOP) conference.  The session is scheduled for 1:30 P.M. on Friday, April 27.  More information is available at the SIOP website.  Please join us and learn about our CAT solutions and assessments!

ASC Update – February 2012

Here are some brief updates and news for February 2012…

ASC to conduct IRT/CAT Workshop in Botswana – 15 seats open!

ASC will be conducting an applied workshop on item response theory (IRT) and computerized adaptive testing (CAT), March 5-9 at the Grand Palm resort in Gaborone, Botswana.  The workshop is hosted by the Botswana Examinations Council for the benefit of countries in the South African Development Community (SADC).  Approximately 15 seats remain open for researchers and practitioners from Africa.  The cost is approximately $275; participants are responsible for their travel and lodging expenses.  If you are interested in attending, please email solutions@54.89.150.95.

CATSim update

CATSim, the essential program for CAT simulation research, is now available as Version 1.0.6.  Changes include (1) when the classification termination option is used, a new table reports all examinee classifications in addition to the “high-confidence” classifications, and the examinee report differentiates between the two types of classifications; (2) with variable termination and constraints, a new termination type reports that examinees exhausted the item bank before other termination options were reached; and (3) an error using the graded response model with “reversed” item parameters was fixed.  Current license holders can download the no-cost update from https://assess.com/xcart/pages.php?pageid=10.  If you do not yet have CATSim, a free demo version is available at https://assess.com/xcart/product.php?productid=555.

Tech Report: Comparison of Xcalibre 4 and BILOG-MG

A technical report was recently released which provides a comparison of Xcalibre 4 and BILOG-MG, the world’s two leading software programs for item response theory (IRT) analysis.  The two are compared in terms of parameter recovery, which is an appropriate methodology to evaluate the accuracy of IRT parameter estimates.  For the majority of conditions studied, Xcalibre 4 produced more accurate calibrations.  A full copy of the report is at https://assess.com/research.php.

FastTEST Web implements powerful new lockdown browser

FastTEST Web, the world’s most advanced system for internet-based item banking and test delivery, has implemented a new “lockdown” delivery system, WebLock.  WebLock is a special browser that greatly enhances test security by 1) preventing the examinee from accessing any internet sites, 2) preventing the examinee from accessing any information/programs on their computer, and 3) checking for specific software that is a security concern, such as screen-capture programs.  These advantages make it ideal for high-security tests in fields like education, certification, and employment.  Moreover, this is in addition to other important FTW security advantages like role-based access, examinee passwords, and the use of an internal adaptive testing engine.

Happiness and work

http://www.ted.com/talks/shawn_achor_the_happy_secret_to_better_work.html Interesting talk on psychology and work – with some unintentional commentary on the Rasch model around 4:00. (removing some data points simply because you don’t like them, with the excuse of measurement error)

I spent yesterday presenting at the Minnesota Assessment Conference.  My congratulations and thanks to the folks at the Minnesota Department of Education for again organizing a wonderful event with a huge range of topics, from technical psychometric issues all the way to practical district-level topics.  ASC’s plans for upcoming conferences include the CLEAR conference (Sept.  23-25), the European AEA conference (Nov. 4-6), and the Maryland Assessment Conference (Nov. 17-19).  We are also holding a workshop on Sept. 4-5 in Tokyo, Japan.

 

I often get inquiries from around the world about where the next ASC workshop is going to take place.  Given today’s economy, traveling long distances is not always an option, but assessment professionals still want to learn more about IRT and CAT.  For this reason, we have released a recording of our workshop in June 2009.  Now anyone can attend a workshop, for a fraction of the cost of attending in person.

 

FastTEST Web (www.fasttestweb.com) continues to be improved.  Development in July focused on the addition of a Reporting module.  Its purpose is to make test development more efficient by providing informative reports, such as a detailed assessment of the item bank to indicate where more questions are needed, or the current status of tests in your testing program.  The next addition will be computerized adaptive testing (CAT), which will allow anyone to easily create and deliver adaptive tests.

 

The new Iteman 4.1 is slated for release in August.   The most recent additions include a complete reliability analysis and a wizard-type interface for setting up multiple runs.  If you haven’t looked at Iteman 4 yet, I encourage you to check it out.  It is unique in that it is not designed just for statistical analysis, but to actually produce customized reports of high enough quality to provide to stakeholders – in mere seconds.

May, 2010

As of May 5, FastTEST Web is now available to the public!  Please visit www.fasttestweb.com to see some of the features and watch a video of it in action.  We continue to add new features every day – so if there is something you would like to see in an online test development system, please contact me.

The buzz on Iteman 4 continues to grow!  It has been out for approximately a month now, and orders continue to roll in.  Iteman 4 is designed not just to analyze test data, but to produce formal reports – saving you the time of copying and pasting tables and figures for each item.  Remember that there are discounts available both for multiple users and for multiple years (e.g., purchasing 3 years up front rather than the annual renewal).

We continue to add new features to Iteman 4, including the choice of quantile plot groups, and support for polytomous and mixed-format tests.  In addition, Xcalibre is being re-engineered in the same spirit as Iteman 4.

The first IACAT election is complete; results are posted at iacat.org.  The Board is now working on establishing a vision and future for the organization.  Of course, the most immediate future is the conference in June.  A program will be posted shortly.

IACAT 2010 Deadline

First thing to note this month: the deadline for proposals to present at IACAT 2010 (Arnhem, the Netherlands) is February 15, 2010.

 

New research

I added a good amount of research to the ASC research page, https://assess.com/psychometric-research/.  There are now links to 15 published articles, technical reports, white papers, and conference presentations.  Please peruse if you have not already.

 

In the works

ASC is currently in the final stages of two huge projects.  Iteman 4 provides extensive classical analysis of multiple-choice tests.  The previous version, Iteman 3.6, produced simple ASCII text output, but version 4 will produce professional reports, complete with colored figures and tables.  The other project is FastTEST Web, which will provide a web-based solution for item banking, test assembly, examinee management, and online testing.  If you are interested in either of these products, for purchase or as an advanced beta tester, please contact me at nthompson *at* assess.com.

 

CITAS

Most of the software ASC sells is targeted towards researchers and other users that are psychometrically sophisticated, and have the budgets to buy sophisticated software.  An important addition to our arsenal last year was CITAS: the classical item and test analysis spreadsheet.  CITAS is designed to provide a detailed classical psychometric analysis of your test, but with two notably differences from other software.  First, it is free, so it fits into everybody’s budget.  Second, it is designed so that anyone can use it, because it requires no “running” the program by knowing what buttons to click or special codes to write.  All you have to do is type in your data (the item responses), and the output automatically populates.  This requires no expertise in either software or psychometrics, making it ideal for people on the front lines of education, such as school teachers, professors, or training directors.