Subject Matter Experts (SME) in Exam Development

Subject matter experts (SMEs) play a critical role in the the development and management of credentialing programs.  Certification Managers, Program Managers, and Psychometricians are also critical, but they are typically outsiders to the profession, and have to rely on the subject matter experts for anything that requires knowledge of the content or profession.  This post provides an introduction to the role of subject matter experts in exam development.  This is by no means exhaustive, but covers the main points.

Selecting Subject Matter Experts

If there are SMEs involved in the certification, you need to recruit them.  Usually, they are volunteers that serve because they care about the profession.  If they work on the certification exam, they are typically assigned to a Certification Committee or similar name.  This is to distinguish them from subject matter experts that work on other aspects of the organization, such as initial training or continuing education.

Sometimes the expert panel is limited to the Certification Committee, but sometimes not.  For example, you can recruit a large pool of item writers who each submit 5 items but are not involved in the overall process.

SMEs usually do this in a volunteer role, especially if for a nonprofit.  In return, they have professional development and networking opportunities, and of course can put this on their CV.

Defining the Assessment

If an exam exists, it means that at some point, a group of professionals sat around a table and said “We need a exam that covers ____ and will serve a role in our industry to ______.”  While this might seem flippant, it is actually the most important step in the process, because it lays the groundwork for validity.  Validity is the concept that we have an intended purpose for using test scores, and documentation that supports this purpose.  So, we need to first define that purpose!

In the educational realm, this is often driven by curriculum and sometimes legislation, such as a State or national government passing a law that there is end-of-year benchmark exams on Math and English, per the State curriculum.

Job Task Analysis

Job Task Analysis studies are a key step in the development of a defensible credentialing program.  This includes certification, licensure, and certificates.  If an educational assessment, again, the design is typically driven by curriculum.

It is the second step in the process, after the initial definition, and sets the stage for everything that comes afterward.  Moreover, if you seek to get your credential accredited by organizations such as NCCA or ANSI, you need to re-perform the job task analysis study periodically. JTAs are sometimes called job analysis, practice analysis, or role delineation studies.

The job task analysis study in credentialing relies heavily on the experience of Subject Matter Experts (SMEs), just like Cutscore studies. The SMEs have the best tabs on where the profession is evolving and what is most important, which is essential both for the initial JTA and the periodic re-set of the exam. The frequency depends on how quickly your field is evolving, but a cycle of 5 years is often recommended.

The goal of the job task analysis study in certification testing is to gain quantitative data on the structure of the profession.  Therefore, it typically utilizes a survey approach to gain data from as many professionals as possible.  This starts with a group of SMEs generating an initial list of on-the-job tasks, categorizing them, and then publishing a survey.  The end goal is a formal report with a blueprint of what knowledge, skills, and abilities (KSAs) are required for certification in a given role or field, and therefore what are the specifications of the certification test.

• Observe— Typically the psychometrician (that’s us) shadows a representative sample of people who perform the job in question (chosen through Panel Composition) to observe and take notes. After the day(s) of observation, the SMEs sit down with the observer so that he or she may ask any clarifying questions.

  The goal is to avoid doing this during the observation so that the observer has an untainted view of the job.  Alternatively, your SMEs can observe job incumbents – which is often the case when the SMEs are supervisors.

• Generate— The SMEs now have a corpus of information on what is involved with the job, and generate a list of tasks that describe the most important job-related components. Not all job analysis uses tasks, but this is the most common approach in certification testing, hence you will often hear the term job task analysis as a general term.

• Survey— Now that we have a list of tasks, we send a survey out to a larger group of SMEs and ask them to rate various features of each task.

  How important is the task? How often is it performed? What larger category of tasks does it fall into?

• Analyze— Next, we crunch the data and quantitatively evaluate the SMEs’ subjective ratings to determine which of the tasks and categories are most important.

• Review— As a non-SME, the psychometrician needs to take their findings back to the SME panel to review the recommendation and make sure it makes sense.

• Report— We put together a comprehensive report that outlines what the most important tasks/categories are for the given job.  This in turn serves as the foundation for a test blueprint, because more important content deserves more weight on the test.

  This connection is one of the fundamental links in the validity argument for an assessment.

Test Specifications / Blueprints

After the job task analysis is done, the results are statistically analyzed by a psychometrician to produce recommended test blueprints.  This might be one recommendation, or several.  They need to be reviewed and approved by the panel of subject matter experts.

If you are new to this topic, here is a free software tool to help you.

Item Writing and Review

A test needs items, and items need to be written.  In almost all cases, they need to be written by someone with expertise on the topic.  You can’t pull someone off the street and tell them to write questions for an exam on anesthesiology; you will need to have anesthesiologists write those questions.  And it is best practice to have items reviewed by at least one other subject matter expert, typically as part of a predefined workflow.

Standard setting studies (AKA cutscore or passing point studies)

When the JTA is completed, we have to determine who should pass the assessment, and who should fail. This is most often done using the modified Angoff process, where the SMEs conceptualize a minimally competent candidate (MCC) and then set pass/fail point so that the MCC would just barely pass.  There are other methods too, such as Bookmark or Contrasting Groups.

The modified-Angoff process requires the SMEs to meet and discuss the MCC concept.  They then rate all items based on the perceived difficulty to the MCC.  The ratings are evaluated for inter-rater reliability and agreement, and then the SME panel convenes again to discuss any outliers.

Policies and Procedures

Any real certification has a Candidate Handbook which lays out many important policies and procedures.   The SMEs need to define and approve these, though a Certification Manager or consultant can often do much of the drafting.  Here are just a few of the questions:

• How does someone become eligible to take the exam? (education level, job experience)
• What is the price?
• How will the test be delivered? (paper, at conferences, at testing centers, remote proctoring)
• What is the retake policy if someone fails?
• What are the continuing education requirements?

Item Bank Maintenance

As a test is delivered, the results are statistically analyzed.  Psychometricians will flag issues such as items that are too hard or too confusing.  When such items are flagged, it falls to the subject matter expert to review them and figure out how to fix them.  “Why did half the candidates choose B when C was the correct answer?”  “Oh yeah, I can see how this part of the stem could be interpreted in two ways, so we need to change _______.”

Some organizations have annual meetings dedicated to bring the SMEs together for this discussion.

Conclusion

As you can see, a certification exam cannot be built effectively without a team of dedicated subject matter experts to help at each step of the way.  It is a lot of work!  However, many subject matter experts feel a sense of pride by contributing to their profession.  As a bonus, it looks really good on a resume.

Nathan Thompson, PhD

Nathan Thompson, PhD, is CEO and Co-Founder of Assessment Systems Corporation (ASC). He is a psychometrician, software developer, author, and researcher, and evangelist for AI and automation. His mission is to elevate the profession of psychometrics by using software to automate psychometric work like item review, job analysis, and Angoff studies, so we can focus on more innovative work. His core goal is to improve assessment throughout the world.

Nate was originally trained as a psychometrician, with an honors degree at Luther College with a triple major of Math/Psych/Latin, and then a PhD in Psychometrics at the University of Minnesota. He then worked multiple roles in the testing industry, including item writer, test development manager, essay test marker, consulting psychometrician, software developer, project manager, and business leader. He is also cofounder and Membership Director at the International Association for Computerized Adaptive Testing (iacat.org). He’s published 100+ papers and presentations, but his favorite remains https://scholarworks.umass.edu/pare/vol16/iss1/1/.