Likert scales (items) are a type of item used in human psychoeducational assessment, primarily to assess noncognitive constructs.  That is, while item types like multiple choice or short answer are used to measure knowledge or ability, Likert scales are better suited to measuring things like anxiety, conscientiousness, or motivation.

In the realm of psychology, surveys, and market analysis, Likert scales stand tall as one of the most versatile and widely used tools. Whether you’re a researcher, a marketer, or simply someone interested in understanding human attitudes and opinions, grasping the essence of Likert scales can significantly enhance your understanding of data collection and analysis. In this guide, we’ll delve into what Likert scales are, why they’re indispensable, the types of items they’re suited for, and how to score them effectively.

What is a Likert Scale/Item?

A Likert scale, named after its creator Rensis Likert, is a psychometric scale used to gauge attitudes, opinions, perceptions, and behaviors. It typically consists of a series of statements or questions that respondents are asked to rate based on a specified scale. The scale often ranges from strongly disagree to strongly agree, with varying degrees of intensity or frequency in between. Likert scales are primarily used in survey research but have found applications in various fields, including psychology, sociology, marketing, and education.

We’ve all seen these in our past; they are the items that say something like “Rate on a scale of 1 to 5.”  Sometimes the numbers have descriptive text anchors, like you see below.  If these are behaviorally-based, they are called Behaviorally Anchored Rating Scales (BARS).

You can consider the Likert Scale to be the notion of 1 to 5 or Strongly Disagree to Strongly Agree.  A Likert Item is an item on an assessment that uses a Likert Scale.  In many cases, the scale is reused over items; in the example above, we have two items that use the same scale.  However, the terms are often used interchangeably.

Why Use a Likert Scale?

The popularity of Likert scales stems from their simplicity, flexibility, and ability to capture nuanced responses. Here are several reasons why Likert scales are favored:

• Ease of Administration: Likert items are easy to administer, making them suitable for both online and offline surveys.
• Quantifiable Data: Likert scales generate quantitative data, allowing for statistical analysis and comparison across different groups or time points. Open response items, where an examinee might type in how they feel about something, are much harder to quantify.
• Flexibility: They can accommodate a wide range of topics and attitudes, from simple preferences to complex opinions.
• Standardization: Likert scales provide a standardized format for measuring attitudes, enhancing the reliability and validity of research findings.
• Ease of Interpretation: Likert responses are straightforward to interpret, making them accessible to both researchers.  For example, in the first example above, if the average response is 4.1, we can say that respondents generally Agree with the statement.
• Ease of understanding: Since these are so commonly used, everyone is familiar with the format and can respond quickly.

What Sort of Assessments Use a Likert Scale?

Likert scales are well-suited for measuring various constructs, including:

• Attitudes: Assessing attitudes towards a particular issue, product, or service (e.g., “I believe climate change is a pressing issue”).
• Opinions: Gauging opinions on controversial topics or current events (e.g., “I support the legalization of marijuana”).
• Perceptions: Capturing perceptions of quality, satisfaction, or trust (e.g., “I am satisfied with the customer service provided”).
• Behaviors: Examining self-reported behaviors or intentions (e.g., “I exercise regularly”).
• Agreement or Frequency: Measuring agreement with statements or the frequency of certain behaviors (e.g., “I often recycle household waste”).

How Do You Score a Likert Item?

Scoring a Likert scale item involves assigning numerical values to respondents’ selected options. Typically, the scale is assigned values from 1 to 5 (or more), representing varying degrees of agreement, frequency, or intensity.  In the example above, the possible scores for each item are 1, 2, 3, 4, 5.  There are then two ways we can use this to obtain scores for examinees.

• Classical test theory: Either sum or average. For the former, simply add up the
• scores for all items within the scale for each respondent. If they respond as 4 to both items, their score is 8.  For the latter, we find their average answer.  If they answer a 3 and a 4, their score is 3.50.  Note that both of these are easily interpretable.
• Item Response Theory: In large scale assessment, Likert scales are often analyzed and scored with polytomous IRT models such as the Rating Scale Model and Graded Response Model.  An example of this sort of analysis is shown here.

Other important considerations:

• Reverse Coding: If necessary, reverse code items to ensure consistency (e.g., strongly disagree = 1, strongly agree = 5).  In the example above, we are clearly assessing Extraversion; the first item is normal scoring, while the second item is reverse-scored.  So actually, answering a 2 to the second question is really a 4 in the direction of Extraversion, and we would score it as such.
• Collapsing Categories: Sometimes, if few respondents answer a 2, you might collapse 1 and 2 into a single category.  This is especially true if using IRT.  The image here also shows an example of that.
• Norms: Because most traits measured by Likert scales are norm-referenced (see more on that here!), we often need to set norms.  In the simple example, what does a score of 8/10 mean?  The meaning is quite different if the average is 6 with a standard deviation of 1, than if the average is 8.  Because of this, scores might be reported as z-scores or T-scores.

Item Analysis of Likert items

We use statistical techniques to item quality, such as average score per item or item-total correlation (discrimination). You can also perform more advanced analyses like factor analysis or regression to uncover underlying patterns or relationships.  Here are some initial considerations.

• Frequency of each response: How many examinees selected each?  This is the N column in the graph above.  The Prop column is the same thing but converted to proportion.
• Mean score per response: This is evidence that the item is working well.  Did people who answered “1” score lower overall on the Extraversion score than people who scored 3?  This is definitely the case above.
• Rpbis per response, or overall R: We want the item to correlate with total score.  This is strong evidence for the validity of the item.  In this example, the correlation is 0.620, which is great.
• Item response theory:  We can evaluate threshold values and overall item discrimination, as well as issues like item fit.  This is extremely important, but beyond the scope of this post!

We also want to validate the overall test.  Scores and subscores can be evaluated with descriptive statistics, and for reliability with indices like coefficient alpha.

Summary

In conclusion, Likert scales are invaluable tools for capturing and quantifying human attitudes, opinions, and behaviors. Understanding their utility and nuances can empower researchers, marketers, and decision-makers to extract meaningful insights from their data, driving informed decisions and actions. So, whether you’re embarking on a research project, designing a customer satisfaction survey, or conducting employee assessments, remember to leverage Likert scales to efficiently assess the noncognitive traits and opinions.

Test publishing is the process of preparing computer-based assessments for delivery on an electronic platform. Test publishing is like a car rolling off the assembly line. It’s the culmination of a great deal of effort in developing the assessment. Just as a car undergoes extensive checks before leaving the factory, a computer-based assessment requires meticulous quality control procedures to make sure that it functions as intended. Errors may have significant consequences for the sponsoring organization, including a loss of reputation, and can even have legal implications, depending upon the type of error.

The test publishing quality control process begins prior to the  start of the publishing process. The key steps in the process are as follows:

Step 1: Determine the test publishing specifications

Quality control begins with the completion of a test specifications document. The test specifications document provides the pattern or the playbook for how the test should be published. It typically includes the following information:

• Test design
  • Administration model (i.e., linear fixed form, LOFT, CAT)
  • Scoring strategy (dichotomous/polytomous item-level scoring, compensatory/conjunctive domain/sectional scoring)
  • Test length (number of items shown to each candidate)
  • Test duration (time allowed for exam/sections)
• Content specifications
  • List of included items (and which are scored/unscored)
  • Mapping of items to domains/sections/subscales (if applicable)
  • Mapping of stimuli to items (if applicable)
  • Item keys
• Ancillary delivery components
  • Non-disclosure agreement
  • Tutorial
  • Customized help screens
  • Calculator
• Features and functionality
  • Navigation (e.g., review of previous items allowed)
  • Review screens
  • Electronic scratch pad
  • Item-level comments/feedback

Note that this is not a comprehensive list, and information needed for the test specifications documents may vary depending upon the type of assessment and the specific testing platform used for delivery. Some of the data on the test specifications are relatively static and will change only with changes to the test design. Other data, such as the list of included items, are dynamic and will typically change each time the assessment is republished.

The test specifications document becomes the authoritative source of truth used by the test publisher for how the assessment should be published. It is a key communication tool between the sponsoring organization and their test publishing vendor or partner. 

Step 2: Identify sources of test publishing errors

A comprehensive determination of everything that could possibly go wrong in the test publishing process should serve as the guide for the quality control checks that need to be performed before the test goes live. A tool that can assist in developing a comprehensive list of potential errors is a fishbone diagram, also known as an Ishikawa diagram or a cause-and-effect diagram.  It is a visual representation used to identify and organize possible causes of a specific problem or effect. The diagram takes the form of a fish skeleton (hence, its name), with the “head” representing the problem or effect, and the “bones” representing different categories of potential causes. Along each bone, smaller branches represent sub-causes, which are specific elements that may contribute to the problem.

Fishbone diagrams are created by having a team representing all disciplines involved in a process brainstorm potential problems, or in the case of test publishing, potential errors that can be introduced into the test publishing process. Determining potential categories of errors first and then brainstorming more specific errors enables a comprehensive analysis of the test publishing process and the potential occasions in which errors can be introduced in that process.

Here’s a sample fishbone diagram for test publishing errors: 

Step 3: Review against source of truth

The examination should be reviewed against the source of truth for each potential error. The test specifications will be the key source of truth against which the published examination is compared to identify any errors. The item bank is the source of truth for item presentation and item metadata.

Error-free test publishing is central to preserving the test sponsor’s reputation. Even minor mistakes, such as misspelled words, can be damaging. More importantly, errors in a published examination can have deleterious effects for candidates. A scoring error might mean the difference between a candidate failing and passing, and in the case of a certification or licensure examination, that can have dire consequences for the candidate’s career and livelihood. 

As mathematician Nassem Nicholas Taleb stated, “Quality is the result of an intelligent effort, not a chance happening.” A rigorous quality control procedure aids in making the publishing process an intelligent effort.

Predictive Validity is a type of test score validity which evaluates how well a test predicts something in the future, usually with a goal of making more effective decisions about people.  For instance, it is often used in the world of pre-employment testing, where we want a test to predict things like job performance or tenure, so that a company can hire people that do a good job and stay a long time – a very good result for the company, and worth the investment.

Validity, in a general sense, is evidence that we have to support intended interpretations of test scores.  There are different types of evidence that we can gather to do so.  Predictive validity refers to evidence that the test predicts things that it should predict.  If we have quantitative data to support such conclusions, it makes the test more defensible and can improve the efficiency of its use.  For example, if a university admissions test does a great job of predicting success at university, then universities will want to use it to select students that are more likely to succeed.

Examples of Predictive Validity

Predictive validity evidence can be gathered for a variety of assessment types.

1. Pre-employment: Since the entire purpose of a pre-employment test is to positively predict good things like job performance or negatively predict bad things like employee theft or short tenure, a ton of effort goes into developing tests to function in this way, and then documenting that they do.
2. University Admissions: Like pre-employment testing, the entire purpose of university admissions exams is predictive.  They should positively correlate with good things (first year GPA, four year graduation rate) and negatively predict the negative outcomes like academic probation or dropping out.
3. Prep Exams: Preparatory or practice tests are designed to predict performance on their target test.  For example, if a prep test is designed to mimic the Scholastic Aptitude Test (SAT), then one way to validate it is to gather the SAT scores later, after the examinees take it, and correlate with the prep test.
4. Certification & Licensure: The primary purpose of credentialing exams is not to predict job performance, but to ensure that the candidate has mastered the material necessary to practice their profession.  Therefore, predictive validity is not important, compared to content-related validity such as blueprints based on a job analysis. However, some credentialing organizations do research on the “value of certification” linking it to improved job performance, reduced clinical errors, and often external third variables such as greater salary.
5. Medical/Psychological: There are some assessments that are used in a clinical situation, and the predictive validity is necessary in that sense.  For instance, there might be an assessment of knee pain used during initial treatment (physical therapy, injections) that can be predictively correlated with later surgery.  The same assessment might then be used after the surgery to track rehabilitation.

Predictive Validity in Pre-employment Testing

The case of pre-employment testing is perhaps the most common use of this type of validity evidence.  A new study (Sacket, Zhang, Berry, & Lievens, 2022) was recently released that was a meta-analysis of the various types of pre-employment tests and other selection procedures (e.g., structured interview), comparing their predictive validity power.  This was a modern update to the classic article by Schmidt & Hunter (1998).  While in the past the consensus has been that cognitive ability tests provide the best predictive power in the widest range of situations, the new article suggests otherwise.  It recommends the use of structured interview and job knowledge tests, which are more targeted towards the role in question, and therefore not surprising that they are well-performing.  This in turn suggests that you should not buy pre-fab ability tests and use them in a shotgun approach with the assumption of validity generalization, but instead leverage an online testing platform like FastTest that allows you to build high-quality exams that are more specific to your organization.

Why do we need predictive validity?

There are a number of reasons that you might need predictive validity for an exam.  They are almost always regarding the case where the test is used to make important decisions about people.

1. Smarter decision-making: Predictive validity provides valuable insights for decision-makers. It helps recruiters identify the most suitable candidates, educators tailor their teaching methods to enhance student learning, and universities to admit the best students.
2. Legal defensibility: If a test is being used for pre-employment purposes, it is legally required in the USA to either show that the test is obviously job-related (e.g., knowledge of Excel for a bookkeeping job) or that you have hard data demonstrating predictive validity.  Otherwise, you are open for a lawsuit.
3. Financial benefits: Often, the reason for needing improved decisions is very financial.  It is often costly for large companies to recruit and train personnel.  It’s entirely possible that spending $100,000 per year on pre-employment tests could save millions of dollars in the long run.
4. Benefits to the examinee: Sometimes, there is directly a benefit to the examinee.  This is often the case with medical assessments.

How to implement predictive validity

The simplest case is that of regression and correlation.  How well does the test score correlate with the criterion variable?  Below is a oversimplified example, of predicting university GPA from scores on an admissions test.  Here, the correlation is 0.858 and the regression is GPA = 0.34*SCORE + 0.533.  Of course, in real life, you would not see this strong of a predictive power, as there are many other factors which influence GPA.

Advanced Issues

It is usually not a simple situation of two straightforward variables, such as one test and one criterion variable.  Often, there are multiple predictor variables (quantitative reasoning test, MS Excel knowledge test, interview, rating of the candidate’s resume), and moreover there are often multiple criterion variables (job performance ratings, job tenure, counterproductive work behavior).  When you use multiple predictors and a second or third predictor adds some bit of predictive power over that of the first variable, this is known as incremental validity.

You can also implement more complex machine learning models, such as neural networks or support vector machines, if they fit and you have sufficient sample size.

When performing such validation, you need to also be aware of bias.  There can be test bias where the test being used as a predictor is biased against a subgroup.  There can also be predictive bias where two subgroups have the same performance on the test, but one is overpredicted for the criterion and the other is underpredicted.  A rule of thumb for investigating this in the USA is the four-fifths rule.

Summary

Predictive validity is one type of test score validity, referring to evidence that scores from a certain test can predict their intended target variables.  The most common application of it is to pre-employment testing, but it is useful in other situations as well.  But validity is an extremely important and wide-ranging topic, so it is not the only type of validity evidence that you should gather.