Situational judgment tests (SJTs) are a type of assessment typically used in a pre-employment context to assess candidates’ soft skills and decision-making abilities. As the name suggests, we are not trying to assess something like knowledge, but rather the judgments or likely behaviors of candidates in specific situations, such as an unruly customer. These tests have become a critical component of modern recruitment, offering employers valuable insights into how applicants approach real-world scenarios, with higher fidelity than traditional assessments.

The importance of tools like SJTs becomes even clearer when considering the significant costs of poor hiring decisions. The U.S. Department of Labor suggests that the financial impact of a poor hiring decision can amount to roughly 30% of the employee’s annual salary. Similarly, CareerBuilder reports that around three-quarters of employers face an average loss of approximately $15,000 for each bad hire due to various costs such as training, lost productivity, and recruitment expenses. Gallup’s State of the Global Workplace Report 2022 further emphasizes the broader implications, revealing that disengaged employees—often a result of poor hiring practices—cost companies globally $8.8 trillion annually in lost productivity.

In this article, we’ll define situational judgment tests, explore their benefits, and provide an example question to better understand how they work.

What is a Situational Judgment Test?

A Situational Judgment Test (SJT) is a psychological assessment tool designed to evaluate how individuals handle hypothetical workplace scenarios. These tests present a series of realistic situations and ask candidates to choose or rank responses that best reflect how they would act. Unlike traditional aptitude tests that measure specific knowledge or technical skills, SJTs focus on soft skills like problem-solving, teamwork, communication, and adaptability. They can provide a critical amount of incremental validity over cognitive and job knowledge assessments.

SJTs are widely used in recruitment for roles where interpersonal and decision-making skills are critical, such as management, customer service, and healthcare. They can be administered in various formats, including multiple-choice questions, multiple-response items, video scenarios, or interactive simulations.

Example of a Situational Judgment Test Question

Here’s a typical SJT question to illustrate the concept:

Scenario:

You are leading a team project with a tight deadline. One of your team members, who is critical to the project’s success, has missed several key milestones. When you approach them, they reveal they are overwhelmed with personal issues and other work commitments.

Question:

What would you do in this situation?

– Report the issue to your manager and request their intervention.

– Offer to redistribute some of their tasks to other team members to ease their workload.

– Have a one-on-one meeting to understand their challenges and develop a plan together.

– Leave them to handle their tasks independently to avoid micromanaging.

Answer Key:

While there’s no definitive “right” answer in SJTs, some responses align better with desirable workplace behaviors. In this example, Option 3 demonstrates empathy, problem-solving, and leadership, which are highly valued traits in most professional settings.

Because SJTs typically do not have an overtly correct answer, they will sometimes have a partial credit scoring rule. In the example above, you might elect to give 2 points to Option 3 and 1 point to Option 2. Perhaps even a negative point to some options!

Potential topics for SJTs

Customer service – Given a video of an unruly customer, and how would you respond?

Difficult coworker situation – Like the previous example, how would you find a solution?

Police/Fire – It you made a routine traffic stop and the driver was acting intoxicated and belligerent, what would you do?

How to Develop and Deliver an SJT

Development of an SJT is typically more complex than knowledge-based tests, both because it is more difficult to come up with the topic/content of the item as well as plausible distractors and scoring rules. It can also get expensive if you are utilizing simulation formats or high-quality videos for which you hire real actors!

Here are some suggested steps:

1. Define the construct you want to measure
2. Draft item content
3. Establish the scoring rules
4. Have items reviewed by experts
5. Create videos/simulations
6. Set your cutscore (Standard setting)
7. Publish the test

SJTs are almost always delivered by computer nowadays because it is so easy to include multimedia. Below is an example of what this will look like, using ASC’s FastTest platform.

Advantages of Situational Judgment Tests

1. Realistic Assessment of Skills

Unlike theoretical tests, SJTs mimic real-world situations, making them a practical way to evaluate how candidates might behave in the workplace. This approach helps employers identify individuals who align with their organizational values and culture.

2. Focus on Soft Skills

Technical expertise can often be measured through other assessments or qualifications, but soft skills like emotional intelligence, adaptability, and teamwork are harder to gauge. SJTs provide insights into these intangible qualities that are crucial for success in many roles.

3. Reduced Bias

SJTs focus on behavior rather than background, making them a fairer assessment tool. They can help level the playing field by emphasizing practical decision-making over academic credentials or prior experience.

4. Efficient Screening Process

For roles that receive a high volume of applications, SJTs offer a quick and efficient way to filter candidates. By identifying top performers early, organizations can save time and resources in subsequent hiring stages.

5. Improved Candidate Experience

Interactive and scenario-based assessments often feel more engaging to candidates than traditional tests. This positive experience can enhance a company’s employer brand and attract top talent.

Tips for Success in Taking an SJT

If you’re preparing to take a situational judgment test, keep these tips in mind:

– Understand the Role: Research the job to better understand the types of situations that might be encountered, and think through your responses ahead of time.

– Understand the Company: Research organization to align your responses with their values, culture, and expectations.

– Prioritize Key Skills: Many SJTs assess teamwork, leadership, and conflict resolution, so focus on demonstrating these attributes.

– Practice: Familiarize yourself with sample questions to build confidence and improve your response strategy.

Conclusion

Situational judgment tests are a powerful tool for employers to evaluate candidates’ interpersonal and decision-making abilities in a realistic context, and in a way that is much more scalable than 1-on-1 interviews.

For job seekers, they offer an opportunity to showcase soft skills that might not be evident from a resume or educational record alone. As their use continues to grow across industries, understanding and preparing for SJTs can give candidates a competitive edge in the job market.

Additional Resources on SJTs

Lievens, F., & Sackett, P. R. (2012). The validity of interpersonal skills assessment via SJTs: A review. Journal of Applied Psychology, 97(1), 3–17.

Weekley, J. A., & Ployhart, R. E. (Eds.). (2005). Situational judgment tests: Theory, measurement, and application. Psychology Press.

Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validity. Personnel Psychology, 63(1), 83–117.

Confidence intervals (CIs) are a fundamental concept in statistics, used extensively in assessment and measurement to estimate the reliability and precision of data. Whether in scientific research, business analytics, or health studies, confidence intervals provide a range of values that likely contain the true value of a parameter, giving us a better understanding of uncertainty. This article dives into the concept of confidence intervals, how they are used in assessments, and real-world applications to illustrate their importance.

What Is a Confidence Interval?

CI is a range of values, derived from sample data, that is likely to contain the true population parameter. Instead of providing a single estimate (like a mean or proportion), it gives a range of plausible values, often expressed with a specific level of confidence, such as 95% or 99%. For example, if a survey estimates the average height of adult males to be 175 cm with a 95% CI of 170 cm to 180 cm, it means that we can be 95% confident that the true average height of all adult males falls within this range.

These are made by taking plus or minus a standard error times a factor, around the single estimate, creating a lower bound and upper bound for a range.

Upper Bound = Estimate + factor * standard_error

Lower Bound = Estimate – factor * standard_error

The value of the factor depends upon your assumptions and the desired percentage in the range.  You might want 90%, 95%, 0r 99%.  With the standard normal distribution, the factor is 1.96 (see any table of z-scores), which makes for easy quick math of 2 in your head to get a rough idea.  So, in the example above, we might find that the average height for a sample of 100 adult males is 175 cm with a standard deviation of 25.  The standard error of the mean is SD*sqrt(N), which in this case is 25*sqrt(100) = 2.5.  If we take plus or minus 2 times the standard error of 2.5, that is how we get a confidence interval that the true population mean is 95% likely to be between 170 and 180.

This example is from general statistics, but confidence intervals are used for specific reasons in assessment.

How Confidence Intervals Are Used in Assessment and Measurement

1. Statistical Inference

CIs play a crucial role in making inferences about a population based on sample data. For instance, researchers studying the effect of a new drug might calculate a CI for the average improvement in symptoms. This helps determine if the drug has a significant effect compared to a placebo.

2. Quality Control

In industries like manufacturing, CIs help assess product consistency and quality. For example, a factory producing light bulbs may use CIs to estimate the average lifespan of their products. This ensures that most bulbs meet performance standards.

3. Education and Testing

In educational assessments, CIs can provide insights into test reliability and student performance. For instance, a standardized test score might come with a CI to account for variability in test conditions or scoring methods.

Real-World Examples of Confidence Intervals

1. Medical Research

In clinical trials, CIs are often used to estimate the effectiveness of treatments. Suppose a study finds that a new vaccine reduces the risk of a disease by 40%, with a 95% CI of 30% to 50%. This means there’s a high probability that the true effectiveness lies within this range, helping policymakers make informed decisions.

2. Business Analytics

Businesses use CIs to forecast sales, customer satisfaction, or market trends. For example, a company surveying customer satisfaction might report an average satisfaction score of 8 out of 10, with a 95% CI of 7.5 to 8.5. This helps managers gauge customer sentiment while accounting for survey variability.

3. Environmental Studies

Environmental scientists use CIs to measure pollution levels or climate changes. For instance, if data shows that the average global temperature has increased by 1.2°C over the past century, with a CI of 0.9°C to 1.5°C, this range provides a clearer picture of the uncertainty in the estimate.

Confidence Intervals in Education: A Closer Look

CIs are particularly valuable in education, where they help assess the reliability and validity of test scores and other measurements. By understanding and applying CIs, educators, test developers, and policymakers can make more informed decisions that impact students and learning outcomes.

1. Estimating a Range for True Score

CIs are often paired with standard error of measurement (SEM) to provide insights into the reliability of test scores. SEM quantifies the amount of error expected in a score due to various factors like testing conditions or measurement tools.  It gives us a range for a true score around the observed score (see technical note near then end on this).

For example, consider a standardized test with a scaled score range of 200 to 800. If a student scores 700 with an SEM of 20, the 95% CI for their true score is calculated as:

     Score ± (SEM × Z-value for 95% confidence)

$700\pm (20\times 1.96)=700\pm 39.2$

Thus, the 95% CI is approximately 660 to 740. This means we can be 95% confident that the student’s true score lies within this range, accounting for potential measurement error.  Because this is important, it is sometimes factored into important decisions such as setting a cutscore to be hired at a company based on a screening test.

The reasoning for this is accurately described by this quote from Prof. Michael Rodriguez, noted by Mohammed Abulela on LinkedIn:

A test score is a snapshot estimate, based on a sample of knowledge, skills, or dispositions, with a standard error of measurement reflecting the uncertainty in that score-because it is a sample. Fair test score interpretation employs that standard error and does not treat a score as an absolute or precise indicator of performance.

2. Using Standard Error of Estimate (SEE) for Predictions

The standard error of the estimate (SEE) is used to evaluate the accuracy of predictions in models, such as predicting student performance based on prior data.

For instance, suppose that a college readiness score ranges from 0 to 500, and is predicted by a student’s school grades and admissions test score.  If a predictive model estimates a student’s college readiness score to be 450, with an SEE of 25, the 95% confidence interval for this predicted score is:

     450 ± (25 × 1.96) = 450 ± 49

This results in a confidence interval of 401 to 499, indicating that the true readiness score is likely within this range. Such information helps educators evaluate predictive assessments and develop better intervention strategies.

3. Evaluating Group Performance

CIs are also used to assess the performance of groups, such as schools or districts. For instance, if a district’s average math score is 75 with a 95% CI of 73 to 77, policymakers can be fairly confident that the district’s true average falls within this range. This insight is crucial for making fair comparisons between schools or identifying areas that need improvement.

4. Identifying Achievement Gaps

When studying educational equity, CIs help measure differences in achievement between groups, such as socioeconomic or demographic categories. For example, if one group scores an average of 78 with a CI of 76 to 80 and another scores 72 with a CI of 70 to 74, the overlap (or lack thereof) in intervals can indicate whether the gap is statistically significant or might be due to random variability.

5. Informing Curriculum Development

CIs can guide decisions about curriculum and instructional methods. For instance, when pilot-testing a new teaching method, researchers might use CIs to evaluate its effectiveness. If students taught with the new method have scores averaging 85 with a CI of 83 to 87, compared to 80 (78 to 82) for traditional methods, educators might confidently adopt the new approach.

6. Supporting Student Growth Tracking

In long-term assessments, CIs help track student growth by providing a range around estimated progress. If a student’s reading level improves from 60 (58–62) to 68 (66–70), educators can confidently assert growth while acknowledging measurement variability.

Key Benefits of Using Confidence Intervals

• Enhanced Decision-Making: CIs provide a range, rather than a single estimate, making decisions more robust and informed.
• Clarity in Uncertainty: By quantifying uncertainty, confidence intervals allow stakeholders to understand the limitations of the data.
• Improved Communication: Reporting findings with CIs ensures transparency and builds trust in the results.

How to Interpret Confidence Intervals

A common misconception is that a 95% CI means there’s a 95% chance the true value falls within the interval. Instead, it means that if we repeated the study many times, 95% of the calculated intervals would contain the true parameter. Thus, it’s a statement about the method, not the specific interval.  This is similar to the common misinterpretation of an experimental p-value that it is the probability that our alternative hypothesis is true; instead, it is the probability of our experiment’s results if the null is true.

Final Thoughts

CIs are indispensable in assessment and measurement, offering a clearer understanding of data variability and precision. By applying them effectively, researchers, businesses, and policymakers can make better decisions based on statistically sound insights.

Whether estimating population parameters or evaluating the reliability of a new method, CIs provide the tools to navigate uncertainty with confidence. Start using CIs today to bring clarity and precision to your analyses!

Factor analysis is a statistical technique widely used in research to understand and evaluate the underlying structure of assessment data. In fields such as education, psychology, and medicine, this approach to unsupervised machine learning helps researchers and educators identify latent variables, called factors, and which items or tests load on these factors.

For instance, when students take multiple tests, factor analysis can reveal whether these assessments are influenced by common underlying abilities, like verbal reasoning or mathematical reasoning. This insight is crucial for developing reliable and valid assessments, as it helps ensure that test items are measuring the intended constructs. It can also be used to evaluate whether items in an assessment are unidimensional, which is an assumption of both item response theory and classical test theory.

Why Do We Need Factor Analysis?

Factor analysis is a powerful tool for test validation. By analyzing the data, educators and psychometricians can confirm whether the items on a test align with the theoretical constructs they are designed to measure. This ensures that the test is not only reliable but also valid, meaning it accurately reflects the abilities or knowledge it intends to assess. Through this process, factor analysis contributes to the continuous improvement of educational tools, helping to enhance both teaching and learning outcomes.

What is Factor Analysis?

Factor analysis is a comprehensive statistical technique employed to uncover the latent structure underlying a set of observed variables. In the realms of education and psychology, these observed variables are often test scores or scores on individual test items. The primary goal of factor analysis is to identify underlying dimensions, or factors, that explain the patterns of intercorrelations among these variables. By analyzing these intercorrelations, factor analysis helps researchers and test developers understand which variables group together and may be measuring the same underlying construct.

One of the key outputs of factor analysis is the loading table or matrix (see below), which displays the correlations between the observed variables with the latent dimensions, or factors. These loadings indicate how strongly each variable is associated with a particular factor, helping to reveal the structure of the data. Ideally, factor analysis aims to achieve a “simple structure,” where each variable loads highly on one factor and has minimal loadings on others. This clear pattern makes it easier to interpret the results and understand the underlying constructs being measured. By providing insights into the relationships between variables, factor analysis is an essential tool in test development and validation, helping to ensure that assessments are both reliable and valid.

Confirmatory vs. Exploratory Factor Analysis

Factor analysis comes in two main forms: Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA), each serving distinct purposes in research.

Exploratory Factor Analysis (EFA) is typically used when researchers have little to no prior knowledge about the underlying structure of their data. It is a data-driven approach that allows researchers to explore the potential factors that emerge from a set of observed variables. In EFA, the goal is to uncover patterns and identify how many latent factors exist without imposing any preconceived structure on the data. This approach is often used in the early stages of research, where the objective is to discover the underlying dimensions that might explain the relationships among variables.

On the other hand, Confirmatory Factor Analysis (CFA) is a hypothesis-driven approach used when researchers have a clear theoretical model of the factor structure they expect to find. In CFA, researchers specify the number of factors and the relationships between the observed variables and these factors before analyzing the data. The primary goal of CFA is to test whether the data fit the hypothesized model. This approach is often used in later stages of research or in validation studies, where the focus is on confirming the structure that has been previously identified or theoretically proposed. By comparing the model fit indices, researchers can determine how well their proposed factor structure aligns with the actual data, providing a more rigorous test of their hypotheses.

Factor Analysis of Test Batteries or Sections, or Multiple Predictors

Factor analysis is particularly valuable when dealing with test batteries, which are collections of tests designed to measure various aspects of student cognitive abilities, skills, or knowledge. In the context of a test battery, factor analysis helps to identify the underlying structure of the tests and determine whether they measure distinct yet related constructs.

For example, a cognitive ability test battery might include subtests for verbal reasoning, quantitative reasoning, and spatial reasoning. Through factor analysis, researchers can examine how these subtests correlate and whether they load onto separate factors, indicating they measure distinct abilities, or onto a single factor, suggesting a more general underlying ability, often referred to as the “g” factor or general intelligence.

This approach can also incorporate non-assessment data. For example a researcher on employee selection might look at a set of assessments (cognitive ability, job knowledge, quantitative reasoning, MS Word skills, integrity, counterproductive work behavior), but also variables such as interview scores or resume ratings. Below is an oversimplified example of how the loading matrix might look for this.

Table 1

Readers that are familiar with the topic will recognize this as a nod to the work by Walter Borman and Steve Motowidlo on Task vs. Contextual aspects of job performance.  A variable like Job Knowledge would load highly on a factor of task aspects of performing a job.  However, an assessment of counterproductive work behavior might not predict how well they do tasks, but how well they contribute to company culture and other contextual aspects.

This analysis is crucial for ensuring that the test battery provides comprehensive and valid measurements of the constructs it aims to assess. By confirming that each subtest contributes unique information, factor analysis supports the interpretation of composite scores and aids in the design of more effective assessment tools. The process of validating test batteries is essential to maintain the integrity and utility of the test results in educational and psychological settings.

This approach typically uses “regular” factor analysis, which assumes that scores for each input variable are normally distributed. This, of course, is usually the case with something like scores on an intelligence test. But if you are analyzing scores on test items, these are rarely normally distributed, especially for dichotomous data where there is only possible scores of 0 and 1, this is impossible. Therefore, other mathematical approaches must be applied.

Factor Analysis on the Item Level

Factor analysis at the item level is a more granular approach, focusing on the individual test items rather than entire subtests or batteries. This method is used to ensure that each item contributes appropriately to the overall construct being measured and to identify any items that do not align well with the intended factors.

For instance, in a reading comprehension test, factor analysis at the item level can reveal whether each question accurately measures the construct of reading comprehension or whether some items are more aligned with other factors, such as vocabulary knowledge or reasoning skills. Items that do not load strongly onto the intended factor may be flagged for revision or removal, as they could distort the accuracy of the test scores.

This item-level analysis is crucial for developing high-quality educational or knowledge assessments, as it helps to ensure that every question is both valid and reliable, contributing meaningfully to the overall test score. It also aids in identifying “enemy items,” which are questions that could undermine the test’s consistency and fairness.

Similarly, in personality assessments like the Big Five Personality Test, factor analysis is used to confirm the structure of personality traits, ensuring that the test accurately captures the five broad dimensions: openness, conscientiousness, extraversion, agreeableness, and neuroticism. This process ensures that each trait is measured distinctly while also considering how they may interrelate.  Note that the result here was not to show overall unidimensionality in personality, but evidence to support five factors.  An assessment of a given factor is then more or less unidimensional.

An example of this is show in Table 2 below.  Consider if all the descriptive statements are items in a survey where people rate them on a Likert scale of 1 to 5.  The survey might have hundreds of adjectives but these would align themselves with the Big Five with factor analysis, and the simple structure would look like something you see below (2 items per factor in this small example).

Table 2

Tools like MicroFACT, a specialized software for evaluating unidimensionality, are invaluable in this process. MicroFACT enables psychometricians to assess whether each item in a test measures a single underlying construct, ensuring the test’s coherence and effectiveness.

Summary

Factor analysis plays a pivotal role in the field of psychometrics, offering deep insights into the structure and validity of educational assessments. Whether applied to test batteries or individual items, factor analysis helps ensure that tests are both reliable and meaningful.

Overall, factor analysis is indispensable for developing effective educational tools and improving assessment practices. It ensures that tests not only measure what they are supposed to but also do so in a way that is fair and consistent across different groups and over time. As educational assessments continue to evolve, the insights provided by factor analysis will remain crucial in maintaining the integrity and effectiveness of these tools.

References

Geisinger, K. F., Bracken, B. A., Carlson, J. F., Hansen, J.-I. C., Kuncel, N. R., Reise, S. P., & Rodriguez, M. C. (Eds.). (2013). APA handbook of testing and assessment in psychology, Vol. 1. Test theory and testing and assessment in industrial and organizational psychology. American Psychological Association. https://doi.org/10.1037/14047-000

Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). The Guilford Press.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). Tata Mcgraw-Hill Ed.