Posts on psychometrics: The Science of Assessment

FastTest - Situational Judgment Test SJT example

Situational Judgment Tests: Higher Fidelity in Pre-Employment Testing

Nathan Thompson, PhDNovember 30, 2024

Situational judgment tests (SJTs) are a type of assessment typically used in a pre-employment context to assess candidates’ soft skills and decision-making abilities. As the name suggests, we are not trying to assess something like knowledge, but rather the judgments or likely behaviors of candidates in specific situations, such as an unruly customer. These tests have become a critical component of modern recruitment, offering employers valuable insights into how applicants approach real-world scenarios, with higher fidelity than traditional assessments.

The importance of tools like SJTs becomes even clearer when considering the significant costs of poor hiring decisions. The U.S. Department of Labor suggests that the financial impact of a poor hiring decision can amount to roughly 30% of the employee’s annual salary. Similarly, CareerBuilder reports that around three-quarters of employers face an average loss of approximately $15,000 for each bad hire due to various costs such as training, lost productivity, and recruitment expenses. Gallup’s State of the Global Workplace Report 2022 further emphasizes the broader implications, revealing that disengaged employees—often a result of poor hiring practices—cost companies globally $8.8 trillion annually in lost productivity.

In this article, we’ll define situational judgment tests, explore their benefits, and provide an example question to better understand how they work.

What is a Situational Judgment Test?

A Situational Judgment Test (SJT) is a psychological assessment tool designed to evaluate how individuals handle hypothetical workplace scenarios. These tests present a series of realistic situations and ask candidates to choose or rank responses that best reflect how they would act. Unlike traditional aptitude tests that measure specific knowledge or technical skills, SJTs focus on soft skills like problem-solving, teamwork, communication, and adaptability. They can provide a critical amount of incremental validity over cognitive and job knowledge assessments.

SJTs are widely used in recruitment for roles where interpersonal and decision-making skills are critical, such as management, customer service, and healthcare. They can be administered in various formats, including multiple-choice questions, multiple-response items, video scenarios, or interactive simulations.

Example of a Situational Judgment Test Question

Here’s a typical SJT question to illustrate the concept:

 

Scenario:

You are leading a team project with a tight deadline. One of your team members, who is critical to the project’s success, has missed several key milestones. When you approach them, they reveal they are overwhelmed with personal issues and other work commitments.

hr-interview-pre-employment

Question:

What would you do in this situation?

– Report the issue to your manager and request their intervention.

– Offer to redistribute some of their tasks to other team members to ease their workload.

– Have a one-on-one meeting to understand their challenges and develop a plan together.

– Leave them to handle their tasks independently to avoid micromanaging.

 

Answer Key:

While there’s no definitive “right” answer in SJTs, some responses align better with desirable workplace behaviors. In this example, Option 3 demonstrates empathy, problem-solving, and leadership, which are highly valued traits in most professional settings.

 

Because SJTs typically do not have an overtly correct answer, they will sometimes have a partial credit scoring rule. In the example above, you might elect to give 2 points to Option 3 and 1 point to Option 2. Perhaps even a negative point to some options!

Potential topics for SJTs

Customer service – Given a video of an unruly customer, and how would you respond?

Difficult coworker situation – Like the previous example, how would you find a solution?

Police/Fire – It you made a routine traffic stop and the driver was acting intoxicated and belligerent, what would you do?

How to Develop and Deliver an SJT

Development of an SJT is typically more complex than knowledge-based tests, both because it is more difficult to come up with the topic/content of the item as well as plausible distractors and scoring rules. It can also get expensive if you are utilizing simulation formats or high-quality videos for which you hire real actors!

Here are some suggested steps:

  1. Define the construct you want to measure
  2. Draft item content
  3. Establish the scoring rules
  4. Have items reviewed by experts
  5. Create videos/simulations
  6. Set your cutscore (Standard setting)
  7. Publish the test

SJTs are almost always delivered by computer nowadays because it is so easy to include multimedia. Below is an example of what this will look like, using ASC’s FastTest platform.

FastTest - Situational Judgment Test SJT example

Advantages of Situational Judgment Tests

1. Realistic Assessment of Skills

Unlike theoretical tests, SJTs mimic real-world situations, making them a practical way to evaluate how candidates might behave in the workplace. This approach helps employers identify individuals who align with their organizational values and culture.

2. Focus on Soft Skills

Technical expertise can often be measured through other assessments or qualifications, but soft skills like emotional intelligence, adaptability, and teamwork are harder to gauge. SJTs provide insights into these intangible qualities that are crucial for success in many roles.

3. Reduced Bias

SJTs focus on behavior rather than background, making them a fairer assessment tool. They can help level the playing field by emphasizing practical decision-making over academic credentials or prior experience.

4. Efficient Screening Process

For roles that receive a high volume of applications, SJTs offer a quick and efficient way to filter candidates. By identifying top performers early, organizations can save time and resources in subsequent hiring stages.

5. Improved Candidate Experience

Interactive and scenario-based assessments often feel more engaging to candidates than traditional tests. This positive experience can enhance a company’s employer brand and attract top talent.

Tips for Success in Taking an SJT

If you’re preparing to take a situational judgment test, keep these tips in mind:

– Understand the Role: Research the job to better understand the types of situations that might be encountered, and think through your responses ahead of time.

– Understand the Company: Research organization to align your responses with their values, culture, and expectations.

– Prioritize Key Skills: Many SJTs assess teamwork, leadership, and conflict resolution, so focus on demonstrating these attributes.

– Practice: Familiarize yourself with sample questions to build confidence and improve your response strategy.

Conclusion

Situational judgment tests are a powerful tool for employers to evaluate candidates’ interpersonal and decision-making abilities in a realistic context, and in a way that is much more scalable than 1-on-1 interviews.

For job seekers, they offer an opportunity to showcase soft skills that might not be evident from a resume or educational record alone. As their use continues to grow across industries, understanding and preparing for SJTs can give candidates a competitive edge in the job market.

Additional Resources on SJTs

Lievens, F., & Sackett, P. R. (2012). The validity of interpersonal skills assessment via SJTs: A review. Journal of Applied Psychology, 97(1), 3–17.

Weekley, J. A., & Ployhart, R. E. (Eds.). (2005). Situational judgment tests: Theory, measurement, and application. Psychology Press.

Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validity. Personnel Psychology, 63(1), 83–117.

/by
confidence-intervals-avatar

Confidence Intervals in Assessment and Psychometrics

Laila Issayeva M.Sc.November 25, 2024

Confidence intervals (CIs) are a fundamental concept in statistics, used extensively in assessment and measurement to estimate the reliability and precision of data. Whether in scientific research, business analytics, or health studies, confidence intervals provide a range of values that likely contain the true value of a parameter, giving us a better understanding of uncertainty. This article dives into the concept of confidence intervals, how they are used in assessments, and real-world applications to illustrate their importance.

What Is a Confidence Interval?

CI is a range of values, derived from sample data, that is likely to contain the true population parameter. Instead of providing a single estimate (like a mean or proportion), it gives a range of plausible values, often expressed with a specific level of confidence, such as 95% or 99%. For example, if a survey estimates the average height of adult males to be 175 cm with a 95% CI of 170 cm to 180 cm, it means that we can be 95% confident that the true average height of all adult males falls within this range.

These are made by taking plus or minus a standard error times a factor, around the single estimate, creating a lower bound and upper bound for a range.

Upper Bound = Estimate + factor * standard_error

Lower Bound = Estimate – factor * standard_error

The value of the factor depends upon your assumptions and the desired percentage in the range.  You might want 90%, 95%, 0r 99%.  With the standard normal distribution, the factor is 1.96 (see any table of z-scores), which makes for easy quick math of 2 in your head to get a rough idea.  So, in the example above, we might find that the average height for a sample of 100 adult males is 175 cm with a standard deviation of 25.  The standard error of the mean is SD*sqrt(N), which in this case is 25*sqrt(100) = 2.5.  If we take plus or minus 2 times the standard error of 2.5, that is how we get a confidence interval that the true population mean is 95% likely to be between 170 and 180.

This example is from general statistics, but confidence intervals are used for specific reasons in assessment.

 

How Confidence Intervals Are Used in Assessment and Measurement

calculating-confidence-interval

1. Statistical Inference

CIs play a crucial role in making inferences about a population based on sample data. For instance, researchers studying the effect of a new drug might calculate a CI for the average improvement in symptoms. This helps determine if the drug has a significant effect compared to a placebo.

2. Quality Control

In industries like manufacturing, CIs help assess product consistency and quality. For example, a factory producing light bulbs may use CIs to estimate the average lifespan of their products. This ensures that most bulbs meet performance standards.

3. Education and Testing

In educational assessments, CIs can provide insights into test reliability and student performance. For instance, a standardized test score might come with a CI to account for variability in test conditions or scoring methods.

Real-World Examples of Confidence Intervals

1. Medical Research

explaining-confidence-intervals

In clinical trials, CIs are often used to estimate the effectiveness of treatments. Suppose a study finds that a new vaccine reduces the risk of a disease by 40%, with a 95% CI of 30% to 50%. This means there’s a high probability that the true effectiveness lies within this range, helping policymakers make informed decisions.

2. Business Analytics

Businesses use CIs to forecast sales, customer satisfaction, or market trends. For example, a company surveying customer satisfaction might report an average satisfaction score of 8 out of 10, with a 95% CI of 7.5 to 8.5. This helps managers gauge customer sentiment while accounting for survey variability.

3. Environmental Studies

Environmental scientists use CIs to measure pollution levels or climate changes. For instance, if data shows that the average global temperature has increased by 1.2°C over the past century, with a CI of 0.9°C to 1.5°C, this range provides a clearer picture of the uncertainty in the estimate.

Confidence Intervals in Education: A Closer Look

CIs are particularly valuable in education, where they help assess the reliability and validity of test scores and other measurements. By understanding and applying CIs, educators, test developers, and policymakers can make more informed decisions that impact students and learning outcomes.

1. Estimating a Range for True Score

CIs are often paired with standard error of measurement (SEM) to provide insights into the reliability of test scores. SEM quantifies the amount of error expected in a score due to various factors like testing conditions or measurement tools.  It gives us a range for a true score around the observed score (see technical note near then end on this).

For example, consider a standardized test with a scaled score range of 200 to 800. If a student scores 700 with an SEM of 20, the 95% CI for their true score is calculated as:

     Score ± (SEM × Z-value for 95% confidence)

     700 ± (20 × 1.96) = 700 ± 39.2700 ± (20 × 1.96) = 700 ± 39.2

Thus, the 95% CI is approximately 660 to 740. This means we can be 95% confident that the student’s true score lies within this range, accounting for potential measurement error.  Because this is important, it is sometimes factored into important decisions such as setting a cutscore to be hired at a company based on a screening test.

The reasoning for this is accurately described by this quote from Prof. Michael Rodriguez, noted by Mohammed Abulela on LinkedIn:

A test score is a snapshot estimate, based on a sample of knowledge, skills, or dispositions, with a standard error of measurement reflecting the uncertainty in that score-because it is a sample. Fair test score interpretation employs that standard error and does not treat a score as an absolute or precise indicator of performance.

2. Using Standard Error of Estimate (SEE) for Predictions

The standard error of the estimate (SEE) is used to evaluate the accuracy of predictions in models, such as predicting student performance based on prior data.

For instance, suppose that a college readiness score ranges from 0 to 500, and is predicted by a student’s school grades and admissions test score.  If a predictive model estimates a student’s college readiness score to be 450, with an SEE of 25, the 95% confidence interval for this predicted score is:

     450 ± (25 × 1.96) = 450 ± 49

This results in a confidence interval of 401 to 499, indicating that the true readiness score is likely within this range. Such information helps educators evaluate predictive assessments and develop better intervention strategies.

3. Evaluating Group Performance

confidence-intervals-schemes

CIs are also used to assess the performance of groups, such as schools or districts. For instance, if a district’s average math score is 75 with a 95% CI of 73 to 77, policymakers can be fairly confident that the district’s true average falls within this range. This insight is crucial for making fair comparisons between schools or identifying areas that need improvement.

4. Identifying Achievement Gaps

When studying educational equity, CIs help measure differences in achievement between groups, such as socioeconomic or demographic categories. For example, if one group scores an average of 78 with a CI of 76 to 80 and another scores 72 with a CI of 70 to 74, the overlap (or lack thereof) in intervals can indicate whether the gap is statistically significant or might be due to random variability.

5. Informing Curriculum Development

CIs can guide decisions about curriculum and instructional methods. For instance, when pilot-testing a new teaching method, researchers might use CIs to evaluate its effectiveness. If students taught with the new method have scores averaging 85 with a CI of 83 to 87, compared to 80 (78 to 82) for traditional methods, educators might confidently adopt the new approach.

6. Supporting Student Growth Tracking

In long-term assessments, CIs help track student growth by providing a range around estimated progress. If a student’s reading level improves from 60 (58–62) to 68 (66–70), educators can confidently assert growth while acknowledging measurement variability.

Key Benefits of Using Confidence Intervals

  • Enhanced Decision-Making: CIs provide a range, rather than a single estimate, making decisions more robust and informed.
  • Clarity in Uncertainty: By quantifying uncertainty, confidence intervals allow stakeholders to understand the limitations of the data.
  • Improved Communication: Reporting findings with CIs ensures transparency and builds trust in the results.

 

How to Interpret Confidence Intervals

A common misconception is that a 95% CI means there’s a 95% chance the true value falls within the interval. Instead, it means that if we repeated the study many times, 95% of the calculated intervals would contain the true parameter. Thus, it’s a statement about the method, not the specific interval.  This is similar to the common misinterpretation of an experimental p-value that it is the probability that our alternative hypothesis is true; instead, it is the probability of our experiment’s results if the null is true.

Final Thoughts

CIs are indispensable in assessment and measurement, offering a clearer understanding of data variability and precision. By applying them effectively, researchers, businesses, and policymakers can make better decisions based on statistically sound insights.

Whether estimating population parameters or evaluating the reliability of a new method, CIs provide the tools to navigate uncertainty with confidence. Start using CIs today to bring clarity and precision to your analyses!

/by
general-intelligence-avatar

General Intelligence and Its Role in Assessment and Measurement

Laila Issayeva M.Sc.November 10, 2024

General intelligence, often symbolized as “g,” is a concept that has been central to psychology and cognitive science since the early 20th century. First introduced by Charles Spearman, general intelligence represents an individual’s overall cognitive ability. This foundational concept has evolved over the years and remains crucial in both academic and applied settings, particularly in assessment and measurement. Understanding general intelligence can help in evaluating mental abilities, predicting academic and career success, and creating reliable and valid assessment tools. This article delves into the nature of general intelligence, its assessment, and its importance in measurement fields.

What is General Intelligence?

general-intelligence-idea

General intelligence (GI), or “g,” is a theoretical construct referring to the common cognitive abilities underlying performance across various mental tasks. Spearman proposed that a general cognitive ability contributes to performance in a wide range of intellectual tasks. This ability encompasses multiple cognitive skills, such as reasoning, memory, and problem-solving, which are thought to be interconnected. In Spearman’s model, a person’s performance on any cognitive test relies partially on “g” and partially on task-specific skills.

For example, both solving complex math problems and understanding a new language involve specific abilities unique to each task but are also underpinned by an individual’s GI. This concept has been pivotal in shaping how we understand cognitive abilities and the development of intelligence tests.

To further explore the foundational aspects of intelligence, the Positive Manifold phenomenon demonstrates that most cognitive tasks tend to be positively correlated, meaning that high performance in one area generally predicts strong performance in others. You can read more about it in our article on Positive Manifold.

GI in Assessment and Measurement

The assessment of GI has been integral to psychology, education, and organizational settings for decades. Testing for “g” provides insight into an individual’s mental abilities and often serves as a predictor of various outcomes, such as academic performance, job performance, and life success.

  1. Intelligence Testing: Intelligence tests, like the Wechsler Adult Intelligence Scale (WAIS) and Stanford-Binet, aim to provide a measurement of GI. These tests typically consist of a variety of subtests measuring different cognitive skills, including verbal comprehension, working memory, and perceptual reasoning. The results are aggregated to produce an overall IQ score, representing a general measure of “g.” These scores are then compared to population averages to understand where an individual stands in terms of cognitive abilities relative to their peers.
  2. Educational Assessment: GI is often used in educational assessments to help identify students who may need additional support or advanced academic opportunities. For example, cognitive ability tests can assist in identifying gifted students who may benefit from accelerated programs or those who need extra resources. Schools also use “g” as one factor in admission processes, relying on tests like the SAT, GRE, and similar exams, which assess reasoning and problem-solving abilities linked to GI.
  3. Job and Career Assessments: Many organizations use cognitive ability tests as part of their recruitment processes. GI has been shown to predict job performance across many types of employment, especially those requiring complex decision-making and problem-solving skills. By assessing “g,” employers can gauge a candidate’s potential for learning new tasks, adapting to job challenges, and developing in their role. This approach is especially prominent in fields requiring high levels of cognitive performance, such as research, engineering, and management. One notable example is the Armed Services Vocational Aptitude Battery (ASVAB), a multi-test battery that assesses candidates for military service. The ASVAB includes subtests like arithmetic reasoning, mechanical comprehension, and word knowledge, all of which reflect diverse cognitive abilities. These individual scores are then combined into the Armed Forces Qualifying Test (AFQT) score, an overall measure that serves as a proxy for GI. The AFQT score acts as a threshold across military branches, with each branch requiring minimum scores.

Here are a few ASVAB-style sample questions that reflect different cognitive areas while collectively representing general intelligence:

  1. Arithmetic Reasoning:
    If a train travels at 60 mph for 3 hours, how far does it go?
    Answer: 180 miles
  2. Word Knowledge:
    What does the word “arduous” most nearly mean?
    Answer: Difficult
  3. Mechanical Comprehension:
    If gear A turns clockwise, which direction will gear B turn if it is directly connected?
    Answer: Counterclockwise

 

How GI is Measured

studying-cognitive-abilities

In measuring GI, psychometricians use a variety of statistical techniques to ensure the reliability and validity of intelligence assessments. One common approach is factor analysis, a statistical method that identifies the relationships between variables and ensures that test items truly measure “g” as intended.

Tests designed to measure general intelligence are structured to cover a range of cognitive functions, capturing a broad spectrum of mental abilities. Each subtest score contributes to a composite score that reflects an individual’s general cognitive ability. Assessments are also periodically normed, or standardized, so that scores remain meaningful and comparable over time. This standardization process helps maintain the relevance of GI scores in diverse populations.

 

The Importance of GI in Modern Assessment

GI continues to be a critical measure for various practical and theoretical applications:

  • Predicting Success: Numerous studies have linked GI to a wide array of outcomes, from academic performance to career advancement. Because “g” encompasses the ability to learn and adapt, it is often a better predictor of success than task-specific skills alone. In fact, meta-analyses indicate that g accounts for approximately 25% of the variance in job performance, highlighting its unparalleled predictive power in educational and occupational contexts.
  • Validating Assessments: In psychometrics, GI is used to validate and calibrate assessment tools, ensuring that they measure what they intend to. Understanding “g” helps in creating reliable test batteries and composite scores, making it essential for effective educational and professional testing.
  • Advancing Cognitive Research: GI also plays a vital role in cognitive research, helping psychologists understand the nature of mental processes and the structure of human cognition. Studies on “g” contribute to theories about how people learn, adapt, and solve problems, fueling ongoing research in cognitive psychology and neuroscience.

 

The Future of GI in Assessment

With advancements in technology, the assessment of GI is becoming more sophisticated and accessible. Computerized adaptive testing (CAT) and machine learning algorithms allow for more personalized assessments, adjusting test difficulty based on real-time responses. These innovations not only improve the accuracy of GI testing but also provide a more engaging experience for test-takers.

As our understanding of human cognition expands, the concept of GI remains a cornerstone in both educational and occupational assessments. The “g” factor offers a powerful framework for understanding mental abilities and continues to be a robust predictor of various life outcomes. Whether applied in the classroom, the workplace, or in broader psychological research, GI is a valuable metric for understanding human potential and guiding personal and professional development.

/by
Big Five Personality Traits Avatar

Big Five Personality Traits: A Foundation for Modern Assessments

Laila Issayeva M.Sc.November 1, 2024

Personality plays a crucial role in shaping our behaviors, attitudes, and overall interactions with the world. One of the most widely accepted frameworks for understanding personality is the Big Five Personality Traits model—also known as the OCEAN model—consist of five broad dimensions that describe human personality. These traits are:

  1. Openness to Experience
  2. Conscientiousness
  3. Extraversion
  4. Agreeableness
  5. Neuroticism

In the field of psychological and educational assessment, personality measurements serve as powerful tools for understanding individuals and predicting outcomes like job performance, academic success, and personal satisfaction. In this blog post, we will explore the Big Five’s role in assessments and highlight its relevance in various applications.

The Big Five and Assessment: Practical Applications

The Big Five Personality Traits are widely used in assessments across fields, from human resources to clinical psychology, and even in education. These traits provide a measurable, standardized way of understanding an individual’s personality. Through assessments based on this model, employers can predict how well a potential employee might fit within a team, or educational institutions can gauge a student’s learning preferences and potential areas of growth.

Here’s a breakdown of how the Big Five dimensions translate into practical assessments:

Big Five Personality Traits Scheme

  • Openness to Experience: This trait reflects creativity, curiosity, and a willingness to embrace new ideas. Assessments can use this dimension to identify candidates or students who thrive in environments requiring innovation and adaptability.
  • Conscientiousness: Often linked to self-discipline, responsibility, and goal-directed behavior, conscientiousness assessments are crucial in predicting job performance. Employees high in conscientiousness tend to be reliable, organized, and effective in roles requiring careful planning and execution. A study by Sanchez-Ruiz and Khoury (2019) highlights conscientiousness as a strong predictor of academic success, explaining up to 14% of the variance in academic performance (e.g., GPA).
  • Extraversion: Assessing extraversion helps understand how individuals interact with their environment. In team-based settings, extraverted individuals are often proactive, enthusiastic, and effective communicators. This trait is particularly relevant in leadership assessments, as discussed in our post on general intelligence and leadership dynamics.
  • Agreeableness: Personality assessments measuring agreeableness can reveal how cooperative, empathetic, and considerate a person is. High levels of agreeableness are often associated with positive teamwork dynamics, which is valuable for roles involving collaboration and conflict resolution.
  • Neuroticism: This trait measures emotional stability. Individuals with lower neuroticism scores are better able to cope with stress and are less likely to experience negative emotions frequently. Understanding this trait helps organizations manage workplace stress and predict an employee’s reaction to pressure.

A critical method used to develop and validate the Big Five framework is factor analysis, a statistical technique that helps identify underlying variables or factors that explain the patterns observed in data. Factor analysis ensures that each trait in the model reflects distinct personality dimensions, contributing to its scientific rigor and practical applicability in assessments.

Adapting Big Five Assessments Across Populations: Example Items

The Big Five Personality Traits offer a flexible framework for assessing personality across diverse groups. Tailoring assessment items to specific populations—such as high school students, job candidates, or employees—ensures that the results are relevant and actionable. Below are examples of how Big Five items might vary to suit these distinct contexts:

High School Students

High school assessments aim to capture personality traits in a way that is relatable to young learners’ academic and social experiences. The language is straightforward, focusing on school-related scenarios and responsibilities.

  • Openness to Experience: “I enjoy trying new activities in school, even if they’re unfamiliar to me.”
  • Conscientiousness: “I complete my homework on time and organize my assignments.”
  • Extraversion: “I often speak up in class discussions and like working with others on group projects.”
  • Agreeableness: “I try to understand my classmates’ feelings and help them when I can.”
  • Neuroticism: “I get nervous before tests or when I have to speak in front of the class.”

College Students

college-students

For college assessments, items often bridge between academic and emerging professional experiences. Questions highlight independence and social interactions typical in a college environment.

  • Openness to Experience: “I seek out new learning experiences that help broaden my academic interests.”
  • Conscientiousness: “I manage my study schedule effectively to stay on top of my coursework.”
  • Extraversion: “I feel comfortable participating in campus organizations or social activities.”
  • Agreeableness: “I respect my classmates’ ideas during group assignments and try to work toward common goals.”
  • Neuroticism: “I can manage academic pressure and stay calm during exams.”

Job Candidates or Employees

For employee selection and professional development, items are adapted to reflect workplace scenarios. The questions target traits relevant to job performance, teamwork, and adaptability in a professional setting.

  • Openness to Experience: “I am comfortable taking on new projects, even if they involve skills I haven’t fully developed yet.”
  • Conscientiousness: “I am meticulous in planning and meeting deadlines at work.”
  • Extraversion: “I enjoy collaborating with team members and sharing ideas in meetings.”
  • Agreeableness: “I make an effort to understand my colleagues’ perspectives and offer help when needed.”
  • Neuroticism: “I handle work-related stress effectively, staying calm under pressure.”

These examples illustrate the adaptability of the Big Five model, ensuring that assessments are both meaningful and relevant to the unique demands of each population. By adjusting the context and language, personality assessments can yield insights that are more closely aligned with individuals’ day-to-day environments and challenges.

Assessing Personality in Recruitment and Development

Exploring Big Five Personality Traits

Personality assessments based on the Big Five are particularly valuable in recruitment and professional development. These tests allow employers to gain insights into whether a candidate’s personality traits align with the demands of the role or the company culture. For instance, a high score in conscientiousness might indicate a strong fit for roles requiring meticulous attention to detail, while high extraversion may be suited for sales or customer-facing positions.

In the educational domain, personality assessments can guide personalized learning experiences. By understanding student’s traits, teachers and administrators can tailor learning environments that optimize student engagement and motivation, aligning personality with learning approaches. The use of personality data alongside other cognitive measures, like general intelligence tests, provides a more holistic view of an individual’s strengths and areas for growth. For instance, this paper investigates the impact of Big Five personality traits on the academic performance of university students.

The Value of Big Five Assessments in Modern Contexts

Modern assessments incorporating the Big Five model provide organizations and institutions with actionable insights. They help predict employee retention, job satisfaction, academic achievement, and overall well-being. In an era where data-driven decisions are crucial, using personality assessments allows for more informed hiring, training, and development processes.

As technology continues to advance, these assessments are becoming more efficient and accessible through tools like computerized adaptive testing (CAT) and online platforms. These modern approaches ensure that personality assessments are more personalized and adaptive to the test-taker’s responses, enhancing the accuracy and relevance of the results.

Exploring Open-Source Personality Assessments

For those interested in learning more about Big Five personality assessments, the Open Psychometrics Project provides access to an IPIP-based Big Five assessment. This resource offers a practical introduction to personality testing, allowing users to explore the model’s dimensions through a public, scientifically supported tool.

Conclusion

The Big Five Personality Traits are foundational in the field of assessment, offering a reliable and scientifically grounded framework to understand and measure human behavior. Like the RIASEC model, which categorizes individuals based on career-oriented personality factors, the Big Five provides a structured approach to capture personality dimensions, whether applied in recruitment, educational development, or personal growth. Assessments based on the Big Five offer valuable insights that help individuals and organizations thrive.

 

/by
byod assessment

BYOD in Assessment: Enhancing Learning with Personal Devices

Jacob BirnbaumOctober 31, 2024

The introduction of Bring Your Own Device (BYOD) policies by academic institutions has led to significant changes in education, particularly in the context of assessments. These policies permit students to bring and use their personal devices—such as laptops, tablets, or smartphones—during tests and exams, allowing them to take assessments in a familiar environment and potentially improving their performance. According to this source, 59% of companies have implemented BYOD policies by 2024. However, implementing BYOD for assessments also involves encountering challenges alongside the opportunities these policies provide.

Impact on Student Performance

Implementing BYOD in assessments can significantly improve student performance. Students benefit from reduced anxiety and a seamless experience when utilizing a device they are familiar with. For instance, a student accustomed to the accessibility features on their personal tablet, like text-to-speech or customized display settings, can perform better when these features are readily available during assessments.

Due to using a device they are familiar with, students may face fewer technical issues, such as navigating an unfamiliar operating system or using a non-preferred input method. For example, left-handed students might struggle with a right-handed mouse setup on school devices but would be comfortable using their own device configured to their preferences.

BYOD allows for a greater variety of resources to be utilized, such as educational applications and online materials, which can aid students’ problem-solving and critical thinking. In mathematics assessments, students might use graphing tools they’re proficient with, leading to deeper understanding and application of concepts. Furthermore, formative assessments can be conducted through engaging platforms that provide immediate feedback, aiding educators in tailoring instruction to meet individual students’ needs. Platforms like Kahoot! and Quizlet make learning interactive and can be seamlessly integrated into BYOD environments.

Security Concerns

Despite these advantages, BYOD presents potential security concerns, especially regarding the integrity of assessments. Students may access unauthorized resources, such as websites or applications, to find answers during an exam. For instance, a student might use messaging apps to communicate with peers or search engines to look up answers, compromising the fairness of the assessment.

To mitigate these concerns, strict guidelines and monitoring practices should be implemented by institutions and instructors. Secure testing environments are crucial, including both the physical location and the use of specific software that restricts access to unauthorized resources. Lockdown browsers like Respondus LockDown Browser or Safe Exam Browser can prevent students from navigating away from the test interface or opening other applications during the exam.

Creating a culture of academic integrity is of the utmost importance. Educators can discourage students from cheating by informing them of the ethical and academic implications of dishonest practices and emphasizing the importance of honest assessment. Investing in robust security measures, such as remote proctoring services and secure networks, can ensure the integrity of the assessment while still providing the benefits of BYOD. For non-educational exams such as certifications, the use of a non-disclosure agreement (NDA) or similar legal document is essential. For example, this can include language that the candidate will be taking the test on their own device, but will not attempt to cheat or steal content, or could face consequences such as being barred from taking the test again for one year.

Implementation Strategies

Successful implementation of BYOD requires careful planning and execution. Institutions should begin by providing clear guidelines that define acceptable use of personal devices and the responsibilities assigned to students. Specific facets include specifying which devices are permitted and what applications may be used during the assessment.

Training should be provided to staff to ensure they understand the permissions and restrictions and can provide a secure environment for students. Educators should be familiar with the technology used in assessments to troubleshoot issues and prevent security breaches. For example, teachers can be trained to use monitoring software that alerts them if a student tries to access unauthorized resources during an exam.

Considerations should also be given to those who do not own a personal device. Institutions might provide loaned devices or partner with organizations to supply devices, ensuring all students have equal access. For instance, a school could establish a device loan program where students can borrow laptops for the duration of the assessment.

Regular evaluations of the BYOD policy will allow for accommodations for new technologies and evolving student needs, ensuring the policy remains effective and beneficial for all involved parties.

Security Checklist for Implementing BYOD in Assessment

  1. Establish Clear Policies and Guidelines

    • Define acceptable devices and software.

    • Outline prohibited applications and websites.

    • Communicate consequences for policy violations.

  2. Use Secure Assessment Platforms

    • Implement lockdown browsers to prevent access to unauthorized resources.

    • Ensure compatibility with various devices and operating systems.

    • Regularly update platforms to address security vulnerabilities.

  3. Authenticate Student Identity

    • Require secure logins for assessment access.

    • Consider biometric verification for high-stakes exams.

  4. Monitor Assessment Environments

    • Utilize proctoring software to oversee student activity.

    • Train staff to recognize and address suspicious behaviors.

  5. Provide Technical Support

    • Offer assistance before and during assessments.

    • Prepare for common technical issues related to diverse devices.

  6. Promote Academic Integrity

    • Educate students on the importance of honest practices.

    • Include honor codes or integrity pledges in assessments.

  7. Ensure Equity of Access

    • Provide devices for students without personal ones.

    • Offer alternative assessment methods if necessary.

  8. Regularly Review and Update Security Measures

    • Stay informed about emerging security threats.

    • Adjust policies and technologies accordingly.

  9. Secure Network Infrastructure

    • Use encrypted networks during assessments.

    • Restrict network access to authorized users.

  10. Data Privacy Compliance

    • Adhere to laws and regulations regarding student data.

    • Ensure all platforms used comply with privacy standards.

Conclusion

The BYOD approach to assessments offers exciting opportunities to improve student performance but also introduces challenges related to security and implementation. In a world of constant technological innovation, institutions must balance leveraging technology to enhance learning while maintaining the integrity of assessments. Addressing these concerns through effective strategies can allow institutions to benefit from a more dynamic and responsive assessment environment provided by BYOD.

/by
blooms taxonomy 1

Bloom’s Taxonomy and Cognitive Levels in Assessment: A Key to Effective Testing

Fernando Austria CorralesOctober 30, 2024

Bloom’s Taxonomy is a hierarchical classification of cognitive levels ranging from lower to higher order thinking, which provides a valuable framework for test development. The development of effective assessments is a cornerstone of educational practice, essential for measuring student achievement and informing instructional decisions, and effective use of Bloom’s Taxonomy can improve the validity of assessments. 

Why use Bloom’s Taxonomy in Assessment?

By integrating Bloom’s Taxonomy into processes like test blueprint and item design, educators can create assessments that not only evaluate basic knowledge but also foster critical thinking, problem solving, and the application of concepts in new contexts (generalization). 

Utilizing Bloom’s Taxonomy in test blueprints involves aligning learning objectives with their corresponding cognitive levels (remembering, understanding, applying, analyzing, evaluating, creating). A test blueprint serves as a strategic plan that outlines the distribution of test items across various content areas and cognitive skills. By mapping each learning objective to a specific level of Bloom’s Taxonomy educators ensure a balanced assessment that reflects the intended curriculum emphasis. This alignment guarantees that the test measures a range of abilities, from factual recall (remembering) to complex analysis and synthesis (analyzing – evaluating). 

In item design, Bloom’s Taxonomy guides the creation of test questions that target specific cognitive levels. For example, questions aimed at the “understanding” level might ask students to generate a paraphrase from a given passage, while those at the “applying” level could present real-world scenarios requiring the used of the learned principles (e.g. Calculating the perimeter of a rectangle to know how many meters of fencing to buy). Higher – order questions at the “analyzing”, “evaluating” or “creating” levels, challenge students to distinguish between arguments, critique methodologies, or design original solutions. This deliberate crafting of items ensures that assessments are not disproportionately focused on lower order skills, but promote deeper cognitive engagement. 

Moreover, incorporating Bloom’s Taxonomy into test development enhances the validity and reliability of assessments and aids in identifying specific areas where students may struggle, allowing for targeted instructional interventions. By fostering a comprehensive evaluation of both foundational knowledge and advanced thinking skills, Bloom’s Taxonomy contributes to more meaningful assessments that support student growth, achievement, certification among other types of assessments.

Bloom’s Taxonomy is an important tool in developing educational assessments with validity, by targeting the content to the appropriate complexity for the target population. In the world of psychometrics and assessments, understanding cognitive levels is essential for creating effective exams that accurately measure a candidate’s knowledge, skills, and abilities. Cognitive levels, often referred to as levels of cognition, are typically categorized into a hierarchy that reflects how deeply an individual understands and processes information. This concept is foundational in education and testing, including professional certification exams, where assessing not just knowledge but how it is applied is critical.

One of the most widely recognized frameworks for cognitive levels is Bloom’s Taxonomy, developed by educational psychologist Benjamin Bloom in the 1950s. Bloom’s Taxonomy classifies cognitive abilities into six levels, ranging from basic recall of facts to more complex evaluation and creation of new knowledge. For exam creators, this taxonomy is valuable because it helps design assessments that challenge different levels of thinking.

Here’s an overview of the cognitive levels in Bloom’s Taxonomy, with examples relevant to assessment design.

1. Remembering

This is the most basic level of cognition and involves recalling facts or information. In the context of an exam, this could mean asking candidates to memorize specific definitions or procedures.

  • Example: “Define the term ‘psychometrics.’”
  • Exam Insight: While important, questions that assess only the ability to remember facts may not provide a full picture of a candidate’s competence. They’re often used in conjunction with higher-level questions to ensure foundational knowledge.

2. Understanding

The next level involves comprehension—being able to explain ideas or concepts. Rather than simply recalling information, the test-taker demonstrates an understanding of what the information means.

  • Example: “Explain the difference between formative and summative assessments.”
  • Exam Insight: Understanding questions help gauge whether a candidate can interpret concepts correctly, which is essential for fields like psychometrics, where understanding testing methods and principles is key.

3. Applying

Application involves using information in new situations. This level goes beyond understanding by asking candidates to apply their knowledge in a practical context.

  • Example: “Given a set of psychometric data, identify the most appropriate statistical method to analyze test reliability.”
  • Exam Insight: This level of cognition is crucial in high-stakes exams, especially in certification contexts where candidates must demonstrate their ability to apply theoretical knowledge to real-world scenarios.

4. Analyzing

At this level, candidates are expected to break down information into components and explore relationships among the parts. Analysis questions often require deeper thinking and problem-solving skills.

  • Example: “Analyze the factors that could lead to bias in a psychometric assessment.”
  • Exam Insight: Analytical skills are critical for assessing a candidate’s ability to think critically about complex issues, which is essential in roles like test development or evaluation in assessment ecosystems.

5. Evaluating

Evaluation involves making judgments about the value of ideas or materials based on criteria and standards. This might include comparing different solutions or assessing the effectiveness of a particular approach.

  • Example: “Evaluate the effectiveness of different psychometric models for ensuring the fairness of certification exams.”
  • Exam Insight: Evaluation questions are typically found in advanced assessments, where candidates are expected to critique processes and propose improvements. This level is vital for ensuring that individuals in leadership roles can make informed decisions about the tools they use.

6. Creating

The highest cognitive level is creation, where candidates generate new ideas, products, or solutions based on their knowledge and analysis. This level requires innovative thinking and often asks for the synthesis of information. 

  • Example: “Design an assessment framework that incorporates both traditional and modern psychometric theories.”
  • Exam Insight: Creating-level questions are rare in most standardized tests but may be used in specialized certifications where innovation and leadership are critical. This level of cognition assesses whether the candidate can go beyond existing knowledge and contribute to the field in new and meaningful ways.

Bloom’s Taxonomy and Cognitive Levels in High-Stakes Exams

When designing high-stakes exams—such as those used for professional certifications or employment tests—it’s important to strike a balance between assessing lower and higher cognitive levels. While remembering and understanding provide insight into the candidate’s foundational knowledge, analyzing and evaluating help gauge their ability to think critically and apply knowledge in practical scenarios.

For example, a psychometric exam for certifying test developers might include questions across all cognitive levels:

  • Remembering: Questions that assess knowledge of psychometric principles; basic definitions, processes, and so on.
  • Understanding: Questions that ask for an explanation of item response theory.
  • Applying: Scenarios where candidates must apply these theories to improve test design.
  • Analyzing: Situations where candidates analyze a poorly performing test item.
  • Evaluating: Questions that ask candidates to critique the use of certain assessment methods.
  • Creating: Tasks where candidates design new assessment tools.

 

Bloom’s Taxonomy and Learning Psychometrics: An applied example for test blueprint development.  

Developing a test blueprint using Bloom’s Taxonomy ensures that the assessments in psychometrics effectively measures a range of cognitive skills. Here is an example of how you can apply Bloom’s Taxonomy to create a comprehensive test blueprint for a course in psychometrics. 

Step 1. Identify content strands and their cognitive demands 

Let’s consider the following strands and their corresponding cognitive demand:

Content Strand Cognitive Demand
Foundations of Psychometrics Remembering: Recall basic definitions and concepts in psychometrics.

Understanding: Explain fundamental principles and their significance.  
Classical Test Theory Understanding: Describe components of CTT.

Applying: Use CTT formulas to compute test scores. 

Analyzing: Interpret the implications of test scores and error. 
Item Response Theory Understanding: Explain the basics of IRT models. 

Applying: Apply IRT to analyze test items.

Analyzing: Compare IRT with CTT in terms of advantages and limitations. 
Reliability and Validity  Understanding: Define different types of reliability and validity. 

Evaluating: Assess the reliability and validity of given tests. 

Analyzing: Identify factors affecting reliability and validity.
Test development and standardization  Applying: Develop test items following psychometric principles. 

Creating: Design a basic blueprint. 

Evaluating: Critique test items for bias and fairness. 

 

Step 2. Use Bloom’s Taxonomy to elaborate a test blueprint

Using Bloom’s Taxonomy, we can create a test blueprint that aligns the content strands with appropriate cognitive levels.

This would be test blueprint table:

Content strand Bloom’s Level Learning objective # items Item type
Foundational of Psychometrics Remembering Recall definitions and concepts 5 Multiple-choice
Understanding  Explain principles and their significance 4 Short-answer
Classical Test Theory Understanding Describe components of CTT 3 True/False with explanation
Applying  Compute test scores using CTT formulas 5 Calculation problems
Analyzing Interpret test scores and error implications 4 Data analysis questions
Item Response Theory  Understanding Explain basics of IRT models 3 Matching
Applying Analyze test items using IRT 4 Problem solving 
Analyzing  Compare IRT and CTT 3 Comparative essays
Reliability and Validity Understanding  Define types of reliability and validity 4 Fill-in-the-blank
Analyzing  Identify factors affecting reliability and validity 4 Case studies
Evaluating Assess the reliability and validity of tests 5 Critical evaluations
Test development and standardization Applying  Develop test items using psychometric principles 4 Item writing exercises 
Creating Design a basic test blueprint 2 Project-based tasks
Evaluating  Critique test items for bias and fairness 3 Peer review assignments

 

By first delineating the content strands and their associated cognitive demands, and then applying Bloom’s Taxonomy, educators can develop a test blueprint that is both systematic and effective. This method ensures that assessments are comprehensive, balanced, and aligned with educational goals, ultimately enhancing the measurement of student learning in psychometrics.

Benefits of this approach include:

  • Comprehensive Coverage: Ensures all important content areas and cognitive skills are assessed.
  • Balanced Difficulty: Provides a range of item difficulties to discriminate between different levels of student performance.
  • Enhanced Validity: Aligns assessment with learning objectives, improving content validity.
  • Promotes Higher-Order Thinking: Encourages students to engage in complex cognitive processes.

 

Conclusion: Why Cognitive Levels Matter in Assessments

Cognitive levels play a crucial role in assessment design. By incorporating questions that target different levels of cognition, exams can provide a more complete picture of a candidate’s abilities. 

By aligning exam content with cognitive levels, you ensure that your assessments are not just measuring rote memorization but the full spectrum of cognitive abilities—from basic understanding to advanced problem-solving and creativity. This creates a more meaningful and comprehensive evaluation process for both candidates and employers alike.

 

 

References:

 

/by
assessment-technology-improve-exams

What is Psychometrics? How does it improve assessment?

Nathan Thompson, PhDOctober 12, 2024

Psychometrics is the science of educational and psychological assessment, using data to ensure that tests are fair and accurate.  Ever felt like you took a test which was unfair, too hard, didn’t cover the right topics, or was full of questions that were simply confusing or poorly written?  Psychometricians are the people who help organizations fix these things using data science, as well as more advanced topics like how to design an AI algorithm that adapts to each examinee.

Psychometrics is a critical aspect of many fields.  Having accurate information on people is essential to education, human resources, workforce development, corporate training, professional certifications/licensure, medicine, and more.  It scientifically studies how tests are designed, developed, delivered, validated, and scored.

Key Takeaways on Psychometrics

  • Psychometrics is the study of how to measure and assess mental constructs, such as intelligence, personality, or knowledge of accounting law
  • Psychometrics is NOT just screening tests for jobs
  • Psychometrics is dedicated to making tests more accurate and fair
  • Psychometrics is heavily reliant on data analysis and machine learning, such as item response theory

What is Psychometrics? Definition & Meaning

Psychometrician Qualities

Psychometrics is the study of assessment itself, regardless of what type of test is under consideration. In fact, many psychometricians don’t even work on a particular test, they just work on psychometrics itself, such as new methods of data analysis.  Most professionals don’t care about what the test is measuring, and will often switch to new jobs at completely unrelated topics, such as moving from a K-12 testing company to psychological measurement to an Accountant certification exam.  We often refer to whatever we are measuring simply as “theta” – a term from item response theory.

Psychometrics tackles fundamental questions around assessment, such as how to determine if a test is reliable or if a question is of good quality, as well as much more complex questions like how to ensure that a score today on a university admissions exam means the same thing as it did 10 years ago.  Additionally, it examines phenomena like the positive manifold, where different cognitive abilities tend to be positively correlated, supporting the consistency and generalizability of test scores over time.

Psychometrics is a branch of data science.  In fact, it’s been around a long time before that term was even a buzzword.  Don’t believe me?  Check out this Coursera course on Data Science, and the first example they give as one of the foundational historical projects in data science is… psychometrics!  (early research on factor analysis of intelligence).

Even though assessment is everywhere and Psychometrics is an essential aspect of assessment, to most people it remains a black box, and professionals are referred to as “psychomagicians” in jest. However, a basic understanding is important for anyone working in the testing industry, especially those developing or selling tests.

Psychometrics is NOT limited to very narrow types of assessment.  Some people use the term interchangeably with concepts like IQ testing, personality assessment, or pre-employment testing.  These are each but tiny parts of the field!  Also, it is not the administration of a test.

 

Why do we need Psychometrics?

This purpose of tests is providing useful information about people, such as whether to hire them, certify them in a profession, or determine what to teach them next in school.  Better tests mean better decisions.  Why?  The scientific evidence is overwhelming that tests provide better information for decision makers than many other types of information, such as interviews, resumes, or educational attainment.  Thus, tests serve an extremely useful role in our society.

The goal of psychometrics is to provide validity: evidence to support that interpretations of scores from the test are what we intended.  If a certification test is supposed to mean that someone passing it meets the minimum standard to work in a certain job, we need a lot of evidence about that, especially since the test is so high stakes in that case.  Meta-analysis, a key tool in psychometrics, aggregates research findings across studies to provide robust evidence on the reliability and validity of tests. By synthesizing data from multiple studies, meta-analysis strengthens the validity claims of tests, especially crucial in high-stakes certification exams where accuracy and fairness are paramount.

 

What does Psychometrics do?

Building and maintaining a high-quality test is not easy.  A lot of big issues can arise.  Much of the field revolves around solving major questions about tests: what should they cover, what is a good question, how do we set a good cutscore, how do we make sure that the test predicts job performance or student success, etc.  Many of these questions align with the test development cycle – more on that later.

How do we define what should be covered by the test? (Test Design)

test dev cycle

Before writing any items, you need to define very specifically what will be on the test.  If the test is in credentialing or pre-employment, psychometricians typically run a job analysis study to form a quantitative, scientific basis for the test blueprints.  A job analysis is necessary for a certification program to get accredited.  In Education, the test coverage is often defined by the curriculum.

How do we ensure the questions are good quality? (Item Writing)

There is a corpus of scientific literature on how to develop test items that accurately measure whatever you are trying to measure.  A great overview is the book by Haladyna.  This is not just limited to multiple-choice items, although that approach remains popular.  Psychometricians leverage their knowledge of best practices to guide the item authoring and review process in a way that the result is highly defensible test content.  Professional item banking software provides the most efficient way to develop high-quality content and publish multiple test forms, as well as store important historical information like item statistics.

How do we set a defensible cutscore? (Standard Setting)

Test scores are often used to classify candidates into groups, such as pass/fail (Certification/Licensure), hire/non-hire (Pre-Employment), and below-basic/basic/proficient/advanced (Education).  Psychometricians lead studies to determine the cutscores, using methodologies such as Angoff, Beuk, Contrasting-Groups, and Borderline.

How do we analyze results to improve the exam? (Psychometric Analysis)

Psychometricians are essential for this step, as the statistical analyses can be quite complex.  Smaller testing organizations typically utilize classical test theory, which is based on simple mathematics like proportions and correlations.  Large, high-profile organizations typically use item response theory (IRT), which is based on a type of nonlinear regression analysis.  Psychometricians evaluate overall reliability of the test, difficulty and discrimination of each item, distractor analysis, possible bias, multidimensionality, linking multiple test forms/years, and much more.  Software such as  Iteman  and  Xcalibre  is also available for organizations with enough expertise to run statistical analyses internally.  Scroll down below for examples.

How do we compare scores across groups or years? (Equating)

This is referred to as linking and equating.  There are some psychometricians that devote their entire career to this topic.  If you are working on a certification exam, for example, you want to make sure that the passing standard is the same this year as last year.  If you passed 76% last year and this year you passed 25%, not only will the candidates be angry, but there will be much less confidence in the meaning of the credential.

How do we know the test is measuring what it should? (Validity)

Validity is the evidence provided to support score interpretations.  For example, we might interpret scores on a test to reflect knowledge of English, and we need to provide documentation and research supporting this.  There are several ways to provide this evidence.  A straightforward approach is to establish content-related evidence, which includes the test definition, blueprints, and item authoring/review.  In some situations, criterion-related evidence is important, which directly correlates test scores to another variable of interest.  Delivering tests in a secure manner is also essential for validity.

 

Where is Psychometrics Used?

Certification/Licensure/Credentialing

In certification testing, psychometricians develop the test via a documented chain of evidence following a sequence of research outlined by accreditation bodies, typically: job analysis, test blueprints, item writing and review, cutscore study, and statistical analysis.  Web-based item banking software like  FastTest  is typically useful because the exam committee often consists of experts located across the country or even throughout the world; they can then easily log in from anywhere and collaborate.

Pre-Employment

In pre-employment testing, validity evidence relies primarily on establishing appropriate content (a test on PHP programming for a PHP programming job) and the correlation of test scores with an important criterion like job performance ratings (shows that the test predicts good job performance).  Adaptive tests are becoming much more common in pre-employment testing because they provide several benefits, the most important of which is cutting test time by 50% – a big deal for large corporations that test a million applicants each year. Adaptive testing is based on item response theory, and requires a specialized psychometrician as well as specially designed software like  FastTest.

K-12 Education

Most assessments in education fall into one of two categories: lower-stakes formative assessment in classrooms, and higher-stakes summative assessments like year-end exams.  Psychometrics is essential for establishing the reliability and validity of higher-stakes exams, and on equating the scores across different years.  They are also important for formative assessments, which are moving towards adaptive formats because of the 50% reduction in test time, meaning that student spend less time testing and more time learning.

Universities

Universities typically do not give much thought to psychometrics even though a significant amount of testing occurs in higher education, especially with the move to online learning and MOOCs.  Given that many of the exams are high stakes (consider a certificate exam after completing a year-long graduate program!), psychometricians should be used in the establishment of legally defensible cutscores and in statistical analysis to ensure reliable tests, and professionally designed assessment systems used for developing and delivering tests, especially with enhanced security.

Medicine/Psychology

Have you ever taken a survey at your doctor’s office, or before/after a surgery?  Perhaps a depression or anxiety inventory at a psychotherapist?  Psychometricians have worked on these.

 

The Test Development Cycle

Psychometrics is the core of the test development cycle, which is the process of developing a strong exam.  It is sometimes called similar names like assessment lifecycle.

test development cycle job task analysis psychometrics

You will recognize some of the terms from the introduction earlier.  What we are trying to demonstrate here is that those questions are not standalone topics, or something you do once and simply file a report.  An exam is usually a living thing.  Organizations will often be republishing a new version every year or 6 months, which means that much of the cycle is repeated on that timeline.  Not all of it is; for example, many orgs only do a job analysis and standard setting every 5 years.

Consider a certification exam in healthcare.  The profession does not change quickly because things like anatomy never change and medical procedures rarely change (e.g., how to measure blood pressure).  So, every 5 years it does a job analysis of its certificants to see what they are doing and what is important.  This is then converted to test blueprints.  Items are re-mapped if needed, but most likely do not need it because there are probably only minor changes to the blueprints.  Then a new cutscore is set with the modified-Angoff method, and the test is delivered this year.  It is delivered again next year, but equated to this year rather than starting again.  However, the item statistics are still analyzed, which leads to a new cycle of revising items and publishing a new form for next year.

 

Example of Psychometrics in Action

Here is some output from our Iteman software.  This is deeply analyzing a single question on English vocabulary, to see if the student knows the word alleviate.  About 70% of the students answered correctly, with a very strong point-biserial.  The distractor P values were all in the minority and the distractor point-biserials were negative, which adds evidence to the validity.  The graph shows that the line for the correct answer is going up while the others are going down, which is good.  If you are familiar with item response theory, you’ll notice how the blue line is similar to an item response function.  That is not a coincidence.

FastTest Iteman Psychometrics Analysis

Now, let’s look at another one, which is more interesting.  Here’s a vocab question about the word confectioner.  Note that only 37% of the students get it right… even though there is a 25% chance just of guessing!!!  However, the point-biserial discrimination remains very strong at 0.49.  That means it is a really good item.  It’s just hard, which means it does a great job to differentiate amongst the top students.

Confectioner confetti

A Glossary of Psychometric Terms

Accreditation: Accreditation by an outside agency affirms that an organization has met a certain level of standards. Certification testing programs may become accredited by meeting specified standards in test development, psychometrics, bylaws, management, etc.  Learn more.

Adaptive Test: A test that is delivered with an AI-based algorithm that personalizes it to each examinee, thereby making it much more secure and accurate while decreasing test length. Learn more.

Achievement: The psychometric term for measuring something that a student has learned, such as 9th grade biology curriculum knowledge, rather than an innate construct such as intelligence of conscientiousness.

Aptitude: A construct that is measured which is innate, usually in a cognitive context.  For example, logical reasoning ability.

Biserial Correlation: A classical index of item discrimination, highly similar to the more commonly used point-biserial. The biserial correlation assumes that the item scores and test scores reflect an underlying normal distribution, which is not always the case

Blueprint: A test blueprint, or test specification, details how an exam is to be constructed. It includes important information, such as the total number of items, the number of items in each content area or domain, the number of items that are recall verses reasoning, and the item formats to be utilized.

Certification: A non-mandatory testing program that certifies the candidates have achieved a minimum standard or knowledge or performance.

Classical Test Theory (CTT): A psychometric analysis and test development paradigm based on correlations, proportions, and other statistics that are relatively simple compared to IRT. It is, therefore, more appropriate for smaller samples, especially for fewer than 100.

Classification: The use of tests for classifying candidates into categories, such as pass/fail, nonmaster/master, or basic/proficient/advanced.

Cognitive Diagnostic Models (CDMs) aka Diagnostic Measurement Models (DMMs): A relatively new psychometric paradigm that frames the measurement problem not as one latent trait, but rather individual skills that must be mastered.  So rather than 4th grade match achievement as a scale, there are locations for adding fractions, dividing fractions, multiplying decimals, etc.  Can be used in concert with IRT.  Learn more.

Computerized Adaptive Testing (CAT): A dynamic method of test administration where items are selected one at a time to match item difficulty and candidate ability as closely as possible. This helps prevent candidates from being presented with items that are too difficult or too easy for them, which has multiple benefits. Often, the test only takes half as many items to obtain a similar level of accuracy to form-based tests. This reduces the testing time per examinee and also reduces the total number of times an item is exposed, as well as increasing security by the fact that nearly every candidate will receive a different set of items.

Computerized Classification Testing (CCT): An approach similar to CAT, but with different algorithms to reflect the fact that the purpose of the test is only to make a broad classification and not obtain a highly accurate point estimate of ability.

Concurrent Validity: An aspect of validity (see below) that correlates a test to other variables at the same time, to which we hope it correlates.  A university admissions test should correlate with high school grade point average – but not perfectly, since they are not exactly the same construct, and then what is the point of having the test?

Cutscore: Also known as a passing score, the cutscore is the score that a candidate must achieve to obtain a certain classification, such as “pass” on a licensure or certification exam.

Criterion-Referenced: A test score (not a test) is criterion-referenced if it is interpreted with regard to a specified criterion and not compared to scores of other candidates. For instance, providing the number-correct score does not relate any information regarding a candidate’s relative standing.

Differential item functioning: A specific type of analysis that evaluates whether an item is biased towards a subgroup.  This is different than overall test bias.

Distractors: Distractors are the incorrect options of a multiple-choice item. A distractor analysis is an important part of psychometric review, as it helps determine if one is acting as a keyed response.  Learn more.

Equating: A psychometric term for the process of determining comparable scores on different forms of an examination. For example, if Form A is more difficult than Form B, it might be desirable to adjust scores on Form A upward for the purposes of comparing them to scores on Form B. Usually, this is done statistically based on items that are on both forms, which are called equator, anchor, or common items. Because the groups who took the two forms are different, this is called a common items non-equivalent groups design.

Factor Analysis: An approach to analyzing complex data that seeks to break it down into major components or factors.  Use in many fields nowadays, but originally developed for psychometrics.  Two of the most common examples are the extensive research which finds that personality items/measures boil down to the Big Five, and that intelligence items/measures boil down to general cognitive ability (though there is evidence of different aspects with massive cognitive manifold).

Form: Forms are specific sets of items that are administered together for a test. For example, if a test included a certain set of 100 items this year and a different set of 100 items next year, these would be two distinct forms.

Item: The basic component of a test, often colloquially referred to as a “question,” but items are not necessary phrased as a question. They can be as varied as true/false statements, rating scales, and performance task simulations, in addition to the ubiquitous multiple-choice item.

Item Bank: A repository of items for a testing program, including items at all stages, such as newly written, reviewed, pretested, active, and retired.

Item Banker: A specialized software program that facilitates the maintenance and growth of an item bank by recording item stages, statistics, notes, and other characteristics.

Item Difficulty: A statistical index of how easy/hard the item is with respect to the underlying ability/trait. That is, an item is difficult if not many people get it correct or respond in the keyed direction.

Item Discrimination: A statistical index of the quality of the item, assessing how well it differentiates examinees of high verses low ability. Items with low discrimination are considered poor quality and are candidates to be revised or retired.

Item Response Theory (IRT): A comprehensive approach to psychometric analysis and test development that utilizes complex mathematical models. This provides several benefits, including the ability to design CATs, but requires larger sample sizes. A common rule of thumb is 100 candidates for the one-parameter model and 500 for the three-parameter model.

     a: The item response theory index of item discrimination, analogous to the point-biserial and biserial correlations in classical test theory. It reflects the slope of the item response function. Often ranging from 0.1 to 2.0 in practice, a higher value indicates a better-performing item.

     b: The item response theory index of item difficulty or location, analogous to the P-value (P+) of classical test theory. Typically ranging from -3.0 to 3.0 in practice, a higher value indicates a more difficult item.

     c: The item response theory pseudo-guessing parameter, representing the lower asymptote of the item response function. It is theoretically near the value of 1/k, where k is the number of alternatives. For example, with the typical four-option multiple-choice item, a candidate has a base chance of 25% of guessing the correct answer.

Item Type: Items (test questions) can be a huge range of formats.  We are all familiar with single best answer multiple choice, but there are many others.   Some of these are: multiple response, drag and drop, essay, scored short answer, and equation editor.

Job Analysis: Also known as practice analysis or role delineation study, job analysis is a formal study used to determine the structure of a job and the KSAs important to success or competence. This is then used to establish the test blueprint for a professional testing program, a critical step in the chain of evidence for validity.

Key: The key is the correct response to an item.

KSA: KSA is an acronym for knowledge, skills, and abilities. A critical step in testing for employment or professional credentials is to determine the KSAs that are important in a job. This is often done via a job analysis study.

Licensure: A testing program mandated by a government body. The test must be passed in order to perform the task in question, whether it is to work in the profession or drive a car.

Norm-Referenced: A test score (not a test) is norm-referenced if it is interpreted with regard to the performance of other candidates. Percentile rank is an example of this because it does not provide any information regarding how many items the candidate got correct.

P-value: A classical index of item difficulty, presented as the proportion of candidates who correctly responded to the item. A value above 0.90 indicates an easy item, while a value below 0.50 indicates a relatively difficult item. Note that it is inverted; a higher value indicates less difficulty.

Point-Biserial Correlation: A classical index of item discrimination, calculated as the Pearson correlation between the item score and the total test score. If below 0.0, low-scoring candidates are actually doing better than high-scoring candidates, and the item should be revised or retired. Low positive values are marginal, higher positive values are ideal.

Polytomous: A psychometric term for data where there are 2 or more possible points.  Multiple choice items, while having 3-5 options, are usually still only dichotomous (0/1 points).  Examples of a polytomous item are a Likert-style rating scale (“rate on a scale of 1 to 5”) and partial credit items or rubrics (scoring an essay as 0 to 5 points).

Power Test: A test where the goal is to measure the maximal knowledge, ability, or trait of the examinee.  For example, a medical certification exam with a generous time limit.

Predictive Validity: An aspect of validity (see below) that focuses on how well the test predicts important outcomes.  A university admissions test should predict 4-year graduation probability very well, and a pre-employment test on MS Excel should predict job performance for bookkeepers.

Pretest (or Pilot) Item: An item that is administered to candidates simply for the purposes of obtaining data for future psychometric analysis. The results on this item are not included in the score. It is often prudent to include a small number of pretest items in a test.

Reliability: A psychometric term for the repeat-ability or consistency of the measurement process. Often, this is indexed by a single number, most commonly the internal consistency index coefficient alpha or its dichotomous formulation, KR-20. Under most conditions, these range from 0.0 to 1.0, with 1.0 being a perfectly reliable measurement. However, just because a test is reliable does not mean that it is valid (i.e.,  measures what it is supposed to measure).

Scaling: Scaling is a process of converting scores obtained on an exam to an arbitrary scale. This is done so that all the forms and exams used by a testing organization are on a common scale. For example, suppose an organization had two testing programs, one with 50 items and one with 150 items. All scores could be put on the same scale to standardize score reporting.

Speeded Test: A test where the purpose is to see how fast the examinee can answer questions.  The questions are therefore not usually knowledge based.  For example, seeing how many 5-digit zip codes they can correctly type in 60 seconds.  Learn more.

Standard Error of Measurement: A psychometric term for a concept that quantifies the amount of error in an examinee’s score, since psychometrics is not perfect; even with the best Math test, a student might have variation in their result today vs next week.  The concept differs substantially in classical test theory vs. item response theory.

Standard-Setting Study: A formal study conducted by a testing organization to determine standards for a testing program, which are manifested as a cutscore. Common methods include the Angoff, Bookmark, Contrasting Groups, and Borderline Survey methods.

Subject Matter Expert (SME): An extremely knowledgeable person within the test development process. SMEs are necessary to write items, review items, participate in standard-setting studies, and job analyses, and oversee the testing program to ensure its fidelity to its true intent.

Validity: Validity is the concept that test scores can be interpreted as intended. For example, a test for certification in a profession should reflect basic knowledge of that profession, and not intelligence or other constructs, and scores can, therefore, be interpreted as evidencing professional competence. Validity must be formally established and maintained by empirical studies as well as sound psychometric and test development practices.  Learn more.

Psychometrics looks fun!  How can I join the band?

You will need a graduate degree.  I recommend you look at the NCME website (ncme.org) with resources for students.  Good luck!

Already have a degree and looking for a job?  Here’s the two sites that I recommend:

  • NCME – Also has a job listings page that is really good (ncme.org)
  • Horizon Search – Headhunter for Psychometricians and I/O Psychologists
/by
Learning Management System Avatar

What is a Learning Management System (LMS)?

Fernando Austria CorralesOctober 5, 2024

In today’s digital-first world, educational institutions and organizations are leveraging technology to deliver training and instruction in more dynamic and efficient ways. A core component of this shift is the Learning Management System (LMS). According to this website, the global LMS market reflects the following growing adoption: valued at $16.1 billion in 2022, it reached $24.05 billion in 2024 and is projected to expand further to $61.8 billion by 2032. This growth corresponds to a robust compound annual growth rate (CAGR) of 14.8% over the forecast period (2024–2032). But what exactly is an LMS, and why is it so critical to modern education and training? Let’s explore this transformative technology and its key features.

Understanding the Basics: What is a Learning Management System?

LMS is a software application or platform used to plan, implement, and assess a specific learning process. It provides educators, administrators, and learners with a single location for communication, course material, and assessment tools. LMS platforms are commonly used in schools, universities, corporate training programs, and online learning environments. LMS have faced a massive growth in usage due to the emphasis on remote learning during the COVID-19 pandemic. 

The core function of an LMS is to make educational content accessible to users anytime, anywhere, and often at their own pace. This flexibility is crucial in accommodating the diverse needs of learners and organizations.

Key Features of a Learning Management System

Learning Management Systems are designed to simplify the process of delivering training and educational content. Here are some of the primary features that make LMS platforms so valuable:

LMS - Connect

  1. Course Management: Create, organize, and manage courses with ease. This feature often includes the ability to upload different types of content, such as videos, presentations, PDFs, and quizzes.
  2. Assessment and Tracking: LMS allows for automated assessments and grading. It can track progress, monitor engagement, and provide insights through data analytics.
  3. User Management: Manage user roles and permissions to control access to different parts of the platform. Instructors, administrators, and learners each have unique permissions and access.
  4. Communication Tools: Many LMS platforms include integrated messaging, discussion forums, and video conferencing, fostering communication between learners and educators.
  5. Learning Analytics: LMS often incorporates dashboards to track student progress and performance. LMS can report key items like: completion rates and success likelihood. Administrators, educators and learners can use these metrics to better understand gaps in knowledge.

Examples of Popular Learning Management System Platforms

LMS - Modules

There are hundreds of LMS platforms available on the market, catering to various educational and corporate needs. The options range from open-source platforms like Moodle and Chamilo, which offer extensive customization but require technical expertise, to commercial solutions such as Blackboard and Canvas, known for their robust feature sets and support services. Pricing can vary significantly based on factors like the number of users, features, and deployment options.

Some platforms, like Google Classroom, are free for qualifying institutions. There are three paid plans. First, the Google Workspace for Education Standard plan costs $3 per student, per year and adds on a security center, advanced device and app management features, Gmail and Classroom logs for export into BigQuery, and audit logs. Then there’s the Teaching and Learning Upgrade plan that costs $4 per license, per month and includes additional features like advanced Google Meet features, unlimited originality reports and the ability to check for peer matches across a private repository. Finally, the Google Workspace for Education Plus plan costs $5 per student, per year and includes all of the features of the other plans, plus live streams with up to 100,000 in-domain viewers, syncing rosters from SISs to Google Classroom, personalized cloud search and prioritized support (Better Buys, 2023).

It’s essential to evaluate your needs and budget before choosing an LMS, as costs can quickly escalate with additional modules and support services.

Below are some widely used options:

  • Moodle: An open-source platform favored by educational institutions due to its flexibility and community support. Moodle is highly customizable and can be tailored to meet specific learning needs.

LMS - Moodle

  • Canvas: A popular choice for both K-12 and higher education, Canvas offers a clean interface and extensive integrations with third-party tools, making it ideal for tech-savvy institutions.

LMS - Canvas

  • Blackboard: Widely adopted by universities and colleges, Blackboard focuses on providing comprehensive features for large-scale educational organizations.

LMS - Blackboard

  • Google Classroom: A simple and intuitive tool, Google Classroom is popular in K-12 settings. It integrates seamlessly with other Google products, making it a convenient option for schools already using Google Workspace.

LMS - Google Classroom

When implementing an LMS, there are several additional expenses to consider beyond the platform’s base pricing. These include:

  1. Implementation and Setup Costs: Depending on the complexity of the LMS and your organization’s specific requirements, there may be initial setup costs. This could involve customizing the platform, integrating it with existing systems, and migrating existing content and user data.
  2. Training and Support: It’s crucial to allocate a budget for training administrators, instructors, and learners to use the LMS effectively. Some platforms offer onboarding and support as part of their package, while others charge separately for these services.
  3. Content Creation and Licensing: Developing new courses, multimedia content, or interactive assessments can be time-consuming and expensive. Additionally, if you’re using third-party content or e-learning modules, you may need to pay licensing fees.
  4. Maintenance and Upgrades: Keeping the LMS up-to-date with software patches, security updates, and new feature releases often incurs ongoing costs. Organizations that opt for self-hosted solutions will also need to consider server maintenance and IT support costs.
  5. Integration with Other Tools: If you plan to integrate the LMS with other systems like HR software, CRM platforms, or data analytics tools, there may be costs associated with custom integrations or purchasing additional licenses for these tools.
  6. Compliance and Security: Ensuring that your LMS complies with regulations (e.g., GDPR, ADA) may involve additional expenses for compliance assessments, legal consultations, and security enhancements.
  7. Scalability: If your organization grows, you might need to expand your LMS capacity, which could mean upgrading your plan, adding new features, or expanding server capacity—all of which can increase costs.

By considering these additional expenses, organizations can develop a more accurate budget and avoid unexpected costs during the LMS implementation process.

Why Your Organization Needs a Learning Management System

Whether you’re running a university, a corporate training program, or a small online course, an LMS can streamline your educational process. With the ability to host and organize content, track learner progress, and provide insights through analytics, an LMS offers much more than just a place to upload learning materials. It can be a strategic tool to enhance the learning experience, increase engagement, and ensure that your educational objectives are met.

Advantages of Using a Learning Management System

Learning Management Systems have become a cornerstone for modern education and corporate training environments. Here are six key benefits that define the value and effectiveness of an LMS.

  1. Interoperability: Seamless Integration Across Systems

One of the most significant advantages of an LMS is its ability to integrate seamlessly with other systems through standardized data formats and protocols. LMS platforms adhere to standards such as SCORM (Sharable Content Object Reference Model), xAPI (Experience API), and LTI (Learning Tools Interoperability), which enable the exchange of content and data between different applications. This level of interoperability simplifies the process of sharing resources and tracking learner progress across multiple platforms, ensuring a cohesive learning experience.

  1. Accessibility: Inclusive Learning for All Students

Accessibility is a critical factor in modern education, and LMS platforms are designed to support students with diverse needs, including those with disabilities. Most LMS platforms adhere to accessibility standards like the Web Content Accessibility Guidelines (WCAG), providing features such as screen reader support, keyboard navigation, and closed captioning for videos. Consistent layouts and interfaces make it easier for all users to navigate the platform and access content. By fostering an inclusive environment, an LMS can help organizations comply with legal requirements such as the Americans with Disabilities Act (ADA) and ensure that learning opportunities are available to everyone, regardless of physical or cognitive limitations.

  1. Reusability: Maximizing the Value of Educational Content

Reusability is a key strength of LMS platforms, enabling organizations to develop educational content once and reuse it across different courses, training programs, or departments. This feature significantly reduces the time and costs associated with creating new content for each learning module. Content created within an LMS can be structured into reusable learning objects that can be easily updated, repurposed, and shared. This flexibility is especially valuable for large organizations and educational institutions looking to standardize training materials and curricula while keeping them up-to-date with minimal effort.

  1. Durability: A Sustainable Solution for Long-Term Growth

As technology continues to transform education and training, the LMS market is poised for significant growth. Reports suggest that the global LMS market is expected to achieve a compound annual growth rate (CAGR) of 17.1% by 2028 (Reports, Valuates, 2022). This growth is driven by the increasing demand for flexible learning solutions, remote training, and the incorporation of new technologies like artificial intelligence and virtual reality into the learning process. By choosing a durable and scalable LMS, organizations can ensure that their investment remains relevant and adaptable to future educational trends and technologies.

  1. Maintainability: Ensuring a Continuously Evolving Platform

LMS platforms are designed with maintainability in mind, allowing developers to make updates, add new features, and fix bugs without disrupting the user experience. This is crucial in a rapidly changing educational landscape where learner needs and technological standards are constantly evolving. With cloud-based LMS platforms, maintenance is often handled automatically by the provider, ensuring that the system is always up-to-date with the latest security patches and performance optimizations. This continuous improvement cycle enables organizations to keep their learning environments modern, secure, and aligned with user expectations.

  1. Adaptability: Evolving with the Needs of Learners

Since their inception in the 1990s, LMS platforms have evolved significantly to keep up with changing societal needs and educational practices. Modern LMS platforms are highly adaptable, supporting a wide range of learning methodologies, such as blended learning, flipped classrooms, and competency-based learning. They also offer extensive customization options, allowing organizations to tailor the platform’s look and feel to match their branding and pedagogical approaches. As educational trends and technologies continue to evolve, LMS platforms are equipped to integrate emerging tools and approaches, such as gamification, microlearning, and artificial intelligence-driven personalized learning paths, making them a future-proof solution for delivering high-quality education and training.

By understanding these key advantages, organizations and institutions can leverage LMS platforms to create impactful learning experiences that not only meet current needs but are also prepared for the future of education and training.

Weaknesses of Using a Learning Management System

While Learning Management Systems offer many benefits, there are some limitations to be aware of, especially in specific contexts where advanced features are needed. Here are three key weaknesses to consider:

  1. Limited Functionality for Assessments
    Many LMS platforms lack sophisticated assessment tools. While most systems support basic quizzes and exams, they may not include advanced features like item banking, Item Response Theory (IRT), or adaptive testing capabilities. This limits their use for institutions or organizations looking to implement more complex testing methodologies, such as those used in standardized assessments or psychometric evaluations. In such cases, additional software or integrations with specialized assessment platforms may be required.
  2. Ineffective Student Management
    An LMS is not designed to function as a full-fledged Student Management System (SMS). It typically lacks the robust database management features necessary for handling complex student records, attendance tracking, and detailed progress reporting. This limitation means that many organizations must integrate the LMS with a separate SMS or a Customer Relationship Management (CRM) system to gain comprehensive student management capabilities. Without these integrations, tracking student progress and managing enrollment data can become cumbersome.
  3. Lack of e-Commerce Functionality
    Not all LMS platforms include built-in e-Commerce capabilities, making it difficult to monetize courses directly within the system. For organizations looking to sell courses, certifications, or training materials, the lack of e-Commerce features can be a significant drawback. While some platforms offer plugins or third-party integrations to support payment processing and course sales, these solutions can add complexity and additional costs to the system. If selling courses or certifications is a priority, it’s crucial to choose an LMS with robust e-Commerce support or consider integrating it with an external e-Commerce platform.
  4. Steep Learning Curve for Administrators and Instructors
    LMS platforms can be complex to navigate, especially for administrators and instructors who may not have a technical background. Setting up courses, managing user roles, configuring permissions, and integrating third-party tools often require specialized training and expertise. This learning curve can lead to inefficiencies, particularly in organizations without dedicated IT or instructional design support. Training costs and time investment can add up, reducing the overall efficiency of the platform.
  5. High Implementation and Maintenance Costs
    Implementing an LMS can be expensive, especially when accounting for customization, setup, training, and content creation. Self-hosted solutions may require ongoing IT support, server maintenance, and regular updates, all of which add to the cost. Even cloud-based solutions can have hidden fees for additional features, support, or upgrades. For organizations with limited budgets, these expenses can quickly become a barrier to effective implementation and long-term use.
  6. User Engagement and Retention Challenges
    While LMS platforms offer tools for tracking engagement and participation, they can sometimes struggle to keep learners motivated, especially in self-paced or online-only environments. If the courses are not designed with engaging content or interactive features, learners may lose interest and drop out. This issue is compounded when the LMS interface is not user-friendly, leading to poor user experience and decreased retention rates.
  7. Lack of Support for Personalized Learning Paths
    While some LMS platforms offer rudimentary support for personalized learning, most struggle to deliver truly customized learning paths that adapt to individual learner needs. This limitation can hinder the ability to address diverse learning styles, knowledge levels, or specific skill gaps. As a result, organizations may need to supplement their LMS with other tools or platforms that provide adaptive learning technologies, which adds complexity to the learning ecosystem.
  8. Data Privacy and Compliance Concerns
    Depending on the region and type of data being stored, LMS platforms may not always comply with data privacy regulations such as GDPR, CCPA, or FERPA. Organizations must carefully evaluate the platform’s data security features and ensure compliance with relevant standards. Failure to meet these requirements can result in significant legal and financial repercussions.

Final Thoughts

Understanding what a Learning Management System is and how it can benefit your organization is crucial in today’s education and training landscape. With platforms like Moodle, Canvas, and Blackboard, it’s easier than ever to create engaging and effective learning experiences. Ready to explore your options? Check out some of these LMS comparisons to find the best platform for your needs.

An LMS isn’t just a tool—it’s a bridge to more effective and scalable learning solutions.

References

Reports, Valuates. (2022). “Learning Management System (LMS) Market to Grow USD 40360 Million by 2028 at a CAGR of 17.1% | Valuates Reports”. www.prnewswire.com (Press release). https://www.prnewswire.com/news-releases/learning-management-system-lms-market-to-grow-usd-40360-million-by-2028-at-a-cagr-of-17-1–valuates-reports-301588142.html

Better buys. (2023). How Much Does an LMS Cost? 2024 Pricing Guide. https://www.betterbuys.com/lms/lms-pricing-guide/

/by
Woman thinking of careers with RIASEC assessment

What is RIASEC Assessment?

Nathan Thompson, PhDSeptember 29, 2024

RIASEC assessment is type of personality assessment used to help individuals identify their career interests and strengths. Based on theory from John Holland, a renowned psychologist, this type of assessment is based on the premise that people perform best in environments that align with their level on six personality factors: Realistic, Investigative, Artistic, Social, Enterprising, and Conventional (RIASEC). Additionally, like the Big Five Personality Traits model, the RIASEC model provides valuable insights by linking personality dimensions with career preferences.

The U.S. Department of Labor’s O*NET system, a comprehensive career database used by millions annually, integrates the RIASEC framework to categorize over 900 occupations. Through tools like the Interest Profiler, it helps users match their interests to relevant career paths, enhancing personalized career exploration.

In this blog post, we’ll dive deeper into what RIASEC assessment is, how it works, and why it’s useful for career planning.

Understanding the RIASEC Model

The RIASEC model is structured around six personality factors:Holland Hexagon

  • Realistic: People who enjoy working with their hands, using tools, and engaging in physical activity. Careers in engineering, construction, or athletics are typical for this type.
  • Investigative: Individuals who are analytical, curious, and enjoy solving complex problems. These people often thrive in science, research, and technical fields.
  • Artistic: Creative thinkers who express themselves through art, music, writing, or design. These individuals prefer jobs in the creative industries.
  • Social: Compassionate and helpful individuals who are drawn to teaching, counseling, or healthcare. Social types enjoy working with others and making a positive impact.
  • Enterprising: These people are confident, persuasive, and like to lead. They often excel in business, sales, or management roles.
  • Conventional: Detail-oriented individuals who enjoy structure and organization. Jobs in accounting, administration, or data management typically attract this type.

You can find more in-depth descriptions of the RIASEC personality types on trusted career exploration platforms like O*Net Online’s Interest Profiler here.

How the RIASEC Assessment Works

Taking a RIASEC assessment typically involves answering a series of questions that measure your preferences for different types of work activities. These questions might ask how much you enjoy tasks like solving math problems, drawing, or managing a project. Based on your responses, the test assigns you a score in each of the six categories. The higher your score in a category, the more likely that personality type fits you.

The results usually highlight your top three RIASEC codes, which are referred to as your Holland Code. This combination helps to suggest career paths or work environments that align with your preferences and strengths.

Example items:

  • Realistic: 
    • I enjoy building things with my hands
    • I prefer a job where I am physically active
  • Investigative: 
    • I would enjoy a job where I need to think hard every day
    • I like crossword puzzles and mind teasers
  • Artistic: 
    • I enjoy making my own art
    • I like to create infographics
  • Social: 
    • I would like a job where I can make a personal impact on people
    • I like to help people
  • Enterprising: 
    • People tend to follow me
    • I would enjoy a job where I talk to people a lot
  • Conventional: 
    • I like to perform tasks where there is a clear right answer
    • I would like a job where there are a lot of numerical calculations

Why Take a RIASEC Assessment?

The RIASEC assessment is valuable for people of all ages. Whether you’re a high school student exploring future career options or a professional considering a career change, the RIASEC model can help clarify which fields best align with your personality. Understanding your Holland Code provides direction on potential job satisfaction, helping to avoid career mismatches that might lead to dissatisfaction or burnout.

Sites like Truity offer free RIASEC assessments that give immediate feedback. They can provide useful insights even if you’re in the early stages of career planning.

Applying Your Results

Career counselor with riasec assessmentOnce you’ve taken the RIASEC assessment, it’s important to use your results thoughtfully. Review your top three personality types and start exploring careers that align with those interests. Many resources, like the U.S. Department of Labor’s CareerOneStop, offer tools to match your RIASEC profile with specific career options.

You should also consider combining your RIASEC results with other career planning tools, such as skill assessments or personality tests like the Myers-Briggs Type Indicator (MBTI). Doing so can provide a fuller picture of how your interests and abilities overlap.  Make sure you use assessments that have predictive validity.

In some cases, a career counselor at your university or other professional might help you interpret results, recommend professions for you to consider, and then help you select an educational pathway to achieve your goals.

Conclusion

The RIASEC assessment is a useful and widely recognized tool for identifying careers that align with your personality and interests. By understanding the six personality types and discovering where your preferences lie, you can make more informed decisions about your career path. Whether you’re just starting out or making a mid-career switch, this assessment provides valuable guidance for finding a job that suits you best.

/by
factor analysis

Factor Analysis: Evaluating Dimensionality in Assessment

Laila Issayeva M.Sc.August 16, 2024

Factor analysis is a statistical technique widely used in research to understand and evaluate the underlying structure of assessment data. In fields such as education, psychology, and medicine, this approach to unsupervised machine learning helps researchers and educators identify latent variables, called factors, and which items or tests load on these factors.

For instance, when students take multiple tests, factor analysis can reveal whether these assessments are influenced by common underlying abilities, like verbal reasoning or mathematical reasoning. This insight is crucial for developing reliable and valid assessments, as it helps ensure that test items are measuring the intended constructs. It can also be used to evaluate whether items in an assessment are unidimensional, which is an assumption of both item response theory and classical test theory.

Why Do We Need Factor Analysis?

Factor analysis is a powerful tool for test validation. By analyzing the data, educators and psychometricians can confirm whether the items on a test align with the theoretical constructs they are designed to measure. This ensures that the test is not only reliable but also valid, meaning it accurately reflects the abilities or knowledge it intends to assess. Through this process, factor analysis contributes to the continuous improvement of educational tools, helping to enhance both teaching and learning outcomes.

What is Factor Analysis?

Factor analysis is a comprehensive statistical technique employed to uncover the latent structure underlying a set of observed variables. In the realms of education and psychology, these observed variables are often test scores or scores on individual test items. The primary goal of factor analysis is to identify underlying dimensions, or factors, that explain the patterns of intercorrelations among these variables. By analyzing these intercorrelations, factor analysis helps researchers and test developers understand which variables group together and may be measuring the same underlying construct.

One of the key outputs of factor analysis is the loading table or matrix (see below), which displays the correlations between the observed variables with the latent dimensions, or factors. These loadings indicate how strongly each variable is associated with a particular factor, helping to reveal the structure of the data. Ideally, factor analysis aims to achieve a “simple structure,” where each variable loads highly on one factor and has minimal loadings on others. This clear pattern makes it easier to interpret the results and understand the underlying constructs being measured. By providing insights into the relationships between variables, factor analysis is an essential tool in test development and validation, helping to ensure that assessments are both reliable and valid.

Confirmatory vs. Exploratory Factor Analysis

Factor analysis comes in two main forms: Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA), each serving distinct purposes in research.

Exploratory Factor Analysis (EFA) is typically used when researchers have little to no prior knowledge about the underlying structure of their data. It is a data-driven approach that allows researchers to explore the potential factors that emerge from a set of observed variables. In EFA, the goal is to uncover patterns and identify how many latent factors exist without imposing any preconceived structure on the data. This approach is often used in the early stages of research, where the objective is to discover the underlying dimensions that might explain the relationships among variables.

On the other hand, Confirmatory Factor Analysis (CFA) is a hypothesis-driven approach used when researchers have a clear theoretical model of the factor structure they expect to find. In CFA, researchers specify the number of factors and the relationships between the observed variables and these factors before analyzing the data. The primary goal of CFA is to test whether the data fit the hypothesized model. This approach is often used in later stages of research or in validation studies, where the focus is on confirming the structure that has been previously identified or theoretically proposed. By comparing the model fit indices, researchers can determine how well their proposed factor structure aligns with the actual data, providing a more rigorous test of their hypotheses.

Factor Analysis of Test Batteries or Sections, or Multiple Predictors

Factor analysis is particularly valuable when dealing with test batteries, which are collections of tests designed to measure various aspects of student cognitive abilities, skills, or knowledge. In the context of a test battery, factor analysis helps to identify the underlying structure of the tests and determine whether they measure distinct yet related constructs.

For example, a cognitive ability test battery might include subtests for verbal reasoning, quantitative reasoning, and spatial reasoning. Through factor analysis, researchers can examine how these subtests correlate and whether they load onto separate factors, indicating they measure distinct abilities, or onto a single factor, suggesting a more general underlying ability, often referred to as the “g” factor or general intelligence.

This approach can also incorporate non-assessment data. For example a researcher on employee selection might look at a set of assessments (cognitive ability, job knowledge, quantitative reasoning, MS Word skills, integrity, counterproductive work behavior), but also variables such as interview scores or resume ratings. Below is an oversimplified example of how the loading matrix might look for this.

Table 1

Variable Dimension 1 Dimension 2
Cognitive ability 0.42 0.09
Job knowledge 0.51 0.02
Quantitative reasoning 0.36 -0.02
MS Word skills 0.49 0.07
Integrity 0.03 0.26
Counterproductive work behavior -0.01 0.31
Interview scores 0.16 0.29
Resume ratings 0.11 0.12

Readers that are familiar with the topic will recognize this as a nod to the work by Walter Borman and Steve Motowidlo on Task vs. Contextual aspects of job performance.  A variable like Job Knowledge would load highly on a factor of task aspects of performing a job.  However, an assessment of counterproductive work behavior might not predict how well they do tasks, but how well they contribute to company culture and other contextual aspects.

This analysis is crucial for ensuring that the test battery provides comprehensive and valid measurements of the constructs it aims to assess. By confirming that each subtest contributes unique information, factor analysis supports the interpretation of composite scores and aids in the design of more effective assessment tools. The process of validating test batteries is essential to maintain the integrity and utility of the test results in educational and psychological settings.

This approach typically uses “regular” factor analysis, which assumes that scores for each input variable are normally distributed. This, of course, is usually the case with something like scores on an intelligence test. But if you are analyzing scores on test items, these are rarely normally distributed, especially for dichotomous data where there is only possible scores of 0 and 1, this is impossible. Therefore, other mathematical approaches must be applied.

Factor Analysis on the Item Level

Factor analysis at the item level is a more granular approach, focusing on the individual test items rather than entire subtests or batteries. This method is used to ensure that each item contributes appropriately to the overall construct being measured and to identify any items that do not align well with the intended factors.

For instance, in a reading comprehension test, factor analysis at the item level can reveal whether each question accurately measures the construct of reading comprehension or whether some items are more aligned with other factors, such as vocabulary knowledge or reasoning skills. Items that do not load strongly onto the intended factor may be flagged for revision or removal, as they could distort the accuracy of the test scores.

This item-level analysis is crucial for developing high-quality educational or knowledge assessments, as it helps to ensure that every question is both valid and reliable, contributing meaningfully to the overall test score. It also aids in identifying “enemy items,” which are questions that could undermine the test’s consistency and fairness.

Similarly, in personality assessments like the Big Five Personality Test, factor analysis is used to confirm the structure of personality traits, ensuring that the test accurately captures the five broad dimensions: openness, conscientiousness, extraversion, agreeableness, and neuroticism. This process ensures that each trait is measured distinctly while also considering how they may interrelate.  Note that the result here was not to show overall unidimensionality in personality, but evidence to support five factors.  An assessment of a given factor is then more or less unidimensional.

An example of this is show in Table 2 below.  Consider if all the descriptive statements are items in a survey where people rate them on a Likert scale of 1 to 5.  The survey might have hundreds of adjectives but these would align themselves with the Big Five with factor analysis, and the simple structure would look like something you see below (2 items per factor in this small example).

 

Table 2

Statement Dimension 1 Dimension 2 Dimension 3 Dimension 4 Dimension 5
I like to try new things 0.63 0.02 0.00 -0.03 -0.02
I enjoy exciting sports 0.71 0.00 0.11 -0.08 0.07
I consider myself neat and tidy 0.02 0.56 0.08 0.11 0.08
I am a perfectionist -0.05 0.69 -0.08 0.09 -0.09
I like to go to parties 0.11 0.15 0.74 0.08 0.00
I prefer to spend my free time alone (reverse scored) 0.13 0.07 0.81 0.01 0.05
I tend to “go with the flow” -0.14 0.02 -0.04 0.68 0.08
I enjoy arguments and debates (reverse scored) 0.03 -0.04 -0.05 0.72 0.11
I get stressed out easily (reverse scored) -0.05 0.03 0.03 0.05 0.81
I perform well under pressure 0.02 0.02 0.02 -0.01 0.77

 

Tools like MicroFACT, a specialized software for evaluating unidimensionality, are invaluable in this process. MicroFACT enables psychometricians to assess whether each item in a test measures a single underlying construct, ensuring the test’s coherence and effectiveness.

Summary

Factor analysis plays a pivotal role in the field of psychometrics, offering deep insights into the structure and validity of educational assessments. Whether applied to test batteries or individual items, factor analysis helps ensure that tests are both reliable and meaningful.

Overall, factor analysis is indispensable for developing effective educational tools and improving assessment practices. It ensures that tests not only measure what they are supposed to but also do so in a way that is fair and consistent across different groups and over time. As educational assessments continue to evolve, the insights provided by factor analysis will remain crucial in maintaining the integrity and effectiveness of these tools.

References

Geisinger, K. F., Bracken, B. A., Carlson, J. F., Hansen, J.-I. C., Kuncel, N. R., Reise, S. P., & Rodriguez, M. C. (Eds.). (2013). APA handbook of testing and assessment in psychology, Vol. 1. Test theory and testing and assessment in industrial and organizational psychology. American Psychological Association. https://doi.org/10.1037/14047-000

Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). The Guilford Press.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). Tata Mcgraw-Hill Ed.

 

/by