Before the introduction of online exams in the education sector, the mentioning of the word ‘exams’ was met with anxiety. Exams were limited to four walls and you could cut the tension in the exam center with a razor. The furious scribbling of papers, the sharp glances from the hawk-eyed invigilators, and the constant ticking of the wall clock is not any experience many will forget.

But then came the internet and there was a better way to assess students. Online exams, though not popular at the time, provided a better way to develop and deliver tests. Using psychometric methods such as Computerized Adaptive Testing and Item Response Theory, assessments became more reliable and secure. Delivery mechanisms including remote proctoring provided students with the ability to take their exams anywhere in the world.

However, despite all these numerous benefits, online exams remained in the dark till the pandemic hit. Many educational institutions and businesses embraced online exams and made them the core of their systems. Forward a year later, and the deployment of vaccines is underway. Many education institutions are confused about which examination models to stick to. Should you go on with the online exams model they used when everyone was stuck in their homes? Should you adopt hybrid examination models, or should you go back to the traditional pen-and-paper method?  

This blog post will provide you with an evaluation of whether offline exams are still worth it in 2021. 

Offline Exams; The good, the bad, and the ugly

The Good

Offline exams have been a stepping stone towards the development of modern assessment models that are more effective. We can’t ignore the fact that there are several advantages of traditional exams. 

Some advantages of offline exams include students having familiarity with the system, development of a social connection between learners, exemption from technical glitches, and affordability. Some schools don’t have the resources and pen-and-paper assessments are the only option available. 

The Bad and The Ugly

However, the pen-and-paper method of assessment has not been able to ensure that exams that achieve their core objectives. The traditional method of assessments is paved with uncertainties and inaccuracies.

 How do you develop and decide the main learning objectives? How do you measure performance and know what to do to improve learning outcomes? And how do you evaluate student strengths and weaknesses? These are just a few questions that the traditional assessment method can’t answer.

 Below is a list of challenges pen-and-paper methods face from test development to evaluation:

1. Needs a lot of resources

From test development to evaluation, pen and paper methods require a lot of resources. Resources can range from high human resource fees to materials needed to develop and deliver the exams to students. 

2. Lack of seamless collaboration and scalability

The ability to cater to a bigger audience is important for productivity and saving resources. However,  the pen-and-paper method offers no room for scalability. Only a fixed number of students can take exams at a certain period. This is not only expensive but also wastes valuable time and increases the chances of leakage.

3. Prone to cheating

Most people think that offline exams are cheat-proof but that is not the case. Most offline exams count on invigilators and supervisors to make sure that cheating does not occur. However, many pen-and-paper assessments are open to leakages. High candidate-to-ratio is another factor that contributes to cheating in offline exams.

4. Poor student engagement

We live in a world of instant gratification and that is the same when it comes to assessments. Unlike online exams which have options to keep the students engaged, offline exams are open to constant destruction from external factors.

Offline exams also have few options when it comes to question types. 

5. Flawed evaluation system

To err is human”

 But, when it comes to assessments, accuracy, and consistency. Traditional evaluation methods are slow and labor-intensive. Instructors take a long time to evaluate tests. This defeats the entire purpose of assessments.

6. Poor result analysis

Pen-and-paper exams depend on instructors to analyze the results and come up with insight. This requires a lot of human resources and expensive software. It is also difficult to find out if your learning strategy is working or it needs some adjustments. 

A glimpse into online exams

Also referred to as digital exams or e-exams, online exams are delivered over the internet. The best online examination platforms include modules to facilitate the entire examination process, from development to delivery. Online exams provide instructors with the ability to diversify question types and monitor the assessment process. Using learning analytics, they are also able to modify learning methods to increase the quality of output. 

Online exams work like your typical pen-and-paper methods but with more accuracy, scalability, and reliability. The grading of the papers is done automatically after the assessment is done, depending on the question types. For essays, instructors can use online essay scoring. E-examinations have improved the assessment process and those are just a few examples. 

Here are some pros and cons of online exams to help you contemplate whether online exams are for you

The pros of online exams

1. Scalability

Unlike traditional testing methods which have a fixed number of people who can take an exam in a fixed time, online exams can cater to bigger audiences. This saves education institutions a lot of resources that would be invested in developing and managing examination centers. 

2. Automated report generation and visualization

This is the greatest advantage online exams have over offline exams. The automated report generation and visualization functions integrated into online assessment platforms enable instructors to accurately gauge learning outcomes. This gives them actionable insight to improve the learning process. 

3. Accessibility

Online exams can be taken from anywhere in the world. All one needs is a computer and an internet connection. This has given students access to knowledge from global learning institutions.

4. Support for diversified question types

Unlike traditional exams which are limited to a certain number of question types, online exams offer many question types option. Multiple Choice Questions, video assessments, coding simulators, and many other question types are supported. With this kind of freedom, instructors are able to decide which question types are fit for certain topics. 

5. In-built psychometrics

Psychometrics is an important part of an assessment as it ensures the development of high-quality tests. The implementation of psychometrics into traditional pen-and-paper methods is a difficult process and depends on the experience of instructors. 

With online exams, you can easily capitalize on it through tech-enhanced items, automated test assembly, Computerized Adaptive Testing, etc. 

6. Improved academic integrity

Cheating is the biggest concern when it comes to online exams. Most people wonder, ‘Isn’t giving students access to a computer and a high-speed internet connection just handing over answers to students?’ Well, that is far from the truth. 

In fact, online exams are safer than offline exams. Online exams are protected using advanced technologies such as Lockdown browser, IP-based authentication, AI-flagging, and many other strategies.

 Check out this article to learn how online exams are secured.  

7. Environmental friendliness

Sustainability is an important aspect of modern civilization.  Online exams eliminate the need to use resources that are not environmentally friendly such as paper. 

The cons of online exams

1. Digital transformation challenges

The process of transitioning examination from offline models to online platforms is one that requires intense planning and resources. However, this barrier can be eliminated easily by creating awareness among students and instructors on how to capitalize on digital assessments.

You can also hire firms with experience in migrating to digital assessments to help in the process. 

2. Academic integrity concerns

Cheating concerns still remain a turn-off for institutions that wish to transition to online exams. There are many ways students can circumnavigate security protocols to cheat in exams. Some ways include impersonation, external help, surfing the internet, and many others. 

However, these cheating ‘tricks’ can be avoided using Assess’ online assessment software with security features such as lock-down browser, IP-based authentication, and AI-powered remote proctoring. 

Offline Exams vs Online exams

traditional approach vs modern approach

Offline exams vs E-examinations

Conclusion

Are offline exams still worth it in 2021? No, they are not. As we have seen from the above sections, the traditional exam approach has several flaws that are barriers to effective assessment. However, it’s not as simple as that. There are many instances where the traditional approach would be a better option. Some instances include when students can’t afford infrastructure. But, when you are looking to conduct high-stakes examinations, online exams are the best option. 

How Assess Can Help 

Transitioning from offline exams to offline exams is not a simple task. That is why Assess is here to help you every step of the way, from test development to delivery. We provide you with the best assessment software and access to the most experienced team. Ready to take your assessments online?

 

 

Assessment is an important part of the learning process as it helps enhance the quality of learning outcomes. With the increased adoption of online assessments, especially during and after the Covid-19 pandemic,  it is important to put in place practices that ease the development of effective online assessments.

 

This is because well-crafted online assessments positively influence the ways in which students approach problems. Online assessments also provide many benefits compared to traditional assessments. Some benefits include; improved grading accuracy, accessibility, improved feedback methods, and many others.

 

But, developing effective online assessments is not an easy task. There are a lot of forces at play.

 

 This 2-part blog series aims to provide you with actionable tips and strategies that can help you improve your online assessments.

 

But, before getting into the nitty-gritty, let’s see some characteristics of high-quality online exams;

Characteristics of High-quality online assessments

Here are some characteristics to look for in good online assessments;

 

  • Fair, defensible, and bias-free
  • Cost-effective and practical
  • Keep track of progress (Short-term and long-term)
  • Flexible and able to scale
  • Provide a real learning experience
  • Include a scoring system that reflects a mastery and not a gross score
  • Should provide diversified question types
  • Provide good feedback mechanisms and actionable insight
  • In alignment with the involved curriculum and standards
  • Reliable and accessible to everyone

 

Now let’s design some effective digital exams!

Use online quizzes to spot student misconceptions

Spotting knowledge gaps is an important part of assessments and online quizzes can help with this. This approach involves giving lecture videos to students before class and then testing them on the same. The answers and feedback are given immediately, preferably with some guidance. 

 

The preferable question type for this strategy is the ‘Fill-in-the-blank’ type or MCQ. 5 attempts are given to each student and they can score full marks if they score correctly within that time.

 

The student responses can then be analyzed, especially the first attempt, and the insights used to shape the learning experience in a good way. 

Capitalize on Adaptive Testing

The benefits of adaptive testing are too numerous to miss out on.

 

What is adaptive testing? Adaptive testing is the delivery of a test to an examinee using an algorithm that adapts the difficulty of the test to their ability.  It also adapts to the number of items used (no need to waste their time). 

It’s like the High Jump in Track & Field.  

 

You start the bar in a middling-low position.  If you make it, the bar is raised.  This continues until you fail, and then it is dropped a little.  Or if you fail the first one, it can be dropped until you succeed.

 

Some benefits of integrating adaptive testing in your strategy include shorter tests, improved test security, individualized exam pacing, and increased motivation. For more information about adaptive testing, check out this blog post

Adaptive testing options

 

Want to start using adaptive testing? Contact us to get started right away. 

Choose the right online assessment tools

Choosing the right online assessment software to develop and deliver exams is not something you can avoid. Not only does digital assessment software automate repetitive tasks, increase efficiency but also helps shape the learning process for better outcomes.

 

But, with the concentrated market of online assessment software, it can be difficult to find the right software. Choosing software that does not align with your assessment strategy can be catastrophic.

 

 A good online assessment platform should cater to your needs in every step of the test development cycle. It should offer the best functionality and have a world-class team at your disposal. 

 

Yet, that can’t be enough to choose software that will help you develop effective online assessments. Here is the perfect resource to help you choose the appropriate tool. 

Understand the ‘Why’ Of assessments

Having a clear definition of why you are developing an assessment is key to making them effective. John Biggs and Catherine Tang, in their constructive alignment theory, argue that assessment tasks (AT) and teaching-learning outcomes (TLA) are created to make sure that students achieve their intended learning outcomes (ITL).

 

 Assessments should be developed based on the ILOs for particular topics. Different learning outcomes are achieved when a variety of assessment types are used. 

 

Effective online assessments have clear goals and create a learning process that ensures students have a chance to self-learn, practice, and receive actionable feedback. 

 

Multiple Choice Questions

Multiple-choice questions are unavoidable when it comes to online assessments. They feature all the benefits an overwhelmed teacher would wish to have in an exam, easy to deliver, easy to develop, and easy to score.  While they often face a poor reputation, they remain so common for a reason: they provide the most bang for your buck in assessment, namely contributing to reliable/valid scores without soaking too much time from teachers or students.

 

But, the art of developing effective MCQs is one that very few possess. To help you save some time here is how to develop effective multiple-choice questions.

 

  • Be strategic when developing MCQs. This is because many tutors fall into the trap of how simple it is to create MCQs, that they lose grip of the big picture. 
  • The responses should be direct. No fluff! 
  • Avoid options like ‘None of the above or ‘All of the above like the plague. Slip them into the assessment occasionally though. 
  • Make them engaging 
  • Language matters. A lot! Avoid grammatical errors or logical contradictions. 
  • Ensure consistency in both the right answers and the distractors. 
  • Stems should be direct and clear. 
  • Eliminate barriers such as sensitive issues or bias for students

 

The effective online assessment checklist

That is a lot of information to digest, but here is a simple checklist to help you know if you are on track to develop online assessments that are effective.

 

  • What defines success in your online assessments? You should have well-established performance metrics to make sure that your examinees get the best out of your entire learning journey?
  • What type of feedback system do you have? The feedback method should help both the student and the teacher get something out of the examination process.
  • How diversified are your online assessments? You should offer different question types depending on learning objectives.
  • Do you have a good online assessment tool? A good online exam tool should provide all the necessary functionality to develop good exams. Feel free to check out this blog post to understand what makes a good online assessment software. 
  • Does your strategy involve peer assessments or self-assessments?
  • Do your assessments empower and motivate students to give their best?
  • To what extent do you involve your students in the assessment process? Make sure to involve them as much as possible. 
  • Are your exams in alignment with the best psychometrics and international standards?
  • How secure and defensible are your exams? Feel free to check out this resource to understand online exam security.
  • How often do you try different assessment strategies? You should keep running tests using different approaches to see what works and what doesn’t.

Final thoughts

Did you check most of the boxes in that list? If yes, you are on your way to developing effective online assessments. If not, try implementing the strategies discussed in the blog post. 

Developing good online exams is not an easy task. It requires a lot of dedication and time. 

If you get stuck, feel free to contact us and we will help you with your entire digital assessment journey. Not only do we have a complete suite of online assessment software to help you develop and deliver effective online exams, but also have an experienced team to guide you every step of the way. 

 

Resources for extra reading

https://cetl.ppu.edu/sites/default/files/publications/-John_Biggs_and_Catherine_Tang-_Teaching_for_Quali-BookFiorg-.pdf

 

https://www.nus.edu.sg/cdtl/docs/default-source/professional-development-docs/resources/designing-online-assessments.pdf

 

Digital assessment security is a highly discussed topic in the Ed-Tech community. The use of digital assessments in education has led to increased concerns about academic integrity. While digital assessments have more advantages than the outdated pen-paper method, they are open to a lot of criticism in relation to academic integrity concerns. However, cheating can be eliminated using the right tools and best practices. 

Before we get into the nitty-gritty of creating secure digital assessments, let’s define cheating.

What is cheating? 

Cheating refers to actual, intended, or attempted deception and/or dishonesty in relation to academic or corporate assessments. 

Some smart ways in which examinees cheat in digital assessments include;

  • Plagiarism– Taking someone else’s work and turning it into one’s own.
  • Access to external resources– Using external resources such as the internet, books, etc. 
  • Using freelancing services– Paying someone else especially from academic freelancing sites such as Studypool to do the work for you. 
  • Impersonation– Letting someone else take an assessment in one’s place.
  • Abetting cheating– Selfless acts such as releasing exam papers early, or discussing answers with examinees who have taken the exam. 
  • Using remote access tools to get external help.

When addressing digital assessments infidelity, most professionals ask the wrong questions. How do we stop cheating? Or which digital assessment security techniques should we use?

 But we believe that is not an effective way to approach this problem. The right questions ask are; why do students cheat? how can we make students/employees be confident enough to take assessments without cheating? Which digital assessment tools address all the weakest links in the digital assessment life cycle? For instance, by integrating techniques such as adaptive testing or multistage-testing, the students and employees feel confident to take exams. 

In this blog post, we are going to share digital assessment techniques and best practices that can help you create effective assessments. 

Part 1: Digital exams security techniques 

Online/Remote proctoring is the process of monitoring digital assessments. This gives supervisors and invigilators the freedom to monitor examinations from any location. 

1. Online Proctoring

To increase assessment security in this process, you can utilize these methods;

  • Image Capturing

This is a technique used to increase the effectiveness of online testing by taking photos of activities in a given interval of time. The images are later analyzed to flag down any form of infidelity detected in the image banks. This is a basic mechanism, yet effective in situations where students don’t have access to a stable internet connection. 

  • Audio Capturing

This one is almost similar to image/video capturing but keeps track of audio logs. The audio logs are later analyzed to identify any voice abnormalities in the surrounding environment. This eliminates risks such as getting help from family members or friends in the same room. 

  • Live Video Recording and/ Streaming

This is the most important element of security in digital assessments. Live video streaming keeps the candidate under constant surveillance and invigilators can suspend the examination if they notice suspicious behavior. Some online testing tools are even integrated with AI systems to help in the process of identifying suspicious activity. Video recording on the other hand works like audio capturing. 

Digital assessment security: Remote Proctoring,  Record and Review
Remote Proctoring: Record and Review

 

The video is recorded, stored in the cloud, and later analyzed to spot any suspicious behavior. 

To give you the freedom to be creative and flexible in the assessment process, Assess.com even lets you bring your own proctor.

2. Lockdown Browser

 

High-speed broadband and access to hundreds of tools to do anything. Sounds like the perfect opportunity to cheat? Yes, to some extent. However, with the lockdown browser feature, students are limited to one tab, the one hosting the exam. 

If the student tries accessing any external software or opening a tab, a notification is sent to the invigilator. The feature starts by scanning the computer for video/image capture. Once the test begins, the examinee is restricted from surfing the internet for answers. 

3. Digital assessment security through AI Flagging

Digital Assessment security: AI Flagging In Remote Proctoring
AI Flagging In Remote Proctoring 

 

As seen in the article 10 EdTech Trends and Predictions To Watch Out For, In 2021,  AI will increase in popularity in Ed-Tech. Not only can facial recognition (which is a form of AI) be used to increase security on campus but also in digital assessment.

 By using systems integrated with AI systems, supervisors can identify suspicious behavior and take action, in real-time. Some actions the AI systems are able to identify include two faces in one screen.  This feature works in hand with video streaming/recording. This helps increase examination integrity while cutting back on costs. 

4. IP-Based Authentication

 

Identity authentication is an integral step in achieving the highest form of academic integrity in digital assessments. IP-Based authentication uses a user’s IP address to confirm the identity of students undertaking the exam. This comes in handy when examinees try to cheat by offering remote access to other users. In a recent paper exploring, e-authentication systems, other components of authentication may include biometric systems, mechanisms, knowledge, and document analysis. Using IP-Based authentication in digital assessment may decrease some risks of cheating.

5. Digital assessment security by data encryption

Cybercrimes have become a threat to many industries. Education is one of the industries that face a lot of challenges when it comes to information security. In fact, according to a recent study, 83% of schools in the UK experienced at least one cyberattack, despite having protection mechanisms such as antivirus software and firewalls. Another research also discovered that education records could be sold as high as $265 on the black market

Digital assesment security: CyberSecurity By Industry
CyberSecurity By Industry

 

 This kind of insecurity finds its way into digital assessment because most tests are developed and delivered online. Some students can get early access to digital assessments or change their grades if they can find a loophole in the digital assessment systems. To prevent this, it is important to implement cybersecurity practices in digital assessment systems. Data encryption is a key part of ensuring that exams and personal data are safe.  

  1. Audit Logging

The audit logging function tracks user activity in all aspects of the testing process using unique identifiers such as IP addresses. It keeps track of all activities including clicks, stand-by time, software accessed during the assessment, and so much more! These logs are then analyzed for signs of infidelity. 

Part II: Digital assessment security best practices

Digital assesment life cycle
Digital Assessment Life Circle

 

Most of the technologies discussed above focus on the last few stages of the evaluation process. The strategies below focus on techniques that do not fall under the main stages of the process, yet very critical.

1. Creating Awareness On Cheating

Many institutions don’t create enough awareness of the dangers of cheating in relation to academic integrity. You can use media such as videos to give them guidelines of how to undertake exams authentically, and what is considered cheating. This prepares the students to undertake the exams without thinking about cheating. 

2. Don’t Use Publisher Item Banks

Course textbooks and guides may come with complimentary test banks, and you should refrain from using them in assessments. This is because most course materials are available on the internet, and may encourage infidelity. Instead, make up your own questions, based on your learning objectives and methodologies. 

3. Disable Backtracking

This increases learning efficiency by making sure that students answer what they know, and locate their weakest links. By facilitating backtracking, you give the students time to go back and try to locate the correct answers, which reduces effectiveness. 

4. Diversity In Question Types

Fear of Failure is one thing that makes examinees cheat. By creating diverse question types, you improve the assessment engagement of the students, therefore improving their confidence. Don’t give your examinees multiple-choice questions only, add in some yes/no questions, essays, and so on.

5. Make Them Sign Academic Integrity Contracts

Contracts have a psychological effect on people, and the examinees are more likely to be authentic if they sign some form of contract. 

If you are interested in other best practices to increase online assessment security, here is an important resource that can help.

Conclusion

Online exam security is an important aspect of online education and should not be ignored. Despite the immaturity of existing frameworks and methodologies in the market, techniques such as remote proctoring, lockdown browser, and audit logging have proven to be effective. 

Creating and delivering high-quality digital assessments is not a walk in the park. If you need professional help in creating online assessments with alignment to the best psychometrics practices, contact Assess for a free consultation. 

You can also get access to online assessment tools with security features including, AI-flagging, remorse proctoring, lock-down browser and so much more!

Estimated reading time: 5 minutes

What is computerized adaptive testing?  Computerized adaptive tests (CATs) are a sophisticated method of test delivery based on item response theory (IRT). By using AI algorithms to personalize the test to every examinee, the test becomes shorter, more accurate, more secure, and fairer.

 

What is computerized adaptive testing?

Computerized adaptive testing is an algorithm that drives how a test is delivered.  It is coded into a software platform, using the machine-learning approach of IRT to select items and score examinees.  The algorithm proceeds in a loop until the test is complete.

computerized adaptive testing

Adapting By Difficulty and Quantity

They operate by adapting both the difficulty and quantity of items seen by each examinee.

Difficulty
Most characterizations of adaptive testing focus on how item difficulty is matched to examinee ability. High-ability examinees receive more difficult items, while low ability examinees receive easier items, which has important benefits to the student and the organization. An adaptive test typically begins by delivering an item of medium difficulty; if you get it correct, you get a tougher item, and if you get it incorrect, you get an easier item. This basic algorithm continues until the test is finished, though it usually includes sub algorithms for important things like content distribution and item exposure.

Quantity
A less publicized facet of adaptation is the number of items. Adaptive tests can be designed to stop when certain psychometric criteria are reached, such as a specific level of score precision. Some examinees finish very quickly with few items, so that adaptive tests are typically about half as many questions as a regular test, with at least as much accuracy. Since some examinees have longer tests, these adaptive tests are referred to as variable-length. Obviously, this makes for a massive benefit: cutting testing time in half, on average, can substantially decrease testing costs.

But, some adaptive tests use a fixed length, and only adapt item difficulty. This is merely for public relations issues, namely the inconvenience of dealing with examinees who feel they were unfairly treated by the CAT, even though it is arguably more fair and valid than conventional tests.

Advantages of computerized adaptive testing

By making the test more intelligent, adaptive testing provides a wide range of benefits.  Some of the well-known advantages of adaptive testing, recognized by scholarly psychometric research, are listed below.  However, the development of an adaptive test is a very complex process that requires substantial expertise in item response theory (IRT) and CAT simulation research.  

Our experienced team of psychometricians can provide your organization with the requisite experience to implement adaptive testing and help your organization benefit from these advantages. Contact us or read this white paper to learn more.

  • Shorter tests, anywhere from a 50% to 90% reduction; reduces cost, examinee fatigue, and item exposure
  • More precise scores: CAT will make tests more accurate
  • More control of score precision (accuracy): CAT ensures that all students will have the same accuracy, making the test much more fair.  Traditional tests measure the middle students well but not the top or bottom students.
  • Increased efficiency
  • Greater test security because everyone is not seeing the same form
  • A better experience for examinees, as they only see items relevant for them, providing an appropriate challenge
  • The better experience can lead to increased examinee motivation
  • Immediate score reporting
  • More frequent retesting is possible; minimize practice effects
  • Individual pacing of tests; examinees move at their own speed
  • On-demand testing can reduce printing, scheduling, and other paper-based concerns
  • Storing results in a database immediately makes data management easier
  • Computerized testing facilitates the use of multimedia in items
 

No, you can’t just subjectively rank items!

Computerized adaptive tests (CATs) are the future of assessment. They operate by adapting both the difficulty and number of items to each individual examinee. The development of an adaptive test is no small feat, and requires five steps integrating the expertise of test content developers, software engineers, and psychometricians.

The development of a quality adaptive test is complex and requires experienced psychometricians in both item response theory (IRT) calibration and CAT simulation research. FastTest can provide you the psychometrician and software; if you provide test items and pilot data, we can help you quickly publish an adaptive version of your test.

Step 1: Feasibility, applicability, and planning studies. First, extensive monte carlo simulation research must occur, and the results formulated as business cases, to evaluate whether adaptive testing is feasible, applicable, or even possible.

Step 2: Develop item bank. An item bank must be developed to meet the specifications recommended by Step 1.

Step 3: Pretest and calibrate item bank. Items must be pilot tested on 200-1000 examinees (depends on IRT model) and analyzed by a Ph.D. psychometrician.

Step 4: Determine specifications for final CAT. Data from Step 3 is analyzed to evaluate CAT specifications and determine most efficient algorithms using CAT simulation software such as CATSim.

Step 5: Publish live CAT. The adaptive test is published in a testing engine capable of fully adaptive tests based on IRT. Want to learn more about our one-of-a-kind model? Click here to read the seminal article by two of our psychometricians. More adaptive testing research is available here.

What do I need for adaptive testing?

Minimum requirements:

  • A large item bank piloted on at least 500 examinees
  • 1,000 examinees per year
  • Specialized in IRT calibration and CAT simulation software.
  • Staff with a Ph.D. in psychometrics or an equivalent level of experience. Or, leverage our internationally recognized expertise in the field.
  • Items (questions) that can be scored objectively correct/incorrect in real-time
  • An item banking system and CAT delivery platform
  • Financial resources: Because it is so complex, the development of a CAT will cost at least $10,000 (USD) — but if you are testing large volumes of examinees, it will be a significant positive investment. If you pay $20/hour for proctoring seats and cut a test from 2 hours to 1 hour for just 1,000 examinees… that’s a $20,000 savings.

Adaptive testing: Resources for further reading

Visit the links below to learn more about adaptive testing.  

How can I start developing a CAT?

Contact us for a free account in our industry-leading CAT platform.

ASC has been empowering organizations to develop better assessments since 1979.  Curious as to how things were back then?  Below is a copy of our newsletter from 1988, long before the days of sharing news via email and social media!  Our platform at the time was named MICROCAT.  This later became modernized to FastTest PC (Windows), then FastTest Web, and is now being reincarnated yet again as Assess.ai.

Special thanks to Cliff Donath for finding and sharing!

[pdf-embedder url=”https://assess.com/wp-content/uploads/2019/02/MicroCATNewsAprill1988.pdf”]

Psychometrics is the cornerstone of any high-quality assessment program.  However, most organizations can’t afford an in-house Ph.D. psychometrician, which then necessitates the search for psychometric consulting.  Most organizations, when first searching, are new to the topic and not sure what role the psychometrician plays. 

In this article, we’ll talk about how psychometricians and their tools can help improve your assessments, whether you just want to check on test reliability or pursue the lengthy process of accreditation.

Why ASC?

Whether you are establishing or expanding a credentialing program, streamlining operations, or moving from paper to online testing, ASC has a proven track record of providing practical, cost-efficient solutions with uncompromising quality. We offer a free consultation with our team of experts to discuss your needs and determine which solutions are the best fit, including our enterprise SaaS platforms, consulting on sound psychometrics, or recommending you to one of our respected partners.

At the heart of our business are our people.

Our collaborative team of Ph.D. psychometricians, accreditation experts, and software developers have diverse experience developing solutions that drive best practices in assessment. This real-world knowledge enables us to consult your organization with solutions tailored specifically to your goals, timeline, and budget.

Comprehensive Solutions to Address Specific Measurement Problems

Much of psychometric consulting is project-based around solving a specific problem.  For example, you might be wondering how to set a cut score on a certification/licensure exam that is legally defensible and meets accreditation standards. 

This is a very specific issue, and the scientific literature has suggested a number of sound approaches.  Here are some of the topics where psychometricians can really help:

  • Test Design: Job Analysis & Blueprints
  • Standard and Cutscore Setting Studies
  • Item Writing and Review Workshops
  • Test and Item Statistical Analysis
  • Equating Across Years and Forms
  • Adaptive Testing Research
  • Test Security Evaluation
  • NCCA/ANSI Accreditation

Why psychometric consulting?

All areas of assessment can be smarter, faster, and fairer.

Develop Reliable and Valid Assessments
We’ll help you understand what needs to be done to develop defensible tests and how to implement them in a cost-efficient manner.  Much of the work revolves around establishing a sound test development cycle.

Increase Test Security
We have specific expertise in psychometric forensics, allowing you to flag suspicious candidates or groups in real-time, using our automated forensics report.

Achieve Accreditation
Our dedicated experts will assist in setting your organization up for success with NCCA/ANSI accreditation of professional certification programs.

Comprehensive Psychometric Analytics
We use CTT and IRT with principles of machine learning and AI to deeply understand your data and provide actionable recommendations.

We can help your organization develop and publish certification and licensure exams, based on best practices and accreditation standards, in a matter of months.

Effective assessments; Best practices

Item and Test Statistical Analysis
If you are doing this process at least annually, you are not meeting best practices or accreditation standards. But don’t worry, we can help! In addition to performing these analyses for you, you also have the option of running them yourself in our FastTest platform or using our psychometric software like Iteman and Xcalibre.

Job Analysis
How do you know what a professional certification test should cover?  Well, let’s get some hard data by surveying job incumbents. Knowing and understanding this information and how to use it is essential if you want to test people on whether they are prepared for the job or profession.

Cut score Studies (Standard Setting)
When you use sound psychometric practices like the modified-Angoff, Beuk Compromise, Bookmark, and Contrasting Groups methods, it will help you establish a cutscore that meets professional standards.

 

It’s all much easier if you use the right software!

Once we help you determine the best solutions for your organization, we can train you on best practices, and it’s extremely easy to use our software yourself.  Software like Iteman and Xcalibre is designed to replace much of the manual work done by psychometricians for the item and test analysis, and FastTest automates many aspects of test development and publishing.  We even offer free software like the Angoff Analysis Tool

However, our ultimate goal is your success: Assessment Systems is a full-service company that continues to provide psychometric consulting and support even after you’ve made a purchase. Our team of professionals is available to provide you with additional support at any point in time. We want to ensure you’re getting the most out of our products!  Click below to sign up for a free account in FastTest and see for yourself.

Sign up for a Free Account

ProctorExam and ASC have partnered to offer a much more secure solution for online test administration, which more and more organizations will need in our changing world of technology and online learning. This is because security requires a multi-layered approach: in addition to streaming a webcam, there is also screen capture, and use of a 3rd camera.  Moreover, security also rests on the delivery engine, which should include functionality for screen lockdown, randomization, adaptive testing, LOFT, passwords, and more.  ASC has recently partnered with ProctorExam to leverage their innovative platform for remote proctoring, layering that level of security over the security already available in our test engine.

About ProctorExam

Since launching their services in late 2014, ProctorExam has stood out by offering unique and creative solutions to enhance assessment security. It has been a consistent innovator in the online education market with its innovative 360-degree monitoring with the users’ smart devices, and a full range of proctoring options. Over the past year, ProctorExam has been working to enhance their line of innovative services by introducing Proctoring Infrastructure as a Service. This new paradigm has already begun to decrease cost, increase transparency, and reduce reliance on third-party services.

Daniel Haven, Director of ProctorExam: “Due to the shared vision for test taking, we are very excited about working with Assessment Systems Corporation. In the future, we want to see whether we can share more lifecycle events to further automate the entire testing process and not necessarily only use communal data to understand whether a person is cheating or not, but also to further increase the quality of the test itself, by better understanding whether specific questions can be shorter or longer, whether they’re too difficult or not.”

About ASC

Assessment Systems Corporation, has been a leader in the assessment and testing industry since 1979, providing both world-class software for testing and psychometrics (with a focus on item response theory and computerized adaptive testing) and extensive consulting services for testing organizations. Assessment Systems was the first company to offer item banking and testing software to the public, in the form of the DOS-based MicroCAT adaptive testing system, nearly 30 years ago. For the past 10 years, they have been implementing this comprehensive testing program into a web-based system that is accessible anywhere, both for examinees and users. ASC’s delivery platform can be used by any type of assessment: certification, K12, higher education, workforce, medicine, and more.

David Weiss, President and CEO at Assessment Systems Corporation, states: “More and more clients will have access to the combined technology. The benefit of ProctorExam is that it is comparatively economical, so it’s allowing Assessment Systems Corporation to offer proctoring solutions to people who otherwise wouldn’t be able to do remote delivery. This allows us to open up new opportunities, especially in education – where there are a lot of high-stakes tests delivered.”

Estimated reading time: 6 minutes

Computerized adaptive testing (CAT) has been around since the 1970s and is well-known for the benefits it can provide, most notably that it can reduce testing time 50-90% with no loss of measurement precision.  Developing a sound, defensible CAT is not easy, but our goal is to make it as easy as possible – that is, everything you need is available in clean software UI and you never have to write a single line of code.

Here, we outline the software, data analysis, and project management steps needed to develop a computerized adaptive test that aligns with best practices and international standards.

This approach is based on Thompson and Weiss (2011) model, which refers there for a general treatment of CAT development, especially the use of simulation studies.  Also, this article assumes you have mastered the concepts of item response theory and CAT, including:

  • IRT models (e.g., 3PL, rating scale model)
  • Item response functions
  • Item information functions
  • Theta estimation
  • Conditional standard error of measurement
  • Item selection algorithms
  • Termination criterion.

 

If IRT is new to you, please visit these resources

https://assess.com/what-is-item-response-theory/

https://assess.com/how-do-i-implement-item-response-theory/

https://assess.com/what-do-dichotomous-and-polytomous-mean-in-irt/

If you have some background in IRT but CAT is new, please visit these resources

https://assess.com/monte-carlo-simulation-adaptive-testing/

https://assess.com/adaptive-testing/

Overview: Steps to develop an adaptive test

There are nine steps to developing a CAT on our industry-leading platform, fastest:

Step Work to be done Software
1 Perform feasibility and planning studies CATSim
2 Develop item bank FastTest
3 Pilot items on 100-2000 examinees FastTest
4 Perform item analysis and other due diligence Iteman/Xcalibre
5 IRT calibration Xcalibre
6 Upload IRT parameters into FastTest FastTest
7 Validity study CATSim
8 Publish CAT FastTest
9 Quality assurance FastTest

 

We’ll now talk a little more about each of these.

 

Perform feasibility and planning studies

The first step, before doing anything else, is to confirm that your assessment meets the basic requirements of CAT.  For example, you need to have a decent-sized item bank, data on hundreds of examinees (or the future opportunity), and items that are scoreable in real-time.  See this paper for a full discussion. 

 If there are no huddles, then the next step is to perform monte Carlo simulations that help you scope out the project, using the CATSim software.  For example, you might simulate CATs with three sizes of the item bank, so you have a better idea of how many items to write.

 

Develop item bank

Now that you have some idea of how many items you need and in which ranges of difficulty and/or content constraints, you can leverage the powerful item authoring functionality of FastTest, as well as the item review and workflow management to ensure that subject matter experts are performing quality assurance on each other.

 

Pilot items

Because IRT requires that you have data from real examinees to calibrate item difficulty, you need to get that data.  To do so, create test(s) in FastTest to deliver all your items in a matter that meets your practical situation. That is, some organizations have a captive audience and might be able to have 500 people take all 300 items in their bank next week.  Other organizations might need to create 4 linear forms of 100 items with some overlap. Others might be constrained to still use current test forms and only tack on 20 new items onto the end of every examinee’s test.

Of course, some of you might have existing data.  That is, you might have spreadsheets of data from a previous test delivery system, paper-based delivery, or perhaps even already have your IRT parameters from past efforts.  You can use those too.

If you do deliver the pilot phase with FastTest, you now need to export the data to be analyzed in psychometric analytic software.  FastTest makes it easy to export both the data matrix and the item metadata needed for Xcalibre’s control file.

 

Perform item analysis, DIF, and other due diligence

The purpose of this step is to ensure that items included in your future CAT are of high quality.  Any steps that your organization normally does to review item performance is still relevant. This typically includes a review of items with low point-biserial correlations (poor discrimination), items where more examinees selected a distractor than the correct option (key flags), high or low classical P values, and differential item functioning (DIF) flags.  Our Iteman software is designed exactly for this process. If you have a FastTest account the Iteman analysis report is now available at a single click. If not, Iteman is also available as a standalone program.

 

Calibrate with Xcalibre

Because CAT algorithms rely entirely on IRT parameters (unless you are doing special algorithms like diagnostic measurement models or measurement decision theory), we need to calculate the IRT parameters and get them into our testing platform.  If you have delivered all your items in a single block to examinees, like the example above with 500 people, then that single matrix can just be analyzed with Xcalibre. If you have multiple forms, LOFT, or the “tack-on” approach, you need to worry about IRT equating.

 

Upload IRT parameters into FastTest

Xcalibre will provide all the IRT parameters in a spreadsheet, in addition to the primary Word report.  Import them into your testing platform.  This will associate the IRT parameters with all the items in your CAT pool.  FastTest has functionality to streamline this process.

 

Validity study

Now that you have your final pool of items established, and calculated the IRT parameters, you need to establish the algorithms you are going to use to publish the CAT.  That is, you need to decide on the Initial Theta rule, Item Selection rule (including sub algorithms like content or exposure constraints), and Termination Criterion. To establish these, you need to perform more simulation studies, but now with your final bank as the input rather than a fake bank from the monte Carlo simulations.  The most important aspect is determining the tradeoff between test length and precision; a termination criterion that provides more precise scores will have longer tests, and you can control the exact extent with a CAT.

 

Publish CAT

Assemble a “test form” in FastTest that consists of all the items you intend to use in your CAT tool.  Then select CAT as the delivery method in the Test Options screen, and you’ll see a screen where you can input the results from your CATSim validity study for the three important CAT algorithms.

 

Quality assurance

Your CAT is now ready to go!  Before bringing in real students, however, we recommend that you take it a few times as QA.  Do so with certain students in mind, such as a very low student, a very high student, or one near the cut score (if you have one).  To peek under the hood at the CAT algorithm, you can export the Examinee Test Detail Report from FastTest, which provides an item-by-item picture of how the CAT proceeds.

Adaprive testing examinee report

Summary

As you can see, the development of an adaptive test is not easy and can take months even if you have all the software and expertise you need.  But for something so important, that could be used to make important decisions about people, this is absolutely warranted.

  However, if you have all the data you need today, there’s no reason that it should take months to develop an adaptive test – assessment platforms should make it easy enough for you to do so in an afternoon, which FastTest absolutely does.

Want to talk with one of our experts about applying this process to your exam?  Get in touch or sign up for a free account