Digital assessment security is a highly discussed topic in the Ed-Tech community. The use of digital assessments in education has led to increased concerns about academic integrity. While digital assessments have more advantages than the outdated pen-paper method, they are open to a lot of criticism in relation to academic integrity concerns. However, cheating can be eliminated using the right tools and best practices. 

Before we get into the nitty-gritty of creating secure digital assessments, let’s define cheating.

What is cheating? 

Cheating refers to actual, intended, or attempted deception and/or dishonesty in relation to academic or corporate assessments. 

Some smart ways in which examinees cheat in digital assessments include;

  • Plagiarism– Taking someone else’s work and turning it into one’s own.
  • Access to external resources– Using external resources such as the internet, books, etc. 
  • Using freelancing services– Paying someone else especially from academic freelancing sites such as Studypool to do the work for you. 
  • Impersonation– Letting someone else take an assessment in one’s place.
  • Abetting cheating– Selfless acts such as releasing exam papers early, or discussing answers with examinees who have taken the exam. 
  • Using remote access tools to get external help.

When addressing digital assessments infidelity, most professionals ask the wrong questions. How do we stop cheating? Or which digital assessment security techniques should we use?

 But we believe that is not an effective way to approach this problem. The right questions ask are; why do students cheat? how can we make students/employees be confident enough to take assessments without cheating? Which digital assessment tools address all the weakest links in the digital assessment life cycle? For instance, by integrating techniques such as adaptive testing or multistage-testing, the students and employees feel confident to take exams. 

In this blog post, we are going to share digital assessment techniques and best practices that can help you create effective assessments. 

Part 1: Digital exams security techniques 

Online/Remote proctoring is the process of monitoring digital assessments. This gives supervisors and invigilators the freedom to monitor examinations from any location. 

1. Online Proctoring

To increase assessment security in this process, you can utilize these methods;

  • Image Capturing

This is a technique used to increase the effectiveness of online testing by taking photos of activities in a given interval of time. The images are later analyzed to flag down any form of infidelity detected in the image banks. This is a basic mechanism, yet effective in situations where students don’t have access to a stable internet connection. 

  • Audio Capturing

This one is almost similar to image/video capturing but keeps track of audio logs. The audio logs are later analyzed to identify any voice abnormalities in the surrounding environment. This eliminates risks such as getting help from family members or friends in the same room. 

  • Live Video Recording and/ Streaming

This is the most important element of security in digital assessments. Live video streaming keeps the candidate under constant surveillance and invigilators can suspend the examination if they notice suspicious behavior. Some online testing tools are even integrated with AI systems to help in the process of identifying suspicious activity. Video recording on the other hand works like audio capturing. 

Digital assessment security: Remote Proctoring,  Record and Review
Remote Proctoring: Record and Review

 

The video is recorded, stored in the cloud, and later analyzed to spot any suspicious behavior. 

To give you the freedom to be creative and flexible in the assessment process, Assess.com even lets you bring your own proctor.

2. Lockdown Browser

 

High-speed broadband and access to hundreds of tools to do anything. Sounds like the perfect opportunity to cheat? Yes, to some extent. However, with the lockdown browser feature, students are limited to one tab, the one hosting the exam. 

If the student tries accessing any external software or opening a tab, a notification is sent to the invigilator. The feature starts by scanning the computer for video/image capture. Once the test begins, the examinee is restricted from surfing the internet for answers. 

3. Digital assessment security through AI Flagging

Digital Assessment security: AI Flagging In Remote Proctoring
AI Flagging In Remote Proctoring 

 

As seen in the article 10 EdTech Trends and Predictions To Watch Out For, In 2021,  AI will increase in popularity in Ed-Tech. Not only can facial recognition (which is a form of AI) be used to increase security on campus but also in digital assessment.

 By using systems integrated with AI systems, supervisors can identify suspicious behavior and take action, in real-time. Some actions the AI systems are able to identify include two faces in one screen.  This feature works in hand with video streaming/recording. This helps increase examination integrity while cutting back on costs. 

4. IP-Based Authentication

 

Identity authentication is an integral step in achieving the highest form of academic integrity in digital assessments. IP-Based authentication uses a user’s IP address to confirm the identity of students undertaking the exam. This comes in handy when examinees try to cheat by offering remote access to other users. In a recent paper exploring, e-authentication systems, other components of authentication may include biometric systems, mechanisms, knowledge, and document analysis. Using IP-Based authentication in digital assessment may decrease some risks of cheating.

5. Digital assessment security by data encryption

Cybercrimes have become a threat to many industries. Education is one of the industries that face a lot of challenges when it comes to information security. In fact, according to a recent study, 83% of schools in the UK experienced at least one cyberattack, despite having protection mechanisms such as antivirus software and firewalls. Another research also discovered that education records could be sold as high as $265 on the black market

Digital assesment security: CyberSecurity By Industry
CyberSecurity By Industry

 

 This kind of insecurity finds its way into digital assessment because most tests are developed and delivered online. Some students can get early access to digital assessments or change their grades if they can find a loophole in the digital assessment systems. To prevent this, it is important to implement cybersecurity practices in digital assessment systems. Data encryption is a key part of ensuring that exams and personal data are safe.  

  1. Audit Logging

The audit logging function tracks user activity in all aspects of the testing process using unique identifiers such as IP addresses. It keeps track of all activities including clicks, stand-by time, software accessed during the assessment, and so much more! These logs are then analyzed for signs of infidelity. 

Part II: Digital assessment security best practices

Digital assesment life cycle
Digital Assessment Life Circle

 

Most of the technologies discussed above focus on the last few stages of the evaluation process. The strategies below focus on techniques that do not fall under the main stages of the process, yet very critical.

1. Creating Awareness On Cheating

Many institutions don’t create enough awareness of the dangers of cheating in relation to academic integrity. You can use media such as videos to give them guidelines of how to undertake exams authentically, and what is considered cheating. This prepares the students to undertake the exams without thinking about cheating. 

2. Don’t Use Publisher Item Banks

Course textbooks and guides may come with complimentary test banks, and you should refrain from using them in assessments. This is because most course materials are available on the internet, and may encourage infidelity. Instead, make up your own questions, based on your learning objectives and methodologies. 

3. Disable Backtracking

This increases learning efficiency by making sure that students answer what they know, and locate their weakest links. By facilitating backtracking, you give the students time to go back and try to locate the correct answers, which reduces effectiveness. 

4. Diversity In Question Types

Fear of Failure is one thing that makes examinees cheat. By creating diverse question types, you improve the assessment engagement of the students, therefore improving their confidence. Don’t give your examinees multiple-choice questions only, add in some yes/no questions, essays, and so on.

5. Make Them Sign Academic Integrity Contracts

Contracts have a psychological effect on people, and the examinees are more likely to be authentic if they sign some form of contract. 

If you are interested in other best practices to increase online assessment security, here is an important resource that can help.

Conclusion

Online exam security is an important aspect of online education and should not be ignored. Despite the immaturity of existing frameworks and methodologies in the market, techniques such as remote proctoring, lockdown browser, and audit logging have proven to be effective. 

Creating and delivering high-quality digital assessments is not a walk in the park. If you need professional help in creating online assessments with alignment to the best psychometrics practices, contact Assess for a free consultation. 

You can also get access to online assessment tools with security features including, AI-flagging, remorse proctoring, lock-down browser and so much more!

Estimated reading time: 5 minutes

Computerized adaptive tests (CATs) are a sophisticated method of test delivery based on item response theory (IRT). They operate by adapting both the difficulty and quantity of items seen by each examinee.

Difficulty
Most characterizations of adaptive testing focus on how item difficulty is matched to examinee ability. High-ability examinees receive more difficult items, while low ability examinees receive easier items, which has important benefits to the student and the organization. An adaptive test typically begins by delivering an item of medium difficulty; if you get it correct, you get a tougher item, and if you get it incorrect, you get an easier item. This basic algorithm continues until the test is finished, though it usually includes sub algorithms for important things like content distribution and item exposure.

Quantity
A less publicized facet of adaptation is the number of items. Adaptive tests can be designed to stop when certain psychometric criteria are reached, such as a specific level of score precision. Some examinees finish very quickly with few items, so that adaptive tests are typically about half as many questions as a regular test, with at least as much accuracy. Since some examinees have longer tests, these adaptive tests are referred to as variable-length. Obviously, this makes for a massive benefit: cutting testing time in half, on average, can substantially decrease testing costs.

But, some adaptive tests use a fixed length, and only adapt item difficulty. This is merely for public relations issues, namely the inconvenience of dealing with examinees who feel they were unfairly treated by the CAT, even though it is arguably more fair and valid than conventional tests.

Advantages of adaptive testing

By making the test more intelligent, adaptive testing provides a wide range of benefits.  Some of the well-known advantages of adaptive testing, recognized by scholarly psychometric research, are listed below.  However, the development of an adaptive test is a very complex process that requires substantial expertise in item response theory (IRT) and CAT simulation research.  

Our experienced team of psychometricians can provide your organization with the requisite experience to implement adaptive testing and help your organization benefit from these advantages. Contact us or read this white paper to learn more.

  • Shorter tests, anywhere from a 50% to 90% reduction; reduces cost, examinee fatigue, and item exposure
  • More precise scores: CAT will make tests more accurate
  • More control of score precision (accuracy): CAT ensures that all students will have the same accuracy, making the test much more fair.  Traditional tests measure the middle students well but not the top or bottom students.
  • Increased efficiency
  • Greater test security because everyone is not seeing the same form
  • A better experience for examinees, as they only see items relevant for them, providing an appropriate challenge
  • The better experience can lead to increased examinee motivation
  • Immediate score reporting
  • More frequent retesting is possible; minimize practice effects
  • Individual pacing of tests; examinees move at their own speed
  • On-demand testing can reduce printing, scheduling, and other paper-based concerns
  • Storing results in a database immediately makes data management easier
  • Computerized testing facilitates the use of multimedia in items

No, you can’t just subjectively rank items!

Computerized adaptive tests (CATs) are the future of assessment. They operate by adapting both the difficulty and number of items to each individual examinee. The development of an adaptive test is no small feat, and requires five steps integrating the expertise of test content developers, software engineers, and psychometricians.

The development of a quality adaptive test is complex and requires experienced psychometricians in both item response theory (IRT) calibration and CAT simulation research. FastTest can provide you the psychometrician and software; if you provide test items and pilot data, we can help you quickly publish an adaptive version of your test.

Step 1: Feasibility, applicability, and planning studies. First, extensive monte carlo simulation research must occur, and the results formulated as business cases, to evaluate whether adaptive testing is feasible, applicable, or even possible.


Step 2: Develop item bank. An item bank must be developed to meet the specifications recommended by Step 1.

Step 3: Pretest and calibrate item bank. Items must be pilot tested on 200-1000 examinees (depends on IRT model) and analyzed by a Ph.D. psychometrician.


Step 4: Determine specifications for final CAT. Data from Step 3 is analyzed to evaluate CAT specifications and determine most efficient algorithms using CAT simulation software such as CATSim.

Step 5: Publish live CAT. The adaptive test is published in a testing engine capable of fully adaptive tests based on IRT. Want to learn more about our one-of-a-kind model? Click here to read the seminal article by two of our psychometricians. More adaptive testing research is available here.

What do I need for adaptive testing?

Minimum requirements:

  • A large item bank piloted on at least 500 examinees
  • 1,000 examinees per year
  • Specialized in IRT calibration and CAT simulation software.
  • Staff with a Ph.D. in psychometrics or an equivalent level of experience. Or, leverage our internationally recognized expertise in the field.
  • Items (questions) that can be scored objectively correct/incorrect in real-time
  • An item banking system and CAT delivery platform
  • Financial resources: Because it is so complex, the development of a CAT will cost at least $10,000 (USD) — but if you are testing large volumes of examinees, it will be a significant positive investment. If you pay $20/hour for proctoring seats and cut a test from 2 hours to 1 hour for just 1,000 examinees… that’s a $20,000 savings.

Adative testing: Resources for further reading

Visit the links below to learn more about adaptive testing.  

How can I start developing a CAT?

Contact solutions@assess.com for a free account in our industry-leading CAT platform.

ASC has been empowering organizations to develop better assessments since 1979.  Curious as to how things were back then?  Below is a copy of our newsletter from 1988, long before the days of sharing news via email and social media!  Our platform at the time was named MICROCAT.  This later became modernized to FastTest PC (Windows), then FastTest Web, and is now being reincarnated yet again as Assess.ai.

Special thanks to Cliff Donath for finding and sharing!

[pdf-embedder url=”https://assess.com/wp-content/uploads/2019/02/MicroCATNewsAprill1988.pdf”]

Psychometrics is the cornerstone of any high-quality assessment program.  However, most organizations can’t afford an in-house Ph.D. psychometrician, which then necessitates the search for psychometric consulting.  Most organizations, when first searching, are new to the topic and not sure what role the psychometrician plays. 

In this article, we’ll talk about how psychometricians and their tools can help improve your assessments, whether you just want to check on test reliability or pursue the lengthy process of accreditation.

Why ASC?

Whether you are establishing or expanding a credentialing program, streamlining operations, or moving from paper to online testing, ASC has a proven track record of providing practical, cost-efficient solutions with uncompromising quality. We offer a free consultation with our team of experts to discuss your needs and determine which solutions are the best fit, including our enterprise SaaS platforms, consulting on sound psychometrics, or recommending you to one of our respected partners.

At the heart of our business are our people.

Our collaborative team of Ph.D. psychometricians, accreditation experts, and software developers have diverse experience developing solutions that drive best practices in assessment. This real-world knowledge enables us to consult your organization with solutions tailored specifically to your goals, timeline, and budget.

Comprehensive Solutions to Address Specific Measurement Problems

Much of psychometric consulting is project-based around solving a specific problem.  For example, you might be wondering how to set a cut score on a certification/licensure exam that is legally defensible and meets accreditation standards. 

This is a very specific issue, and the scientific literature has suggested a number of sound approaches.  Here are some of the topics where psychometricians can really help:

  • Test Design: Job Analysis & Blueprints
  • Standard and Cutscore Setting Studies
  • Item Writing and Review Workshops
  • Test and Item Statistical Analysis
  • Equating Across Years and Forms
  • Adaptive Testing Research
  • Test Security Evaluation
  • NCCA/ANSI Accreditation

Why psychometric consulting?

All areas of assessment can be smarter, faster, and fairer.

Develop Reliable and Valid Assessments
We’ll help you understand what needs to be done to develop defensible tests and how to implement them in a cost-efficient manner.  Much of the work revolves around establishing a sound test development cycle.

Increase Test Security
We have specific expertise in psychometric forensics, allowing you to flag suspicious candidates or groups in real-time, using our automated forensics report.

Achieve Accreditation
Our dedicated experts will assist in setting your organization up for success with NCCA/ANSI accreditation of professional certification programs.

Comprehensive Psychometric Analytics
We use CTT and IRT with principles of machine learning and AI to deeply understand your data and provide actionable recommendations.

We can help your organization develop and publish certification and licensure exams, based on best practices and accreditation standards, in a matter of months.

Effective assessments; Best practices

Item and Test Statistical Analysis
If you are doing this process at least annually, you are not meeting best practices or accreditation standards. But don’t worry, we can help! In addition to performing these analyses for you, you also have the option of running them yourself in our FastTest platform or using our psychometric software like Iteman and Xcalibre.

Job Analysis
How do you know what a professional certification test should cover?  Well, let’s get some hard data by surveying job incumbents. Knowing and understanding this information and how to use it is essential if you want to test people on whether they are prepared for the job or profession.

Cut score Studies (Standard Setting)
When you use sound psychometric practices like the modified-Angoff, Beuk Compromise, Bookmark, and Contrasting Groups methods, it will help you establish a cutscore that meets professional standards.

 

It’s all much easier if you use the right software!

Once we help you determine the best solutions for your organization, we can train you on best practices, and it’s extremely easy to use our software yourself.  Software like Iteman and Xcalibre is designed to replace much of the manual work done by psychometricians for the item and test analysis, and FastTest automates many aspects of test development and publishing.  We even offer free software like the Angoff Analysis Tool

However, our ultimate goal is your success: Assessment Systems is a full-service company that continues to provide psychometric consulting and support even after you’ve made a purchase. Our team of professionals is available to provide you with additional support at any point in time. We want to ensure you’re getting the most out of our products!  Click below to sign up for a free account in FastTest and see for yourself.

ProctorExam and ASC have partnered to offer a much more secure solution for online test administration, which more and more organizations will need in our changing world of technology and online learning. This is because security requires a multi-layered approach: in addition to streaming a webcam, there is also screen capture, and use of a 3rd camera.  Moreover, security also rests on the delivery engine, which should include functionality for screen lockdown, randomization, adaptive testing, LOFT, passwords, and more.  ASC has recently partnered with ProctorExam to leverage their innovative platform for remote proctoring, layering that level of security over the security already available in our test engine.

About ProctorExam

Since launching their services in late 2014, ProctorExam has stood out by offering unique and creative solutions to enhance assessment security. It has been a consistent innovator in the online education market with its innovative 360-degree monitoring with the users’ smart devices, and a full range of proctoring options. Over the past year, ProctorExam has been working to enhance their line of innovative services by introducing Proctoring Infrastructure as a Service. This new paradigm has already begun to decrease cost, increase transparency, and reduce reliance on third-party services.

Daniel Haven, Director of ProctorExam: “Due to the shared vision for test taking, we are very excited about working with Assessment Systems Corporation. In the future, we want to see whether we can share more lifecycle events to further automate the entire testing process and not necessarily only use communal data to understand whether a person is cheating or not, but also to further increase the quality of the test itself, by better understanding whether specific questions can be shorter or longer, whether they’re too difficult or not.”

About ASC

Assessment Systems Corporation, has been a leader in the assessment and testing industry since 1979, providing both world-class software for testing and psychometrics (with a focus on item response theory and computerized adaptive testing) and extensive consulting services for testing organizations. Assessment Systems was the first company to offer item banking and testing software to the public, in the form of the DOS-based MicroCAT adaptive testing system, nearly 30 years ago. For the past 10 years, they have been implementing this comprehensive testing program into a web-based system that is accessible anywhere, both for examinees and users. ASC’s delivery platform can be used by any type of assessment: certification, K12, higher education, workforce, medicine, and more.

David Weiss, President and CEO at Assessment Systems Corporation, states: “More and more clients will have access to the combined technology. The benefit of ProctorExam is that it is comparatively economical, so it’s allowing Assessment Systems Corporation to offer proctoring solutions to people who otherwise wouldn’t be able to do remote delivery. This allows us to open up new opportunities, especially in education – where there are a lot of high-stakes tests delivered.”

Estimated reading time: 6 minutes

Computerized adaptive testing (CAT) has been around since the 1970s and is well-known for the benefits it can provide, most notably that it can reduce testing time 50-90% with no loss of measurement precision.  Developing a sound, defensible CAT is not easy, but our goal is to make it as easy as possible – that is, everything you need is available in clean software UI and you never have to write a single line of code.

Here, we outline the software, data analysis, and project management steps needed to develop a computerized adaptive test that aligns with best practices and international standards.

This approach is based on Thompson and Weiss (2011) model, which refers there for a general treatment of CAT development, especially the use of simulation studies.  Also, this article assumes you have mastered the concepts of item response theory and CAT, including:

  • IRT models (e.g., 3PL, rating scale model)
  • Item response functions
  • Item information functions
  • Theta estimation
  • Conditional standard error of measurement
  • Item selection algorithms
  • Termination criterion.

 

If IRT is new to you, please visit these resources

https://assess.com/what-is-item-response-theory/

https://assess.com/how-do-i-implement-item-response-theory/

https://assess.com/what-do-dichotomous-and-polytomous-mean-in-irt/

If you have some background in IRT but CAT is new, please visit these resources

https://assess.com/monte-carlo-simulation-adaptive-testing/

https://assess.com/adaptive-testing/

Overview: Steps to develop an adaptive test

There are nine steps to developing a CAT on our industry-leading platform, fastest:

Step Work to be done Software
1 Perform feasibility and planning studies CATSim
2 Develop item bank FastTest
3 Pilot items on 100-2000 examinees FastTest
4 Perform item analysis and other due diligence Iteman/Xcalibre
5 IRT calibration Xcalibre
6 Upload IRT parameters into FastTest FastTest
7 Validity study CATSim
8 Publish CAT FastTest
9 Quality assurance FastTest

 

We’ll now talk a little more about each of these.

 

Perform feasibility and planning studies

The first step, before doing anything else, is to confirm that your assessment meets the basic requirements of CAT.  For example, you need to have a decent-sized item bank, data on hundreds of examinees (or the future opportunity), and items that are scoreable in real-time.  See this paper for a full discussion. 

 If there are no huddles, then the next step is to perform monte Carlo simulations that help you scope out the project, using the CATSim software.  For example, you might simulate CATs with three sizes of the item bank, so you have a better idea of how many items to write.

 

Develop item bank

Now that you have some idea of how many items you need and in which ranges of difficulty and/or content constraints, you can leverage the powerful item authoring functionality of FastTest, as well as the item review and workflow management to ensure that subject matter experts are performing quality assurance on each other.

 

Pilot items

Because IRT requires that you have data from real examinees to calibrate item difficulty, you need to get that data.  To do so, create test(s) in FastTest to deliver all your items in a matter that meets your practical situation. That is, some organizations have a captive audience and might be able to have 500 people take all 300 items in their bank next week.  Other organizations might need to create 4 linear forms of 100 items with some overlap. Others might be constrained to still use current test forms and only tack on 20 new items onto the end of every examinee’s test.

Of course, some of you might have existing data.  That is, you might have spreadsheets of data from a previous test delivery system, paper-based delivery, or perhaps even already have your IRT parameters from past efforts.  You can use those too.

If you do deliver the pilot phase with FastTest, you now need to export the data to be analyzed in psychometric analytic software.  FastTest makes it easy to export both the data matrix and the item metadata needed for Xcalibre’s control file.

 

Perform item analysis, DIF, and other due diligence

The purpose of this step is to ensure that items included in your future CAT are of high quality.  Any steps that your organization normally does to review item performance is still relevant. This typically includes a review of items with low point-biserial correlations (poor discrimination), items where more examinees selected a distractor than the correct option (key flags), high or low classical P values, and differential item functioning (DIF) flags.  Our Iteman software is designed exactly for this process. If you have a FastTest account the Iteman analysis report is now available at a single click. If not, Iteman is also available as a standalone program.

 

Calibrate with Xcalibre

Because CAT algorithms rely entirely on IRT parameters (unless you are doing special algorithms like diagnostic measurement models or measurement decision theory), we need to calculate the IRT parameters and get them into our testing platform.  If you have delivered all your items in a single block to examinees, like the example above with 500 people, then that single matrix can just be analyzed with Xcalibre. If you have multiple forms, LOFT, or the “tack-on” approach, you need to worry about IRT equating.

 

Upload IRT parameters into FastTest

Xcalibre will provide all the IRT parameters in a spreadsheet, in addition to the primary Word report.  Import them into your testing platform.  This will associate the IRT parameters with all the items in your CAT pool.  FastTest has functionality to streamline this process.

 

Validity study

Now that you have your final pool of items established, and calculated the IRT parameters, you need to establish the algorithms you are going to use to publish the CAT.  That is, you need to decide on the Initial Theta rule, Item Selection rule (including sub algorithms like content or exposure constraints), and Termination Criterion. To establish these, you need to perform more simulation studies, but now with your final bank as the input rather than a fake bank from the monte Carlo simulations.  The most important aspect is determining the tradeoff between test length and precision; a termination criterion that provides more precise scores will have longer tests, and you can control the exact extent with a CAT.

 

Publish CAT

Assemble a “test form” in FastTest that consists of all the items you intend to use in your CAT tool.  Then select CAT as the delivery method in the Test Options screen, and you’ll see a screen where you can input the results from your CATSim validity study for the three important CAT algorithms.

 

Quality assurance

Your CAT is now ready to go!  Before bringing in real students, however, we recommend that you take it a few times as QA.  Do so with certain students in mind, such as a very low student, a very high student, or one near the cut score (if you have one).  To peek under the hood at the CAT algorithm, you can export the Examinee Test Detail Report from FastTest, which provides an item-by-item picture of how the CAT proceeds.

Adaprive testing examinee report

Summary

As you can see, the development of an adaptive test is not easy and can take months even if you have all the software and expertise you need.  But for something so important, that could be used to make important decisions about people, this is absolutely warranted.

  However, if you have all the data you need today, there’s no reason that it should take months to develop an adaptive test – assessment platforms should make it easy enough for you to do so in an afternoon, which FastTest absolutely does.

Want to talk with one of our experts about applying this process to your exam?  Get in touch or sign up for a free account

The traditional Learning Management System (LMS) is designed to serve as a portal between educators and their learners. Platforms like Moodle are successful in facilitating cooperative online learning in a number of groundbreaking ways: course management, interactive discussion boards, assignment submissions, and delivery of learning content. While all of this is great, we’ve yet to see an LMS that implements best practices in assessment and psychometrics to ensure that medium or high stakes tests meet international standards.

To put it bluntly, LMS systems have assessment functionality that is usually good enough for short classroom quizzes but falls far short of what is required for a test that is used to award a credential.  A white paper on this topic is available here, but some examples include:

  • Treatment of items as reusable objects
  • Item metadata and historical use
  • Collaborative item review and versioning
  • Test assembly based on psychometrics
  • Psychometric forensics to search for non-independent test-taking behavior
  • Deeper score reporting and analytics

Assessment Systems is pleased to announce the launch of an easy-to-use bridge between FastTest and Moodle that will allow users to seamlessly deliver sound assessments from within Moodle while taking advantage of the sophisticated test development and psychometric tools available within FastTest. In addition to seamless delivery for learners, all candidate information is transferred to FastTest, eliminating the examinee import process.  The bridge makes use of the international Learning Tools Interoperability standards.

If you are already a FastTest user, watch a step-by-step tutorial on how to establish the connection, in the FastTest User Manual by logging into your FastTest workspace and selecting Manual in the upper right-hand corner. You’ll find the guide in Appendix N.

If you are not yet a FastTest user and would like to discuss how it can improve your assessments while still allowing you to leverage Moodle or other LMS systems for learning content, sign up for a free account here.

Computerized adaptive testing (CAT) is a powerful paradigm for delivering tests that are smarter, faster, and fairer than the traditional linear approach.  However, CAT is not without its challenges.  One is that it is a greedy algorithm that always selects your best items from the pool if it can.

The way that CAT researchers address this issue is with item exposure controls.  These are sub algorithms that are injected into the main item selection algorithm, to alter it from always using the best items. The Sympson-Hetter method is one such approach.

The simplest approach is called the randomesque method.

This selects from the top X items in terms of item information (a term from item response theory), usually for the first Y items in a test.  For example, instead of always selecting the top item, the algorithm finds the 3 top items and then randomly selects between those.

The Sympson-Hetter Method

A more sophisticated method is the Sympson-Hetter method.

Here, the user specifies a target proportion as a parameter for the selection algorithm.  For example, we might decide that we do not want an item seen by more than 75% of examinees.  So, every time that the CAT algorithm goes into the item pool to select a new item, we generate a random number between 0 and 1, which is then compared to the threshold.  If the number is between 0 and 0.75 in this case, we go ahead and administer the item.  If the number is from 0.75 to 1.0, we skip over it and go on to the next most informative item in the pool, though we then do the same comparison for that item.

Why do this?  It obviously limits the exposure of the item.  But just how much it limits it depends on the difficulty of the item.  A very difficult item is likely only going to be a candidate for selection for very high-ability examinees.  Let’s say it’s the top 4%… well, then the approach above will limit it to 3% of the sample overall, but 75% of the examinees in its neighborhood.

On the other hand, an item of middle difficulty is used not only for middle examinees but often for any examinee.  Remember, unless there are some controls, the first item for the test will be the same for everyone!  So if we apply the Sympson-Hetter rule to that item, it limits it to 75% exposure in a more absolute sense.

Because of this, you don’t have to set that threshold parameter to the same value for each item.  The original recommendation was to do some CAT simulation studies, then set the parameters thoughtfully for different items.  Items that are likely to be highly exposed (middle difficulty with high discrimination) might deserve a more strict parameter like 0.40.  On the other hand, that super-difficult item isn’t an exposure concern because only the top 4% of students see it anyway… so we might leave its parameter at 1.0 and therefore not limit it at all.

Is this the only method available?

No.  As mentioned, there’s that simple randomesque approach.  But there are plenty more.  You might be interested in this paper, this paper, or this paper.  The last one reviews the research literature from 1983 to 2005.

What is the original reference?

Sympson, J. B., & Hetter, R. D. (1985, October). Controlling item-exposure rates in computerized adaptive testing. Proceedings of the 27th annual meeting of the Military Testing Association (pp. 973–977). San Diego, CA: Navy Personnel Research and Development Center.

How can I apply this to my tests?

Well, you certainly need a CAT platform first.  Our platform at ASC allows this method right out of the box – that is, all you need to do is enter the target proportion when you publish your exam, and the Sympson-Hetter method will be implemented.  No need to write any code yourself!  Click here to sign up for a free account.