Test Security Archives

Addressing Pre-Knowledge in Exam Cheating

Nathan Thompson, PhDSeptember 9, 2024

In the realm of academic dishonesty and high-stakes exams such as Certification, the term “pre-knowledge” is an important concern in test security and validity. Understanding what pre-knowledge entails and its implications in exam cheating can offer valuable insights into how the integrity of tests and their scores can be compromised.

What is Pre-Knowledge in Exam Cheating?

Pre-knowledge refers to having advance access to information or materials that are not available to all students before an exam. This can manifest in various ways, from receiving the actual questions that will be on the test to obtaining detailed information about the exam’s content through unauthorized means. Essentially, it’s an unfair advantage that undermines the principles of a level playing field in assessments. Psychometricians would refer to it as a threat to validity.

How Pre-Knowledge Occurs

The concept of pre-knowledge encompasses several scenarios. For instance, a student might gain insight into the exam questions through illicit means such as hacking into a teacher’s email, collaborating with insiders, or even through more subtle means like overhearing discussions. This sort of advantage is particularly problematic because it negates the purpose of assessments, which is to evaluate a student’s understanding and knowledge acquired through study and effort.

Perhaps the greatest source of pre-knowledge is “brain dump” websites, where people who take the test go to soon afterwards and write all the questions they can remember, dumping their short-term memory. Over time, this can lead to large portions of the item bank being exposed to the public. Often, these sites are intentially located in countries that do not recognize the concept of cheating and do not have legal recourse to stop.

Implications of Pre-Knowledge

Pre-knowledge not only disrupts the fairness of the examination process but also poses significant challenges for test sponsors. Institutions must be vigilant and proactive in safeguarding against such breaches. Implementing robust security measures, such as secure testing environments and rigorous monitoring, is crucial to prevent the unauthorized dissemination of exam content.

Pre-knowledge and other forms of cheating can have far-reaching consequences beyond the immediate context of the exam. For instance, students who gain an unfair advantage may achieve higher grades, which can influence their future academic and career opportunities. This creates an inequitable situation where hard work and honest efforts are overshadowed by dishonesty.

Engaging in brain-dump activities is actually a form of intellectual property theft. Many organizations spend a lot of money to develop high-quality assessments, sometimes totaling in the thousands of dollars per item! Cheating in this manner will actually increase the costs of certification exams or educational opportunities. Moreover, the organizations can then pursue legal action against cheaters.

Most importantly, cheating on exams can invalidate scores and therefore the entire credential. If everyone is cheating and passing a certification exam, then the certification itself becomes worthless. So by participating in brain-dump activities, you are actually hurting yourself. And if the purpose of the test is to protect the public, such as ensuring that people working in healthcare fields are actually qualified, you are hurting the general public. If you are going into healthcare to help patients, you need to realize that brain-dump activity only ends up hurting patients in the end.

Tips to Prevent Pre-Knowledge

Educational institutions and other test sponsors play a pivotal role in addressing the issue of pre-knowledge. Here are some effective strategies to help prevent pre-knowledge and maintain the integrity of academic assessments. You might also benefit from this discussion on credentialing exams.

Secure Testing Environments: Ensure exams are administered in a controlled environment where access to unauthorized materials is restricted.
Monitor and Supervise: Use proctors and surveillance to oversee exam sessions and detect any suspicious behavior.
Randomize Questions: Utilize question banks with randomized questions to prevent the predictable pattern of exam content.
Strengthen Digital Security: Implement robust cybersecurity measures to protect exam content and prevent unauthorized access.
Educate Examinees: Foster a culture of academic integrity by educating students about the consequences of cheating and the importance of honest efforts.
Regularly Update Procedures: Review and update security measures and examination procedures regularly to address emerging threats.
Encourage Open Communication: Provide support and resources for students struggling with their studies to reduce the temptation to seek unfair advantages.
Strong NDA: This is one of the most important approaches. If someone signs a Non-Disclosure Agreement which tells them that if they engage in cheating that they might not only fail the exam but be blocked from the entire profession and subject to lawsuits, this can be a huge deterrent.
Multiple forms: Make multiple versions of the exam, and rotate them frequently.
LOFT/CAT: Linear on the fly testing (LOFT) makes nearly infinite parallel versions of the exam, increasing security. Computerized adaptive testing (CAT) creates a personalized exam for each candidate.
Scan the internet: You can search the internet for text strings from your items, helping you find the brain dump sites and address them.

What if it happens?

There are two aspects to this: finding that it happened, and then deciding what to do about it.

To find that it happened, there are several types of evidence.

Video – If there was proctoring, do we have video, perhaps of someone taking pictures of the exam with their phone?
Technology – Perhaps a post on a brain dump site was done with someone’s email address or IP address.
Eyewitness – You might get tipped off by another candidate that has higher integrity.
Psychometric forensics – There are some ways to find this. On the brain dump side, you might find candidates that don’t attempt all the items on the test but spend most of the time on a few… because they are memorizing them. On the other end, you might find candidates that get a high score in a very short amount of time. There are also much more advanced methods, as discussed in this book.

What should your organization do? First of all, you need a test security plan ahead of time. In that, you should lay out the processes as well as the penalties. This ensures that examinees have due process. Unfortunately, some organizations just stick their head in the sand; as long as candidates are paying the registration fees and the end stakeholders are not complaining, why rock the boat?

Summary

In conclusion, pre-knowledge in exam cheating represents a serious breach of academic integrity that undermines the educational process. It highlights the need for vigilance, robust security measures, and a culture of honesty within educational institutions. By understanding and addressing the nuances of pre-knowledge, educators and students can work together to uphold the values of fairness and integrity in academic assessments.

September 9, 2024/by Nathan Thompson, PhD

How Students Cheat in Online Exams and How to Catch Them

AdminMay 27, 2024

Student exam cheating has always been a major concern, even before the major shift to remote proctoring and online exams during the pandemic. There is a distinction between primary and secondary collusion, and there are specific collusion detection indices and methods to investigate aberrant testing behavior, such as

According to a study done in 2007, 60.8% of college students admitted to cheating while a whopping 95% of these admitted not getting caught! Another article also revealed that even renowned institutions such as Harvard and Yale turn out to be fragile when it came to exam cheating. This is just but a tip of the iceberg when exploring insights on academic infidelity.

In 2020, things took a turn for the worse as many institutions struggled to implement digital transformation as a response to the pandemic. Coupled with the common misconception that cheating in online exams is easier than traditional exams, everything became catastrophic.

However, this is not a true reflection of cheating in online exams. Is cheating in online exams easy as the students perceive it to be? No. Just like in-person exams, online exams have some loopholes that students can exploit for academic gains. But, these loopholes can be mitigated using the right online assessment software and strategies. Perhaps the most important technology is remote proctoring, which has seen an incredible boom during the pandemic.

This article provides two things:

How students cheat in online tests
Provide you with tips and ideas on how to deter and detect cheating

What is exam cheating?

Cheating refers to actual, intended, or attempted deception and/or dishonesty in relation to academic or employment-related assessments. It is only one part of the larger issue of test integrity; there are other issues, such as teacher help on K-12 summative tests or “faking good” on personality assessments. This in turn is part of the larger scope of academic dishonesty.

There are two pathways to dealing with academic dishonesty and cheating: Deter and Detect.

Deter: Make it harder to cheat, or put punishment policies in place that make examinees avoid cheating. Examples include lockdown browser and remote proctoring.
Detect: Find cheating that has already happened, such as psychometric forensics (download free software for that here), or reviewing a proctor video.

With exam security, experts follow the adage of “A pound of prevention is worth an ounce of cure” so deterrence is important. However, before going forward, let’s define academic dishonesty and cheating, and discuss some definitions.

Need an online exam platform with more security, or want to discuss security with a psychometrician?

Click here to get in touch.

7 ways of cheating in online exams and how to catch or deter them

1. Impersonation (proxy tester)

Impersonation refers to the act of asking or hiring someone else to take an exam on your behalf. This is not a new trick and has also been used in traditional assessments. In online assessments, the risks are even higher. Impersonation can take place in two stages of the examination delivery process; before and during the assessments.

Before the assessments, students share their login details with the imposter. The login details vary depending on the software in use. However, some online proctoring software easily makes this impossible by using advanced identification features such as biometrics or facial recognition systems. Impersonation during the exam process can take place in many ways. One trick used by students is providing remote access to their imposters through tools such as virtual machines. This sounds like something pulled out of a spy movie, but you wouldn’t believe what students are willing to do to avoid failure.

How to curb impersonation in online exams

Using authentication technologies such as biometric systems and two-factor authentication to verify registered candidates.

Use online assessment software with the AI-flagging feature. AI flagging feature enables instructors to spot and act on suspicious activities in exam video and audio recordings. Some suspicious behavior includes extra audio voices and unusual body language.

Use software such as BioSig-ID which uses keystroke analysis to identify metrics such as typing patterns, style, rhythm, and pressure. These are then used to create a unique persona that is almost impossible to replicate.

2. The ‘Old school’

The ‘old school’ is popular in traditional exams. It includes tricks such as writing the notes in your palms or sticky notes and using sign language to communicate with classmates. Many institutions make a mistake of assuming that these tricks cannot be used in online exams. However, these tricks can be used in online proctored exams, but remote proctoring can catch it.

How to deal with the ‘old-school tricks’

Most of the old-school tricks of cheating can be identified with online proctoring software. In many cases, AI-based flagging is sufficient. But for high-stakes exams, you definitely need a human proctor in real time, because that not only protects the integrity of that examinee’s score, but greatly deters them from stealing the test content.

3. Advanced tech gadgets and software

The advancement of technology has turned some students into ‘James Bond’ types of cheaters. They rely on sophisticated tech gadgets and software to cheat in online proctored exams. Here are some creative ways students use software and technology to try and outwit the system;

Tech gadgets such as smartwatches and microphones– These devices enable students to communicate about the exam with people from the outside. Smartwatches can also be used to store answers to the tests.
Using remote software– This is one of the most popular ways of cheating in online assessments. Students install remote software to connect them with help from the outside. Some even use web versions of applications such as WhatsApp to cheat in exams.
Projectors or cameras– This is another way students can cheat in online proctored exams. Students may set up cameras on their desks and have their family or friends project the answers in front of them.
Screen Mirroring– Using certain tools, students can grant monitor access to friends and family in the room, who can help them with the test.

How to stop the ‘tech-savvy’ students from cheating

Use lock-down browser to stop the student from surfing the web or using any software, apart from taking the exam.
Use AI flagging to recognize and act on suspicious behavior
Collaborate with a remote proctoring provider that offers audio and video proctoring services. You can also add a human proctor to the equation to increase assessment security. Good proctoring services will use two cameras, require room scans, lock down the computer, and other approaches to addressing tech-based cheating.

4. “Brain Dumps” and sharing of test content

In many cases, this form of student exam cheating can be referred to as the ‘charity’ of academic infidelity. This is popular in traditional exams where students share exam questions with others who have not taken the exam yet, because they feel they are being helpful – even if it could end up hurting themselves. Candidates may take screenshots or use cameras to record questions and share them with others, or simply try to memorize items.

How to curb sharing of test content

Use ASC’s lockdown browser functionality to eliminate the ability to take screenshots and screen recording. Proctoring services will flag if students are taking pictures with their phone. Psychometric forensics will look for unusual time patterns.

5. The ‘Water or Bathroom’ break

A student pretends to have a ‘bathroom emergency’ or ‘bad connection’ and the first thing they do after logging off the system is getting access to a device or textbook to cheat. To achieve high standards of academic integrity, it’s important for institutions not to take chances.

How to deal with the ‘Water or Bathroom Break’

This is a tricky one, as there is no knowing when the excuses are genuine or not. However, the institutions should try to come up with strict examination policies for such circumstances. For instance, the time for bathroom breaks should be limited, such as there being a 200 item test with two 100-item sections and a 10 minute break between, so there is no chance to come back to items.

6. Help from family, experts, and friends

This is a method that many students consider ‘safe’. There are several ways family, experts, and friends can help students cheat in online assessments. One way is having a family member or friend in the same room a student is taking the exam. The student can share the questions and the friend or family member can browse the internet and provide answers.

Another way students can cheat is through ‘expert tutors’. Students can hire tutors to help them with the exams or take them on their behalf. In fact, there has been an increased demand for online academic tutors.

How to stop students from getting external help

Tighten up your authentication process by using technologies such as biometrics systems and two-factor authentication. This prevents students from receiving external help from experts.
Utilize lockdown browser to make sure that students don’t have access to remote software or have the freedom to surf the internet.
Use AI flagging to make sure that the candidates don’t display suspicious behavior.

7. Googling or ChatGPT-ing the answer

Doesn’t access to a computer and the internet provide students with endless opportunities to cheat in online proctored tests? Yes, if the test is not proctored or locked down. Fortunately, this method is relatively easy to address.

How to stop students from using the internet during an exam.

Use ASC’s lockdown browser function to stop students from accessing other websites or resources on personal devices. Utilize proctoring services. Do not allow cell phones to be present at the desk.

Conclusion

Student exam cheating remains common. These are some of the popular ways in which students cheat on exams. However, these tricks and hacks should not scare you from reaping the benefits of online assessments. In fact, by using the solutions suggested in this article, we believe you can reduce the risks of cheating and achieve high standards of academic integrity. Assessment Systems regularly participates in the Conference on Test Security (COTS), including in 2015, 2016, and 2017, to stay at the forefront of the latest strategies and technologies for preventing cheating in assessments.

Ready to minimize and deter student cheating in your online proctored exams? Contact us today and get a free consultation!

May 27, 2024/by Admin

How to Pass My Online Proctored Exam

Nathan Thompson, PhDMarch 25, 2024

Online proctored exams are assessments that you can take at your home or office, with a video stream of you taking the exam being used to ensure its integrity. The primary purpose is to increase convenience and flexibility, by letting people take their exams at any quiet place they like instead of having to commute to a university or professional testing center.

What is online proctoring?

The proctoring process typically involves the use of a webcam, microphone, and screen sharing to observe the test environment. The service either uses humans or software to monitor for suspicious activities, such as unusual eye movements, background noise, an extra person there to help, or the presence of unauthorized materials. Live proctors are more secure in that they can intervene if they detect any irregularities in real time. Online proctored exams represent a significant advancement in educational technology, balancing the need for convenience with the importance of fair and secure assessment practices.
Online proctored exams have existed for at least a decade, but they have become far more common since the onset of the pandemic. From K-12 assessments to university to pre-employment to high stakes, most people now accept that online proctored exams are part of life. If you are scheduled to take one and it is your first time, you’re likely curious about the experience. This post will provide a brief intro about what to expect and how to do as well as you can.

Why we need online proctored exams

This might seem like a total headache for you, but it is actually very important.

First of all, it ensures that everyone has the same opportunity to succeed, based on how much they know the material. It levels the playing field for all test-takers. How would you feel if you take a test to apply for a job, and you don’t cheat but the next person in line does?

Second, it also protects the integrity of the test. Posting exam items on the internet is downright theft, especially for high-stakes exams where thousands of dollars are spent on each question

Online proctored exams also offer significant benefits. They provide accessibility for students who may have geographical or physical constraints, and they reduce the logistical challenges and costs associated with traditional exam centers. Moreover, they allow institutions to uphold rigorous standards of academic integrity, even in a remote setting.

Types of online proctored exams

First, you must understand that there are actually several types of online proctored exams, and they can differ widely in the experience that you should expect!!!

Type	Security Level	What to expect
Live Stream	High	You will have a live human watching your video stream at all times. Sometimes, there are two video streams: your computer, and your phone on a shelf (see example here). This human is usually a trained proctor, but sometimes it is just university staff. This approach is the most secure, because not only can they catch you cheating, but they can stop the test immediately.
Record & Review:	Medium-High	Your video will be recorded onto a server, for watching later by a human. They will record any flags they see.
Autoproctored/AI	Medium	Your video is recorded onto a server and “watched” by an AI algorithm, which flags you for clearly definable events: no faces on the screen, two faces on the screen, human voices, etc.
Lockdown Browser	Low	There is no video, but the test is delivered in a browser that locks down your computer so that you cannot visit other websites, run screen capture software, etc. This is usually used in conjunction with the other three, and often with in-person proctoring as well.

Sometimes, there is a blend of these as well. You might have a live human to verify your ID and check you in, and then after that it is recorded for later review by human or AI.

How to pass an online proctored exam

Read that candidate/student handbook: This actually VERY important. It will not only tell you what to expect, but also tell you important information about the exam itself.

Practice test: If available, make sure to do this. It will help you understand the process, and feel much more confident come Exam Day.

Download beforehand: Some software for an online proctored exam will require you to download something, such as a secure browser or Chrome plugin. Make sure to do this before Exam Day. Sometimes, it is part of the Practice Test.

Watch instructional videos: Videos like the example above will help you see what to expect.

System check: Similarly, some software to deliver an online proctored exam will perform an automated system check to make sure that you have enough internet bandwidth, that your video is streaming well, etc. Do this before Exam Day, and sometimes it is also part of the Practice Test.

Know the rules: Make sure you are aware of the rules for your particular exam. Are you allowed a bottle of water? Bathroom breaks? Open-book?

Prepare the area: Once you know the rules, make sure your desk area is ready. Do you need to clean the top of your desk from all books and papers? Obtain an external webcam or microphone? Ask your roommates to leave? There are many things like this which might be required, and more than you can do in 30 seconds as you log into your exam.

Have water: If you are allowed water, make sure you have it ready, just in case.

Study: YES!!!!! Obviously, this is more important than anything else… if you actually know the material, you are more likely to pass. Cheating is not the way to go; it isn’t worth the risk if you could just take that effort and put it into actually studying.

March 25, 2024/by Nathan Thompson, PhD

How do I develop a test security plan?

Nathan Thompson, PhDJanuary 5, 2023

A test security plan (TSP) is a document that lays out how an assessment organization address security of its intellectual property, to protect the validity of the exam scores. If a test is compromised, the scores become meaningless, so security is obviously important. The test security plan helps an organization anticipate test security issues, establish deterrent and detection methods, and plan responses. It can also include validity threats not security-related, such as how to deal with examinees that have low motivation. Note that it is not limited to delivery; it can often include topics like how to manage item writers.

Since the first tests were developed 2000 years ago for entry into the civil service of Imperial China, test security has been a concern. The reason is quite straightforward: most threats to test security are also validity threats. The decisions we make with test scores could therefore be invalid, or at least suboptimal. It is therefore imperative that organizations that use or develop tests should develop a TSP.

Why do we need a test security plan?

There are several reasons to develop a test security plan. First, it drives greater security and therefore validity. The TSP will enhance the legal defensibility of the testing program. It helps to safeguard the content, which is typically an expensive investment for any organization that develops tests themselves. If incidents do happen, they can be dealt with more swiftly and effectively. It helps to manage all the security-related efforts.

The development of such a complex document requires a strong framework. We advocate a framework with three phases: planning, implementation, and response. In addition, the TSP should be revised periodically.

Phase 1: Planning

The first step in this phase is to list all potential threats to each assessment program at your organization. This could include harvesting of test content, preknowledge of test content from past harvesters, copying other examinees, proxy testers, proctor help, and outside help. Next, these should be rated on axes that are important to the organization; a simple approach would be to rate on potential impact to score validity, cost to the organization, and likelihood of occurrence. This risk assessment exercise will help the remainder of the framework.

Next, the organization should develop the test security plan. The first piece is to identify deterrents and procedures to reduce the possibility of issues. This includes delivery procedures (such as a lockdown browser or proctoring), proctor training manuals, a strong candidate agreement, anonymous reporting pathways, confirmation testing, and candidate identification requirements. The second piece is to explicitly plan for psychometric forensics.

This can range from complex collusion indices based on item response theory to simple flags, such as a candidate responding to a certain multiple choice option more than 50% of the time or obtaining a score in the top 10% but in the lowest 10% of time. The third piece is to establish planned responses. What will you do if a proctor reports that two candidates were copying each other? What if someone obtains a high score in an unreasonably short time?

What if someone obviously did not try to pass the exam, but still sat there for the allotted time? If a candidate were to lose a job opportunity due to your response, it helps you defensibility to show that the process was established ahead of time with the input of important stakeholders.

Phase 2: Implementation

The second phase is to implement the relevant aspects of the Test Security Plan, such as training all proctors in accordance with the manual and login procedures, setting IP address limits, or ensuring that a new secure testing platform with lockdown is rolled out to all testing locations. There are generally two approaches. Proactive approaches attempt to reduce the likelihood of issues in the first place, and reactive methods happen after the test is given. The reactive methods can be observational, quantitative, or content-focused. Observational methods include proctor reports or an anonymous tip line. Quantitative methods include psychometric forensics, for which you will need software like SIFT. Content-focused methods include automated web crawling.

Both approaches require continuous attention. You might need to train new proctors several times per year, or update your lockdown browser. If you use a virtual proctoring service based on record-and-review, flagged candidates must be periodically reviewed. The reactive methods are similar: incoming anonymous tips or proctor reports must be dealt with at any given time. The least continuous aspect is some of the psychometric forensics, which depend on a large-scale data analysis; for example, you might gather data from tens of thousands of examinees in a testing window and can only do a complete analysis at that point, which could take several weeks.

Phase 3: Response

The third phase, of course, to put your planned responses into motion if issues are detected. Some of these could be relatively innocuous; if a proctor is reported as not following procedures, they might need some remedial training, and it’s certainly possible that no security breach occurred. The more dramatic responses include actions taken against the candidate. The most lenient is to provide a warning or simply ask them to retake the test. The most extreme methods include a full invalidation of the score with future sanctions, such as a five-year ban on taking the test again, which could prevent someone from entering a profession for which they spent 8 years and hundreds of thousands of dollars in educative preparation.

What does a test security plan mean for me?

It is clear that test security threats are also validity threats, and that the extensive (and expensive!) measures warrant a strategic and proactive approach in many situations. A framework like the one advocated here will help organizations identify and prioritize threats so that the measures are appropriate for a given program. Note that the results can be quite different if an organization has multiple programs, from a practice test to an entry level screening test to a promotional test to a professional certification or licensure.

Another important difference between test sponsors/publishers and test consumers. In the case of an organization that purchases off-the-shelf pre-employment tests, the validity of score interpretations is of more direct concern, while the theft of content might not be an immediate concern. Conversely, the publisher of such tests has invested heavily in the content and could be massively impacted by theft, while the copying of two examinees in the hiring organization is not of immediate concern.

In summary, there are more security threats, deterrents, procedures, and psychometric forensic methods than can be discussed in one blog post, so the focus here rather on the framework itself. For starters, start thinking strategically about test security and how it impacts their assessment programs by using the multi-axis rating approach, then begin to develop a Test Security Plan. The end goal is to improve the health and validity of your assessments.

Want to implement some of the security aspects discussed here, like online delivery lockdown browser, IP address limits, and proctor passwords?

Sign up for a free account in FastTest!

January 5, 2023/by Nathan Thompson, PhD

Item Parameter Drift

Laila Issayeva M.Sc.November 4, 2022

Item parameter drift (IPD) refers to the phenomenon in which the parameter values of a given test item change over multiple testing occasions within the item response theory (IRT) framework. This phenomenon is often relevant to student progress monitoring assessments where a set of items is used several times in one year, or across years, to track student growth; the observing of trends in student academic achievements depends upon stable linking (anchoring) between assessment moments over time, and if the item parameters are not stable, the scale is not stable and time-to-time comparisons are not either. Some psychometricians consider IPD as a special case of differential item functioning (DIF), but these two are different issues and should not be confused with each other.

Reasons for Item Parameter Drift

IRT modeling is attractive for assessment field since its property of item parameter invariance to a particular sample of test-takers which is fundamental for their estimation, and that assumption enables important things like strong equating of tests across time and the possibility for computerized adaptive testing. However, item parameters are not always invariant. There are plenty of reasons that could stand behind IPD. One possibility is curricular changes based on assessment results or instruction that is more concentrated. Other feasible reasons are item exposure, cheating, or curricular misalignment with some standards. No matter what has led to IPD, its presence can cause biased estimates of student ability. In particular, IPD can be highly detrimental in terms of reliability and validity in case of high-stakes examinations. Therefore, it is crucial to detect item parameter drift when anchoring assessment occasions over time, especially when the same anchor items are used repeatedly.

Perhaps the simplest example is item exposure. Suppose a 100-item test is delivered twice per year, with 20 items always remaining as anchors. Eventually students will share memories and the topics of those will become known. More students will get them correct over time, making the items appear easier.

Identifying IPD

There are several methods of detecting IPD. Some of them are simpler because they do not require estimation of anchoring constants, and some of them are more difficult due to the need of that estimation. Simple methods include the “3-sigma p-value”, the “0.3 logits”, and the “3-sigma IRT” approaches. Complex methods involve the “3-sigma scaled IRT”, the “Mantel-Haenszel”, and the “area between item characteristic curves”, where the last two approaches are based on consideration that IPD is a special case of DIF, and therefore there is an opportunity to draw upon a massive body of existing research on DIF methodologies.

Handling IPD

Even though not all psychometricians think that removal of outlying anchor items is the best solution for item parameter drift, if we do not eliminate drifting items from the process of equating test scores, they will affect transformations of ability estimates, not only item parameters. Imagine that there is an examination, which classifies examinees as either failing or passing, or into four performance categories; then in case of IPD, 10-40% of students could be misclassified. In high-stakes testing occasions where classification of examinees implies certain sanctions or rewards, IPD scenarios should be minimized as much as possible. As soon as it is detected that some items exhibit IPD, these items should be referred to the subject-matter experts for further investigation. Otherwise, if there is a need in a faster decision, such flagged anchor items should be removed immediately. Afterwards, psychometricians need to re-estimate linking constants and evaluate IPD again. This process should repeat unless none of the anchor items shows item parameter drift.

November 4, 2022/by Laila Issayeva M.Sc.

Certification Exam Delivery: Guidelines For Success

Nathan Thompson, PhDAugust 21, 2022

Certification exam delivery is the process of administering a certification test to candidates. This might seem straightforward, but it is surprisingly complex. The greater the scale and the stakes, the more potential threats and pitfalls.

For starters, you need to make sure the exam was developed well in the first place. Learn more about that here. I also suggest reading another blog post on the topic called ‘Certification Exam Administration and Proctoring‘ for a comprehensive understanding.

1. Determine Approach to Certification Exam Delivery

The first step is to determine how you want to deliver the certification exam. Here are your options for exam delivery.

Test Centers
- Your locations
- Third party
Mode of delivery
- Paper
- Online
  - Unproctored
  - Remotely Proctored (Recorded)
  - Remotely Proctored (Live)
  - Multi-modal (a mix of the above)

Which approach to use for certification exam delivery depends on several factors.

Are you going to seek accreditation?
Do you have an easy way to make your own locations? One example is that you have quarterly regional conferences for your profession, where you could simply get a side room to deliver your test to candidates since they will already be there. Another is that most of your candidates are coming from training programs at universities, and you are able to use classrooms at those universities.
What are the stakes of the exam? In rare cases, it can be unproctored online. Low- to medium-stakes can be delivered with recorded video proctoring. The highest stakes exams will almost always be at test centers with in-person proctoring, including strong measures like fingerprinting and retinal scanning. Learn more about remote proctoring.
How many exams are you planning to deliver? If a large volume, higher-cost approaches might be out of your budget.
What is your testing cycle like? Some exams are on-demand year-round, while some might be one or two Saturdays per year.

2. Shop for Vendors

Once you have some idea what you are looking for, start shopping for vendors that provide services for certification exam delivery, development, and scoring. In some cases, you might not settle on a certain approach right away, and that’s OK. See what is out there and compare prices. Perhaps the cost of Live Remote Proctoring is more affordable than you anticipated, and you can upgrade to that.

Besides a simple Google search some good places to start are the member listings of the Association of Test Publishers and the Institute for Credentialing Excellence.

3. Create Policies and Documentation for Certification Exam Delivery

Once you have finalized your vendors, you need to write policies and documentation around them. For example, if your vendor has a certain login page for proctoring (we have ascproctor.com), you should take relevant screenshots and write up a walkthrough so candidates know what to expect. Much of this should go into your Candidate Handbook. Some of the things to cover:

How to prepare for the exam
How to take a practice test
What is allowed during the exam
What is not allowed
ID needed and the check-in process
Details on specific locations (if using locations)
Time limits and other practical considerations in the exam

4. Let Everyone Know

Once you have written up everything, make sure all the relevant stakeholders know. Publish the new Candidate Handbook and announce to the world. Send emails to all upcoming candidates with instructions and an opportunity for a practice exam. Put a link on your homepage. Get in touch with all the training programs or universities in your field. Make sure that everyone has ample opportunity to know about the new process!

5. Roll Out

Finally, of course, you can implement the new approach to certification exam delivery. You might launch a new certification exam from scratch, or perhaps you are moving one from paper to online with remote proctoring, or some other change. Either way, you need a date to start using it and a change management process. The good news is that, even though it’s probably a lot of work to get here, the new approach is probably going to save you time and money in the long run. Roll it out!

Contact Us To Speak to a Psychometrician

August 21, 2022/by Nathan Thompson, PhD

HR Assessment for Pre-Employment: Approaches and Solutions

AdminJune 30, 2022

HR assessment is a critical part of the HR ecosystem, used to select the best candidates with pre-employment testing, assess training, certify skills, and more. But there is a huge range in quality, as well as a wide range in the type of assessment that it is designed for. This post will break down the different approaches and help you find the best solution.

HR assessment platforms help companies create effective assessments, thus saving valuable resources, improving candidate experience & quality, providing more accurate and actionable information about human capital, and reducing hiring bias. But, finding software solutions that can help you reap these benefits can be difficult, especially because of the explosion of solutions in the market. If you are lost on which tools will help you develop and deliver your own HR assessments, this guide is for you.

What is HR assessment?

HR assessment is a comprehensive process used by human resources professionals to evaluate various aspects of potential and current employees’ abilities, skills, and performance. This process encompasses a wide range of tools and methodologies designed to provide insights into an individual’s suitability for a role, their developmental needs, and their potential for future growth within the organization.

The primary goal of HR assessment is to make informed decisions about recruitment, employee development, and succession planning. During the recruitment phase, HR assessments help in identifying candidates who possess the necessary competencies and cultural fit for the organization.

There are various types of assessments used in HR. Here are four main areas, though this list is by no means exhaustive.

Pre-employment tests to select candidates
Post-training assessments
Certificate or certification exams (can be internal or external)
360-degree assessments and other performance appraisals

Pre-employment tests

Finding good employees in an overcrowded market is a daunting task. In fact, according to the Harvard Business Review, 80% of employee turnover is attributed to poor hiring decisions. Bad hires are not only expensive, but can also adversely affect cultural dynamics in the workforce. This is one area where HR assessment software shows its value.

There are different types of pre-employment assessments. Each of them achieves a different goal in the hiring process. The major types of pre-employment assessments include:

Personality tests: Despite rapidly finding their way into HR, these types of pre-employment tests are widely misunderstood. Personality tests answer questions in the social spectrum. One of the main goals of these tests is to quantify the success of certain candidates based on behavioral traits.

Aptitude tests: Unlike personality tests or emotional intelligence tests which tend to lie on the social spectrum, aptitude tests measure problem-solving, critical thinking, and agility. These types of tests are popular because can predict job performance than any other type because they can tap into areas that cannot be found in resumes or job interviews.

Skills Testing: The kinds of tests can be considered a measure of job experience; ranging from high-end skills to low-end skills such as typing or Microsoft excel. Skill tests can either measure specific skills such as communication or measure generalized skills such as numeracy.

Emotional Intelligence tests: These kinds of assessments are a new concept but are becoming important in the HR industry. With strong Emotional Intelligence (EI) being associated with benefits such as improved workplace productivity and good leadership, many companies are investing heavily in developing these kinds of tests. Despite being able to be administered to any candidates, it is recommended they be set aside for people seeking leadership positions, or those expected to work in social contexts.

Risk tests: As the name suggests, these types of tests help companies reduce risks. Risk assessments offer assurance to employers that their workers will commit to established work ethics and not involve themselves in any activities that may cause harm to themselves or the organization. There are different types of risk tests. Safety tests, which are popular in contexts such as construction, measure the likelihood of the candidates engaging in activities that can cause them harm. Other common types of risk tests include Integrity tests.

Post-training assessments

This refers to assessments that are delivered after training. It might be a simple quiz after an eLearning module, up to a certification exam after months of training (see next section). Often, it is somewhere in between. For example you might take an afternoon sit through a training course, after which you take a formal test that is required to do something on the job. When I was a high school student, I worked in a lumber yard, and did exactly this to become an OSHA-approved forklift driver.

Certificate or certification exams

Sometimes, the exam process can be high-stakes and formal. It is then a certificate or certification, or sometimes a licensure exam. More on that here. This can be internal to the organization, or external.

Internal certification: The credential is awarded by the training organization, and the exam is specifically tied to a certain product or process that the organization provides in the market. There are many such examples in the software industry. You can get certifications in AWS, SalesForce, Microsoft, etc. One of our clients makes MRI and other medical imaging machines; candidates are certified on how to calibrate/fix them.

External certification: The credential is awarded by an external board or government agency, and the exam is industry-wide. An example of this is the SIE exams offered by FINRA. A candidate might go to work at an insurance company or other financial services company, who trains them and sponsors them to take the exam in hopes that the company will get a return by the candidate passing and then selling their insurance policies as an agent. But the company does not sponsor the exam; FINRA does.

360-degree assessments and other performance appraisals

Job performance is one of the most important concepts in HR, and also one that is often difficult to measure. John Campbell, one of my thesis advisors, was known for developing an 8-factor model of performance. Some aspects are subjective, and some are easily measured by real-world data, such as number of widgets made or number of cars sold by a car salesperson. Others involve survey-style assessments, such as asking customers, business partners, co-workers, supervisors, and subordinates to rate a person on a Likert scale. HR assessment platforms are needed to develop, deliver, and score such assessments.

The Benefits of Using Professional-Level Exam Software

Now that you have a good understanding of what pre-employment and other HR tests are, let’s discuss the benefits of integrating pre-employment assessment software into your hiring process. Here are some of the benefits:

Saves Valuable resources

Unlike the lengthy and costly traditional hiring processes, pre-employment assessment software helps companies increase their ROI by eliminating HR snugs such as face-to-face interactions or geographical restrictions. Pre-employment testing tools can also reduce the amount of time it takes to make good hires while reducing the risks of facing the financial consequences of a bad hire.

Supports Data-Driven Hiring Decisions

Data runs the modern world, and hiring is no different. You are better off letting complex algorithms crunch the numbers and help you decide which talent is a fit, as opposed to hiring based on a hunch or less-accurate methods like an unstructured interview. Pre-employment assessment software helps you analyze assessments and generate reports/visualizations to help you choose the right candidates from a large talent pool.

Improving candidate experience

Candidate experience is an important aspect of a company’s growth, especially considering the fact that 69% of candidates admitting not to apply for a job in a company after having a negative experience. Good candidate experience means you get access to the best talent in the world.

Elimination of Human Bias

Traditional hiring processes are based on instinct. They are not effective since it’s easy for candidates to provide false information on their resumes and cover letters. But, the use of pre-employment assessment software has helped in eliminating this hurdle. The tools have leveled the playing ground, and only the best candidates are considered for a position.

What To Consider When Choosing HR assessment tools

Now that you have a clear idea of what pre-employment tests are and the benefits of integrating pre-employment assessment software into your hiring process, let’s see how you can find the right tools.

Here are the most important things to consider when choosing the right pre-employment testing software for your organization.

Ease-of-use

The candidates should be your top priority when you are sourcing pre-employment assessment software. This is because the ease of use directly co-relates with good candidate experience. Good software should have simple navigation modules and easy comprehension.

Here is a checklist to help you decide if a pre-employment assessment software is easy to use;

Are the results easy to interpret?
What is the UI/UX like?
What ways does it use to automate tasks such as applicant management?
Does it have good documentation and an active community?

Tests Delivery and Remote Proctoring

Good online assessment software should feature good online proctoring functionalities. This is because most remote jobs accept applications from all over the world. It is therefore advisable to choose a pre-employment testing software that has secure remote proctoring capabilities. Here are some things you should look for on remote proctoring;

Does the platform support security processes such as IP-based authentication, lockdown browser, and AI-flagging?
What types of online proctoring does the software offer? Live real-time, AI review, or record and review?
Does it let you bring your own proctor?
Does it offer test analytics?

Test & data security, and compliance

Defensibility is what defines test security. There are several layers of security associated with pre-employment test security. When evaluating this aspect, you should consider what pre-employment testing software does to achieve the highest level of security. This is because data breaches are wildly expensive.

The first layer of security is the test itself. The software should support security technologies and frameworks such as lockdown browser, IP-flagging, and IP-based authentication.

The other layer of security is on the candidate’s side. As an employer, you will have access to the candidate’s private information. How can you ensure that your candidate’s data is secure? That is reason enough to evaluate the software’s data protection and compliance guidelines.

A good pre-employment testing software should be compliant with certifications such as GDRP. The software should also be flexible to adapt to compliance guidelines from different parts of the world.

Questions you need to ask;

What mechanisms does the software employ to eliminate infidelity?
Is their remote proctoring function reliable and secure?
Are they compliant with security compliance guidelines including ISO, SSO, or GDPR?
How does the software protect user data?

Psychometrics

Psychometrics is the science of assessment, helping to drive accurate scores from defensible tests, as well as making them more efficient, reducing bias, and a host of other benefits. You should ensure that your solution supports the necessary level of psychometrics. Some suggestions:

Storing item statistics in the item banker
Managing modified-Angoff studies
Using test statistics for form assembly, such as the test information function
Support for item response theory
Adaptive testing

User experience

A good user experience is a must-have when you are sourcing any enterprise software. A new age pre-employment testing software should create user experience maps with both the candidates and employer in mind. Some ways you can tell if a software offers a seamless user experience includes;

User-friendly interface
Simple and easy to interact with
Easy to create and manage item banks
Clean dashboard with advanced analytics and visualizations

Customizing your user-experience maps to fit candidates’ expectations attracts high-quality talent.

Scalability and automation

With a single job post attracting approximately 250 candidates, scalability isn’t something you should overlook. A good pre-employment testing software should thus have the ability to handle any kind of workload, without sacrificing assessment quality.

It is also important you check the automation capabilities of the software. The hiring process has many repetitive tasks that can be automated with technologies such as Machine learning, Artificial Intelligence (AI), and robotic process automation (RPA).

Here are some questions you should consider in relation to scalability and automation;

Does the software offer Automated Item Generation (AIG)?
How many candidates can it handle?
Can it support candidates from different locations worldwide?

Reporting and analytics

A good pre-employment assessment software will not leave you hanging after helping you develop and deliver the tests. It will enable you to derive important insight from the assessments.

The analytics reports can then be used to make data-driven decisions on which candidate is suitable and how to improve candidate experience. Here are some queries to make on reporting and analytics.

Does the software have a good dashboard?
What format are reports generated in?
What are some key insights that prospects can gather from the analytics process?
How good are the visualizations?

Customer and Technical Support

Customer and technical support is not something you should overlook. A good pre-employment assessment software should have an Omni-channel support system that is available 24/7. This is mainly because some situations need a fast response. Here are some of the questions your should ask when vetting customer and technical support;

What channels of support does the software offer/How prompt is their support?
How good is their FAQ/resources page?
Do they offer multi-language support mediums?
Do they have dedicated managers to help you get the best out of your tests?

Conclusion

Finding the right HR assessment software is a lengthy process, yet profitable in the long run. We hope the article sheds some light on the important aspects to look for when looking for such tools. Also, don’t forget to take a pragmatic approach when implementing such tools into your hiring process.

Are you stuck on how you can use pre-employment testing tools to improve your hiring process? Feel free to contact us and we will guide you on the entire process, from concept development to implementation. Whether you need off-the-shelf tests or a comprehensive platform to build your own exams, we can provide the guidance you need. We also offer free versions of our industry-leading software FastTest and Assess.ai – visit our Contact Us page to get started!

If you are interested in delving deeper into leadership assessments, you might want to check out this blog post. For more insights and an example of how HR assessments can fail, check out our blog post called Public Safety Hiring Practices and Litigation. The blog post titled Improving Employee Retention with Assessment: Strategies for Success explores how strategic use of assessments throughout the employee lifecycle can enhance retention, build stronger teams, and drive business success by aligning organizational goals with employee development and engagement.

June 30, 2022/by Admin

AI Remote Proctoring: How To Choose A Solution

Nathan Thompson, PhDMay 17, 2022

AI remote proctoring has seen an incredible increase of usage during the COVID pandemic. ASC works with a very wide range of clients, with a wide range of remote proctoring needs, and therefore we partner with a range of remote proctoring vendors that match to our clients’ needs, and in some cases bring on a new vendor at the request of a client.

At a broad level, we recommend live online proctoring for high-stakes exams such as medical certifications, and AI remote proctoring for medium-stakes exams such as pre-employment assessment. Suppose you have decided that you want to find a vendor for AI remote proctoring. How can you start evaluating solutions?

Well, it might be surprising, but even within the category of “AI remote proctoring” there can be substantial differences in the functionality provided, and you should therefore closely evaluate your needs and select the vendor that makes the most sense – which is the approach we at ASC have always taken. We have also prepared a list of the best online proctoring software platforms for your use.

What is AI Remote Proctoring?

AI remote proctoring is a cutting-edge technology that uses artificial intelligence to monitor and supervise online examinations. Unlike traditional in-person proctoring, which requires human invigilators to oversee exams, AI remote proctoring leverages advanced algorithms to ensure the integrity and security of the testing process. This system typically employs a combination of facial recognition, screen monitoring, and behavior analysis to detect any suspicious activity during the exam.

The main benefits of using AI for remote proctoring

One of the primary advantages of AI remote proctoring is its ability to provide a scalable and flexible solution for educational institutions, certification bodies, and corporations. It allows students and test-takers to complete their exams from the comfort of their own homes or any other location with an internet connection. This not only reduces logistical challenges but also opens up opportunities for a wider range of candidates to participate in assessments.

Moreover, AI remote proctoring enhances the fairness and accuracy of the examination process. The technology can continuously analyze multiple data points in real time, identifying potential instances of cheating such as looking away from the screen, using unauthorized devices, or receiving external help. These automated systems can flag suspicious behavior for further review by human proctors, ensuring a robust and reliable evaluation process.

In addition to maintaining the integrity of the exams, AI remote proctoring also respects the privacy of test-takers. Many systems are designed with stringent data protection measures to ensure that personal information and exam data are securely handled and stored. This combination of security, flexibility, and reliability makes AI remote proctoring an increasingly popular choice for modern educational and professional assessments.

How does AI remote proctoring work?

Because this approach is fully automated, we need to train machine learning models to look for things that we would generally consider to be a potential flag. These have to be very specific! Here are some examples:

Two faces in the screen
No faces in the screen
Voices detected
Black or white rectangle approximately 2-3 inches by 5 inches (phone is present)
Face looking away or down
White rectangle approximately 8 inches by 11 inches (notebook or extra paper is present)

These are continually monitored at each slice of time, perhaps twice per second. The video frames or still images are evaluated with machine learning models such as support vector machines to determine the probability that each flag is happening, and then an overall cheating score or probability is calculated. You can see this happening in real time at this very helpful video.

If you are a fan of the show Silicon Valley you might recall how a character built an app for recognizing food dishes with AI… and the first iteration was merely “hot dog” vs. “not hot dog.” This was a nod towards how many applications of AI break problems down into simple chunks.

Some examples of differences in AI remote proctoring

1. Video vs. Stills: Some vendors record full HD video of the student’s face with the webcam, while others are designed for lower-stakes exams, and only take a still shot every 10 seconds. In some cases, the still-image approach is better, especially if you are delivering exams in low-bandwidth areas.

2. Audio: Some vendors record audio, and flag it with AI. Some do not record audio.

3. Lockdown browser: Some vendors provide their own lockdown browser, such as Sumadi. Some integrate with market leaders like Respondus or SafeExam. Others do not provide this feature.

4. Peripheral detection: Some vendors can detect certain hardware situations that might be an issue, such as a second monitor. Others do not.

5. Second camera: Some vendors have the option to record a second camera; typically this is from the examinee’s phone, which is placed somewhere to see the room, since the computer webcam can usually only see their face.

6. Screen recording: Some vendors record full video of the examinee’s screen as they take the exam. It can be argued that this increases security, but is often not required if there is lockdown browser. It can also be argued that this decreases security, because now images of all your items reside in someplace other than your assessment platform.

7. Additional options: For example, Examus has a very powerful feature for Bring Your Own Proctors, where staff would be able to provide live online proctoring, enhanced with AI in real time. This would allow you to scale up the security for certain classifications that have high stakes but low volume, where live proctoring would be more appropriate.

Want some help in finding a solution?

We can help you find the right solution and implement it quickly for a sound, secure digital transformation of your assessments. Contact solutions@assess.com to request a meeting with one of our assessment consultants. Or, sign up for a free account in our assessment platforms, FastTest and Assess.ai and see how our Rapid Assessment Development (RAD) can get you up and running in only a few days. Remember that the assessment platform itself also plays a large role in security: leveraging modern methods like computerized adaptive testing or linear on-the-fly testing will help a ton.

What are current topics and issues regarding AI remote proctoring?

ASC hosted a webinar in August 2022, specifically devoted to this topic. You can view the video below.

May 17, 2022/by Nathan Thompson, PhD

$math educational assessment$

Paper-and-Pencil Testing: Still Around?

AdminAugust 16, 2021

Paper-and-pencil testing used to be the only way to deliver assessments at scale. The introduction of computer-based testing (CBT) in the 1980s was a revelation – higher fidelity item types, immediate scoring & feedback, and scalability all changed with the advent of the personal computer and then later the internet. Delivery mechanisms including remote proctoring provided students with the ability to take their exams anywhere in the world. This all exploded tenfold when the pandemic arrived. So why are some exams still offline, with paper and pencil?

Many education institutions are confused about which examination models to stick to. Should you go on with the online model they used when everyone was stuck in their homes? Should you adopt multi-modal examination models, or should you go back to the traditional pen-and-paper method?

This blog post will provide you with an evaluation of whether paper-and-pencil exams are still worth it in 2021.

Paper-and-pencil testing; The good, the bad, and the ugly

The Good

Answer Bubble Sheet Orange Offline exams have been a stepping stone towards the development of modern assessment models that are more effective. We can’t ignore the fact that there are several advantages of traditional exams.

Some advantages of paper-and-pencil testing include students having familiarity with the system, development of a social connection between learners, exemption from technical glitches, and affordability. Some schools don’t have the resources and pen-and-paper assessments are the only option available.

This is especially true in areas of the world that do not have the internet bandwidth or other technology necessary to deliver internet-based testing.

Another advantage of paper exams is that they can often work better for students with special needs, such as blind students which need a reader.

Paper and pencil testing is often more cost-efficient in certain situations where the organization does not have access to a professional assessment platform or learning management system.

The Bad and The Ugly

However, the paper-and-pencil testing does have a number of shortfalls.

1. Needs a lot of resources to scale

Delivery of paper-and-pencil testing at large scale requires a lot of resources. You are printing and shipping, sometimes with hundreds of trucks around the country. Then you need to get all the exams back, which is even more of a logistical lift.

2. Prone to cheating

Most people think that offline exams are cheat-proof but that is not the case. Most offline exams count on invigilators and supervisors to make sure that cheating does not occur. However, many pen-and-paper assessments are open to leakages. High candidate-to-ratio is another factor that contributes to cheating in offline exams.

3. Poor student engagement

We live in a world of instant gratification and that is the same when it comes to assessments. Unlike online exams which have options to keep the students engaged, offline exams are open to constant destruction from external factors.

Offline exams also have few options when it comes to question types.

4. Time to score

“To err is human.” But, when it comes to assessments, accuracy, and consistency. Traditional methods of hand-scoring paper tests are slow and labor-intensive. Instructors take a long time to evaluate tests. This defeats the entire purpose of assessments.

5. Poor result analysis

Pen-and-paper exams depend on instructors to analyze the results and come up with insight. This requires a lot of human resources and expensive software. It is also difficult to find out if your learning strategy is working or it needs some adjustments.

6. Time to release results

Online exams can be immediate. If you ship paper exams back to a single location, score them, perform psychometrics, then mail out paper result letters? Weeks.

7. Slow availability of results to analyze

Similarly, psychometricians and other stakeholders do not have immediate access to results. This prevents psychometric analysis, timely feedback to students/teachers, and other issues.

8. Accessibility

Online exams can be built with tools for zoom, color contrast changes, automated text-to-speech, and other things to support accessibility.

9. Convenience

Online tests are much more easily distributed. If you publish one on the cloud, it can immediately be taken, anywhere in the world.

10. Support for diversified question types

Unlike traditional exams which are limited to a certain number of question types, online exams offer many question types. Videos, audio, drag and drop, high-fidelity simulations, gamification, and much more are possible.

11. Lack of modern psychometrics

Paper exams cannot use computerized adaptive testing, linear-on-the-fly testing, process data, computational psychometrics, and other modern innovations.

12. Environmental friendliness

Sustainability is an important aspect of modern civilization. Online exams eliminate the need to use resources that are not environmentally friendly such as paper.

Conclusion

Is paper-and-pencil testing still useful? In most situations, it is not. The disadvantages outweigh the advantages. However, there are many situations where paper remains the only option, such as poor tech infrastructure.

How ASC Can Help

Transitioning from paper-and-pencil testing to the cloud is not a simple task. That is why ASC is here to help you every step of the way, from test development to delivery. We provide you with the best assessment software and access to the most experienced team of psychometricians. Ready to take your assessments online? Contact us!

August 16, 2021/by Admin

Seven Technology Hacks to Deliver Assessments More Securely

Nathan Thompson, PhDMarch 27, 2021

So, yeah, the use of “hacks” in the title is definitely on the ironic and gratuitous side, but there is still a point to be made: are you making full use of current technology to keep your tests secure? Gone are the days when you are limited to linear test forms on paper in physical locations. Here are some quick points on how modern assessment technology can deliver assessments more securely, effectively, and efficiently than traditional methods:

1. AI delivery like CAT and LOFT

Psychometrics was one of the first areas to apply modern data science and machine learning (see this blog post for a story about a MOOC course). But did you know it was also one of the first areas to apply artificial intelligence (AI)? Early forms of computerized adaptive testing (CAT) were suggested in the 1960s and had become widely available in the 1980s. CAT delivers a unique test to each examinee by using complex algorithms to personalize the test. This makes it much more secure, and can also reduce test length by 50-90%.

2. Psychometric forensics

Modern psychometrics has suggested many methods for finding cheaters and other invalid test-taking behavior. These can range from very simple rules like flagging someone for having a top 5% score in a bottom 5% time, to extremely complex collusion indices. These approaches are designed explicitly to keep your test more secure.

3. Tech enhanced items

Tech enhanced items (TEIs) are test questions that leverage technology to be more complex than is possible on paper tests. Classic examples include drag and drop or hotspot items. These items are harder to memorize and therefore contribute to security.

4. IP address limits

Suppose you want to make sure that your test is only delivered in certain school buildings, campuses, or other geographic locations. You can build a test delivery platform that limits your tests to a range of IP addresses, which implements this geographic restriction.

5. Lockdown browser

A lockdown browser is a special software that locks a computer screen onto a test in progress, so for example a student cannot open Google in another tab and simply search for answers. Advanced versions can also scan the computer for software that is considered a threat, like a screen capture software.

6. Identity verification

Tests can be built to require unique login procedures, such as requiring a proctor to enter their employee ID and the test-taker to enter their student ID. Examinees can also be required to show photo ID, and of course, there are new biometric methods being developed.

7. Remote proctoring

The days are gone when you need to hop in the car and drive 3 hours to sit in a windowless room at a community college to take a test. Nowadays, proctors can watch you and your desktop via webcam. This is arguably as secure as in-person proctoring, and certainly more convenient and cost-effective.

So, how can I implement these to deliver assessments more securely?

Some of these approaches are provided by vendors specifically dedicated to that space, such as ProctorExam for remote proctoring. However, if you use ASC’s FastTest platform, all of these methods are available for you right out of the box. Want to see for yourself? Sign up for a free account!

March 27, 2021/by Nathan Thompson, PhD