Before the introduction of online exams in the education sector, the mentioning of the word ‘exams’ was met with anxiety. Exams were limited to four walls and you could cut the tension in the exam center with a razor. The furious scribbling of papers, the sharp glances from the hawk-eyed invigilators, and the constant ticking of the wall clock is not any experience many will forget.

But then came the internet and there was a better way to assess students. Online exams, though not popular at the time, provided a better way to develop and deliver tests. Using psychometric methods such as Computerized Adaptive Testing and Item Response Theory, assessments became more reliable and secure. Delivery mechanisms including remote proctoring provided students with the ability to take their exams anywhere in the world.

However, despite all these numerous benefits, online exams remained in the dark till the pandemic hit. Many educational institutions and businesses embraced online exams and made them the core of their systems. Forward a year later, and the deployment of vaccines is underway. Many education institutions are confused about which examination models to stick to. Should you go on with the online exams model they used when everyone was stuck in their homes? Should you adopt hybrid examination models, or should you go back to the traditional pen-and-paper method?  

This blog post will provide you with an evaluation of whether offline exams are still worth it in 2021. 

Offline Exams; The good, the bad, and the ugly

The Good

Offline exams have been a stepping stone towards the development of modern assessment models that are more effective. We can’t ignore the fact that there are several advantages of traditional exams. 

Some advantages of offline exams include students having familiarity with the system, development of a social connection between learners, exemption from technical glitches, and affordability. Some schools don’t have the resources and pen-and-paper assessments are the only option available. 

The Bad and The Ugly

However, the pen-and-paper method of assessment has not been able to ensure that exams that achieve their core objectives. The traditional method of assessments is paved with uncertainties and inaccuracies.

 How do you develop and decide the main learning objectives? How do you measure performance and know what to do to improve learning outcomes? And how do you evaluate student strengths and weaknesses? These are just a few questions that the traditional assessment method can’t answer.

 Below is a list of challenges pen-and-paper methods face from test development to evaluation:

1. Needs a lot of resources

From test development to evaluation, pen and paper methods require a lot of resources. Resources can range from high human resource fees to materials needed to develop and deliver the exams to students. 

2. Lack of seamless collaboration and scalability

The ability to cater to a bigger audience is important for productivity and saving resources. However,  the pen-and-paper method offers no room for scalability. Only a fixed number of students can take exams at a certain period. This is not only expensive but also wastes valuable time and increases the chances of leakage.

3. Prone to cheating

Most people think that offline exams are cheat-proof but that is not the case. Most offline exams count on invigilators and supervisors to make sure that cheating does not occur. However, many pen-and-paper assessments are open to leakages. High candidate-to-ratio is another factor that contributes to cheating in offline exams.

4. Poor student engagement

We live in a world of instant gratification and that is the same when it comes to assessments. Unlike online exams which have options to keep the students engaged, offline exams are open to constant destruction from external factors.

Offline exams also have few options when it comes to question types. 

5. Flawed evaluation system

To err is human”

 But, when it comes to assessments, accuracy, and consistency. Traditional evaluation methods are slow and labor-intensive. Instructors take a long time to evaluate tests. This defeats the entire purpose of assessments.

6. Poor result analysis

Pen-and-paper exams depend on instructors to analyze the results and come up with insight. This requires a lot of human resources and expensive software. It is also difficult to find out if your learning strategy is working or it needs some adjustments. 

A glimpse into online exams

Also referred to as digital exams or e-exams, online exams are delivered over the internet. The best online examination platforms include modules to facilitate the entire examination process, from development to delivery. Online exams provide instructors with the ability to diversify question types and monitor the assessment process. Using learning analytics, they are also able to modify learning methods to increase the quality of output. 

Online exams work like your typical pen-and-paper methods but with more accuracy, scalability, and reliability. The grading of the papers is done automatically after the assessment is done, depending on the question types. For essays, instructors can use online essay scoring. E-examinations have improved the assessment process and those are just a few examples. 

Here are some pros and cons of online exams to help you contemplate whether online exams are for you

The pros of online exams

1. Scalability

Unlike traditional testing methods which have a fixed number of people who can take an exam in a fixed time, online exams can cater to bigger audiences. This saves education institutions a lot of resources that would be invested in developing and managing examination centers. 

2. Automated report generation and visualization

This is the greatest advantage online exams have over offline exams. The automated report generation and visualization functions integrated into online assessment platforms enable instructors to accurately gauge learning outcomes. This gives them actionable insight to improve the learning process. 

3. Accessibility

Online exams can be taken from anywhere in the world. All one needs is a computer and an internet connection. This has given students access to knowledge from global learning institutions.

4. Support for diversified question types

Unlike traditional exams which are limited to a certain number of question types, online exams offer many question types option. Multiple Choice Questions, video assessments, coding simulators, and many other question types are supported. With this kind of freedom, instructors are able to decide which question types are fit for certain topics. 

5. In-built psychometrics

Psychometrics is an important part of an assessment as it ensures the development of high-quality tests. The implementation of psychometrics into traditional pen-and-paper methods is a difficult process and depends on the experience of instructors. 

With online exams, you can easily capitalize on it through tech-enhanced items, automated test assembly, Computerized Adaptive Testing, etc. 

6. Improved academic integrity

Cheating is the biggest concern when it comes to online exams. Most people wonder, ‘Isn’t giving students access to a computer and a high-speed internet connection just handing over answers to students?’ Well, that is far from the truth. 

In fact, online exams are safer than offline exams. Online exams are protected using advanced technologies such as Lockdown browser, IP-based authentication, AI-flagging, and many other strategies.

 Check out this article to learn how online exams are secured.  

7. Environmental friendliness

Sustainability is an important aspect of modern civilization.  Online exams eliminate the need to use resources that are not environmentally friendly such as paper. 

The cons of online exams

1. Digital transformation challenges

The process of transitioning examination from offline models to online platforms is one that requires intense planning and resources. However, this barrier can be eliminated easily by creating awareness among students and instructors on how to capitalize on digital assessments.

You can also hire firms with experience in migrating to digital assessments to help in the process. 

2. Academic integrity concerns

Cheating concerns still remain a turn-off for institutions that wish to transition to online exams. There are many ways students can circumnavigate security protocols to cheat in exams. Some ways include impersonation, external help, surfing the internet, and many others. 

However, these cheating ‘tricks’ can be avoided using Assess’ online assessment software with security features such as lock-down browser, IP-based authentication, and AI-powered remote proctoring. 

Offline Exams vs Online exams

traditional approach vs modern approach

Offline exams vs E-examinations

Conclusion

Are offline exams still worth it in 2021? No, they are not. As we have seen from the above sections, the traditional exam approach has several flaws that are barriers to effective assessment. However, it’s not as simple as that. There are many instances where the traditional approach would be a better option. Some instances include when students can’t afford infrastructure. But, when you are looking to conduct high-stakes examinations, online exams are the best option. 

How Assess Can Help 

Transitioning from offline exams to offline exams is not a simple task. That is why Assess is here to help you every step of the way, from test development to delivery. We provide you with the best assessment software and access to the most experienced team. Ready to take your assessments online?

 

 

Finding good employees in an overcrowded market is a daunting task. In fact, according to research by Career builder, 74% of employers admit to hiring the wrong employees. Bad hires are not only expensive, but can also adversely affect cultural dynamics in the workforce. This is where pre-employment assessment software shows its value.

Pre-employment testing tools help companies create effective assessments, thus saving valuable resources, improving candidate experience & quality hire, and reducing hiring bias.  But, finding a pre-employment testing software that can help you reap these benefits can be difficult, especially because of the explosion of software solutions in the market.  If you are lost on which tools will help you develop and deliver your own pre-employment assessments, this guide is for you.

First things first: you need to understand the basics of pre-employment tests. 

What is a pre-employment test?

A pre-employment test refers to an examination given to job seekers before hiring. The main reasons for administering these tests include determining important candidate metrics such as cognitive abilities, job experience, and personality traits.  The popularity of pre-employment tests has sky-rocketed in the past years. This is because of their ability to help companies manage large banks of candidate applications.  This helps increase quality hires by providing access to a diversified network of professionals while eliminating roadblocks such as ‘Resume Spammers’.  

Types of pre-employment tests

There are different types of pre-employment assessments. Each of them achieves a different goal in the hiring process. The major types of pre-employment assessments include:

Personality tests: Despite rapidly finding their way into HR, these types of pre-employment tests are widely misunderstood. Personality tests answer questions in the social spectrum.  One of the main goals of these tests is to quantify the success of certain candidates based on behavioral traits. 

Aptitude tests: Unlike personality tests or emotional intelligence tests which tend to lie on the social spectrum, aptitude tests measure problem-solving, critical thinking, and agility.  These types of tests are popular because can predict job performance than any other type because they can tap into areas that cannot be found in resumes or job interviews. 

Skills Testing: The kinds of tests can be considered a measure of job experience; ranging from high-end skills to low-end skills such as typing or Microsoft excel. Skill tests can either measure specific skills such as communication or measure generalized skills such as numeracy. 

Emotional Intelligence tests: These kinds of assessments are a new concept but are becoming important in the HR industry. With strong Emotional Intelligence (EI) being associated with benefits such as improved workplace productivity and good leadership, many companies are investing heavily in developing these kinds of tests.  Despite being able to be administered to any candidates, it is recommended they be set aside for people seeking leadership positions, or those expected to work in social contexts. 

Risk tests: As the name suggests, these types of tests help companies reduce risks. Risk assessments offer assurance to employers that their workers will commit to established work ethics and not involve themselves in any activities that may cause harm to themselves or the organization.  There are different types of risk tests. Safety tests, which are popular in contexts such as construction, measure the likelihood of the candidates engaging in activities that can cause them harm. Other common types of risk tests include Integrity tests

Pre-employment testing software: The Benefits 

Now that you have a good understanding of what pre-employment tests are, let’s discuss the benefits of integrating pre-employment assessment software into your hiring process. Here are some of the benefits:

Saves Valuable resources

Unlike the lengthy and costly traditional hiring processes, pre-employment assessment software helps companies increase their ROI by eliminating HR snugs such as face-to-face interactions or geographical restrictions. Pre-employment testing tools can also reduce the amount of time it takes to make good hires while reducing the risks of facing the financial consequences of a bad hire. 

Supports Data-Driven Hiring Decisions

Data runs the modern world, and hiring is no different. You are better off letting complex algorithms crunch the numbers and help you decide which talent is a fit, as opposed to hiring based on a hunch. 

Pre-employment assessment software helps you analyze assessments and generate reports/visualizations to help you choose the right candidates from a large talent pool. 

Improving candidate experience 

Candidate experience is an important aspect of a company’s growth, especially considering the fact that 69% of candidates admitting not to apply for a job in a company after having a negative experience. Good candidate experience means you get access to the best talent in the world. 

Elimination of Human Bias

Traditional hiring processes are based on instinct. They are not effective since it’s easy for candidates to provide false information on their resumes and cover letters. 

But, the use of pre-employment assessment software has helped in eliminating this hurdle. The tools have leveled the playing ground, and only the best candidates are considered for a position. 

Need some help deciding how you can reap the mentioned benefits of pre-employment assessment software? Click the button below to get help.


What To Consider When Choosing pre-employment assessment software

Now that you have a clear idea of what pre-employment tests are and the benefits of integrating pre-employment assessment software into your hiring process, let’s see how you can find the right tools. 

Here are the most important things to consider when choosing the right pre-employment testing software for your organization.

Ease-of-use

The candidates should be your top priority when you are sourcing pre-employment assessment software. This is because the ease of use directly co-relates with good candidate experience. Good software should have simple navigation modules and easy comprehension. 

Here is a checklist to help you decide if a pre-employment assessment software is easy to use;

  • Are the results easy to interpret?
  • What is the UI/UX like?
  • What ways does it use to automate tasks such as applicant management?
  • Does it have good documentation and an active community?

Tests Delivery (Remote proctoring)

Remote proctoring (Courtesy of FastTest)

Good online assessment software should feature good online proctoring functionalities. This is because most remote jobs accept applications from all over the world. It is therefore advisable to choose a pre-employment testing software that has secure remote proctoring capabilities. Here are some things you should look for on remote proctoring;

  • Does the platform support security processes such as IP-based authentication, lockdown browser, and AI-flagging?
  • What types of online proctoring does the software offer? Live real-time, AI review, or record and review?
  • Does it let you bring your own proctor?
  • Does it offer test analytics?

Test & data security, and compliance

Defensibility is what defines test security. There are several layers of security associated with pre-employment test security. When evaluating this aspect, you should consider what pre-employment testing software does to achieve the highest level of security. This is because data breaches are wildly expensive. 

The first layer of security is the test itself. The software should support security technologies and frameworks such as lockdown browser, IP-flagging, and IP-based authentication. If you are interested in knowing how to secure your assessments, check this post out.

The other layer of security is on the candidate’s side. As an employer, you will have access to the candidate’s private information. How can you ensure that your candidate’s data is secure? That is reason enough to evaluate the software’s data protection and compliance guidelines.

A good pre-employment testing software should be compliant with certifications such as GDRP. The software should also be flexible to adapt to compliance guidelines from different parts of the world. 

Questions you need to ask;

  • What mechanisms does the software employ to eliminate infidelity?
  • Is their remote proctoring function reliable and secure?
  • Are they compliant with security compliance guidelines including ISO, SSO, or GDPR?
  • How does the software protect user data?

User experience

A good user experience is a must-have when you are sourcing any enterprise software. A new age pre-employment testing software should create user experience maps with both the candidates and employer in mind. Some ways you can tell if a software offers a seamless user experience includes;

  • User-friendly interface
  • Simple and easy to interact with
  • Easy to create and manage item banks
  • Clean dashboard with advanced analytics and visualizations

Customizing your user-experience maps to fit candidates’ expectations attracts high-quality talent. 

Scalability and automation

With a single job post attracting approximately 250 candidates, scalability isn’t something you should overlook. A good pre-employment testing software should thus have the ability to handle any kind of workload, without sacrificing assessment quality. 

It is also important you check the automation capabilities of the software. The hiring process has many repetitive tasks that can be automated with technologies such as Machine learning, Artificial Intelligence (AI), and robotic process automation (RPA).  

Here are some questions you should consider in relation to scalability and automation; 

  • Does the software offer Automated Item Generation (AIG)?
  • How many candidates can it handle? 
  • Can it support candidates from different locations worldwide?


Reporting and analytics

Reporting and visualization

Example of Reporting and visualization functionality to help you make data-driven hiring decisions  (Courtesy of FastTest and Assess.ai)

A good pre-employment assessment software will not leave you hanging after helping you develop and deliver the tests. It will enable you to derive important insight from the assessments.

The analytics reports can then be used to make data-driven decisions on which candidate is suitable and how to improve candidate experience. Here are some queries to make on reporting and analytics;

  • Does the software have a good dashboard?
  • What format are reports generated in?
  • What are some key insights that prospects can gather from the analytics process?
  • How good are the visualizations?

Customer and Technical Support

Customer and technical support is not something you should overlook. A good pre-employment assessment software should have an Omni-channel support system that is available 24/7. This is mainly because some situations need a fast response. Here are some of the questions your should ask when vetting customer and technical support;

  • What channels of support does the software offer/How prompt is their support?
  • How good is their FAQ/resources page?
  • Do they offer multi-language support mediums?
  • Do they have dedicated managers to help you get the best out of your tests?

Conclusion

Finding the right pre-employment testing software is a lengthy process, yet profitable in the long run. We hope the article sheds some light on the important aspects to look for when looking for such tools. Also, don’t forget to take a pragmatic approach when implementing such tools into your hiring process.

Are you stuck on how you can use pre-employment testing tools to improve your hiring process? Feel free to contact us and we will guide you on the entire process, from concept development to implementation. Whether you need off-the-shelf tests or a comprehensive platform to build your own exams, we can provide the guidance you need.  We also offer free versions of our industry-leading software FastTest and Assess.ai– visit our Contact Us page to get started!

 

Estimated reading time: 3 minutes

AI remote proctoring has seen an incredible increase of usage during the COVID pandemic. ASC works with a very wide range of clients, with a wide range of remote proctoring needs, and therefore we partner with a range of remote proctoring vendors that match to our clients’ needs, and in some cases bring on a new vendor at the request of a client.

At a broad level, we recommend live online proctoring for high-stakes exams such as medical certifications, and AI remote proctoring for medium-stakes exams such as pre-employment assessment. Suppose you have decided that you want to find a vendor for AI remote proctoring. How can you start evaluating solutions?

Well, it might be surprising, but even within the category of “AI remote proctoring” there can be substantial differences in the functionality provided, and you should therefore closely evaluate your needs and select the vendor that makes the most sense – which is the approach we at ASC have always taken.

Some examples of differences in AI remote proctoring

   1. Video vs. Stills: Some vendors record full HD video of the student’s face with the webcam, while others are designed for lower-stakes exams or lower-bandwidth countries, and only take a still shot every 30 seconds.

   2. Audio: Some vendors record audio, and flag it with AI.  Some do not record audio.

   3. Lockdown browser: Some vendors provide their own lockdown browser.  Some integrate with market leaders like Respondus or SafeExam.  Others do not provide this feature.

   4. Peripheral detection: Some vendors can detect certain hardware situations that might be an issue, such as a second monitor. Others do not.

   5. Second camera: Some vendors have the option to record a second camera; typically this is from the examinee’s phone, which is placed somewhere to see the room, since the computer webcam can usually only see their face.

   6. Screen recording: Some vendors record full video of the examinee’s screen as they take the exam. It can be argued that this increases security, but is often not required if there is lockdown browser. It can also be argued that this decreases security, because now images of all your items reside in someplace other than your assessment platform.

   7. Additional options: For example, Examus has a very powerful feature for Bring Your Own Proctors, where staff would be able to provide live online proctoring, enhanced with AI in real time. This would allow you to scale up the security for certain classifications that have high stakes but low volume, where live proctoring would be more appropriate. 

Want some help in finding a solution?

We can help you find the right solution and implement it quickly for a sound, secure digital transformation of your assessments. Contact solutions@assess.com to request a meeting with one of our assessment consultants. Or, sign up for a free account in our assessment platforms, fastest and Assess.ai and see how our Rapid Assessment Development (RAD) can get you up and running in only a few days. Remember that the assessment platform itself also plays a large role in security: leveraging modern methods like computerized adaptive testing or linear on-the-fly testing will help a ton.

Digital assessment security is a highly discussed topic in the Ed-Tech community. The use of digital assessments in education has led to increased concerns about academic integrity. While digital assessments have more advantages than the outdated pen-paper method, they are open to a lot of criticism in relation to academic integrity concerns. However, cheating can be eliminated using the right tools and best practices. 

Before we get into the nitty-gritty of creating secure digital assessments, let’s define cheating.

What is cheating? 

Cheating refers to actual, intended, or attempted deception and/or dishonesty in relation to academic or corporate assessments. 

Some smart ways in which examinees cheat in digital assessments include;

  • Plagiarism– Taking someone else’s work and turning it into one’s own.
  • Access to external resources– Using external resources such as the internet, books, etc. 
  • Using freelancing services– Paying someone else especially from academic freelancing sites such as Studypool to do the work for you. 
  • Impersonation– Letting someone else take an assessment in one’s place.
  • Abetting cheating– Selfless acts such as releasing exam papers early, or discussing answers with examinees who have taken the exam. 
  • Using remote access tools to get external help.

When addressing digital assessments infidelity, most professionals ask the wrong questions. How do we stop cheating? Or which digital assessment security techniques should we use?

 But we believe that is not an effective way to approach this problem. The right questions ask are; why do students cheat? how can we make students/employees be confident enough to take assessments without cheating? Which digital assessment tools address all the weakest links in the digital assessment life cycle? For instance, by integrating techniques such as adaptive testing or multistage-testing, the students and employees feel confident to take exams. 

In this blog post, we are going to share digital assessment techniques and best practices that can help you create effective assessments. 

Part 1: Digital exams security techniques 

Online/Remote proctoring is the process of monitoring digital assessments. This gives supervisors and invigilators the freedom to monitor examinations from any location. 

1. Online Proctoring

To increase assessment security in this process, you can utilize these methods;

  • Image Capturing

This is a technique used to increase the effectiveness of online testing by taking photos of activities in a given interval of time. The images are later analyzed to flag down any form of infidelity detected in the image banks. This is a basic mechanism, yet effective in situations where students don’t have access to a stable internet connection. 

  • Audio Capturing

This one is almost similar to image/video capturing but keeps track of audio logs. The audio logs are later analyzed to identify any voice abnormalities in the surrounding environment. This eliminates risks such as getting help from family members or friends in the same room. 

  • Live Video Recording and/ Streaming

This is the most important element of security in digital assessments. Live video streaming keeps the candidate under constant surveillance and invigilators can suspend the examination if they notice suspicious behavior. Some online testing tools are even integrated with AI systems to help in the process of identifying suspicious activity. Video recording on the other hand works like audio capturing. 

Digital assessment security: Remote Proctoring,  Record and Review
Remote Proctoring: Record and Review

 

The video is recorded, stored in the cloud, and later analyzed to spot any suspicious behavior. 

To give you the freedom to be creative and flexible in the assessment process, Assess.com even lets you bring your own proctor.

2. Lockdown Browser

 

High-speed broadband and access to hundreds of tools to do anything. Sounds like the perfect opportunity to cheat? Yes, to some extent. However, with the lockdown browser feature, students are limited to one tab, the one hosting the exam. 

If the student tries accessing any external software or opening a tab, a notification is sent to the invigilator. The feature starts by scanning the computer for video/image capture. Once the test begins, the examinee is restricted from surfing the internet for answers. 

3. Digital assessment security through AI Flagging

Digital Assessment security: AI Flagging In Remote Proctoring
AI Flagging In Remote Proctoring 

 

As seen in the article 10 EdTech Trends and Predictions To Watch Out For, In 2021,  AI will increase in popularity in Ed-Tech. Not only can facial recognition (which is a form of AI) be used to increase security on campus but also in digital assessment.

 By using systems integrated with AI systems, supervisors can identify suspicious behavior and take action, in real-time. Some actions the AI systems are able to identify include two faces in one screen.  This feature works in hand with video streaming/recording. This helps increase examination integrity while cutting back on costs. 

4. IP-Based Authentication

 

Identity authentication is an integral step in achieving the highest form of academic integrity in digital assessments. IP-Based authentication uses a user’s IP address to confirm the identity of students undertaking the exam. This comes in handy when examinees try to cheat by offering remote access to other users. In a recent paper exploring, e-authentication systems, other components of authentication may include biometric systems, mechanisms, knowledge, and document analysis. Using IP-Based authentication in digital assessment may decrease some risks of cheating.

5. Digital assessment security by data encryption

Cybercrimes have become a threat to many industries. Education is one of the industries that face a lot of challenges when it comes to information security. In fact, according to a recent study, 83% of schools in the UK experienced at least one cyberattack, despite having protection mechanisms such as antivirus software and firewalls. Another research also discovered that education records could be sold as high as $265 on the black market

Digital assesment security: CyberSecurity By Industry
CyberSecurity By Industry

 

 This kind of insecurity finds its way into digital assessment because most tests are developed and delivered online. Some students can get early access to digital assessments or change their grades if they can find a loophole in the digital assessment systems. To prevent this, it is important to implement cybersecurity practices in digital assessment systems. Data encryption is a key part of ensuring that exams and personal data are safe.  

  1. Audit Logging

The audit logging function tracks user activity in all aspects of the testing process using unique identifiers such as IP addresses. It keeps track of all activities including clicks, stand-by time, software accessed during the assessment, and so much more! These logs are then analyzed for signs of infidelity. 

Part II: Digital assessment security best practices

Digital assesment life cycle
Digital Assessment Life Circle

 

Most of the technologies discussed above focus on the last few stages of the evaluation process. The strategies below focus on techniques that do not fall under the main stages of the process, yet very critical.

1. Creating Awareness On Cheating

Many institutions don’t create enough awareness of the dangers of cheating in relation to academic integrity. You can use media such as videos to give them guidelines of how to undertake exams authentically, and what is considered cheating. This prepares the students to undertake the exams without thinking about cheating. 

2. Don’t Use Publisher Item Banks

Course textbooks and guides may come with complimentary test banks, and you should refrain from using them in assessments. This is because most course materials are available on the internet, and may encourage infidelity. Instead, make up your own questions, based on your learning objectives and methodologies. 

3. Disable Backtracking

This increases learning efficiency by making sure that students answer what they know, and locate their weakest links. By facilitating backtracking, you give the students time to go back and try to locate the correct answers, which reduces effectiveness. 

4. Diversity In Question Types

Fear of Failure is one thing that makes examinees cheat. By creating diverse question types, you improve the assessment engagement of the students, therefore improving their confidence. Don’t give your examinees multiple-choice questions only, add in some yes/no questions, essays, and so on.

5. Make Them Sign Academic Integrity Contracts

Contracts have a psychological effect on people, and the examinees are more likely to be authentic if they sign some form of contract. 

If you are interested in other best practices to increase online assessment security, here is an important resource that can help.

Conclusion

Online exam security is an important aspect of online education and should not be ignored. Despite the immaturity of existing frameworks and methodologies in the market, techniques such as remote proctoring, lockdown browser, and audit logging have proven to be effective. 

Creating and delivering high-quality digital assessments is not a walk in the park. If you need professional help in creating online assessments with alignment to the best psychometrics practices, contact Assess for a free consultation. 

You can also get access to online assessment tools with security features including, AI-flagging, remorse proctoring, lock-down browser and so much more!

Estimated reading time: 4 minutes

Psychometric forensics is a surprisingly deep and complex field.  Many of the indices are incredibly sophisticated, but a good high-level and simple analysis to start with is overall time vs. scores, which I call Time-Score Analysis.  This approach uses simple flagging on two easily interpretable metrics (total test time in minutes and number correct raw score) to identify possible pre-knowledge, clickers, and harvester/sleepers.  Consider the four quadrants that a bivariate scatterplot of these variables would produce.

Quadrant Interpretation Possible threat? Suggested flagging
Upper right High scores and taking their diligent time Good examinees NA
Upper left High scores with low time Pre-knowledge Top 50% score and bottom 5% time
Lower left Low scores with low time “Clickers” or other low motivation Bottom 5% time and score
Lower right Low scores with high time Harvesters, sleepers, or just very low ability Top 5% time and bottom 5% scores

An example of Time-Score Analysis

Consider the example data below.  What can this tell us about the performance of the test in general, and about specific examinees?

This test had 100 items, scored classically (number-correct), and a time limit of 60 minutes.  Most examinees took 45-55 minutes, so the time limit was appropriate.  A few examinees spent 58-59 minutes; there will usually be some diligent students like that.  There was a fairly strong relationship of time with the score, in that examinees who took longer, scored highly.

Now, what about the individuals?  I’ve highlighted 5 examples.

  1. This examinee had the shortest time, and one of the lowest scores.  They apparently did not care very much.  They are an example of a low motivation examinee that moved through quickly.  One of my clients calls these “clickers.”
  2. This examinee also took a short time but had a suspiciously high score.  They definitely are an outlier on the scatterplot, and should perhaps be investigated.
  3. This examinee is simply super-diligent.  They went right up to the 60-minute limit and achieved one of the highest scores.
  4. This examinee also went right up to the 60-minute limit but had one of the lowest scores.  They are likely low ability or low motivation.  That same client of mine calls these “sleepers” – a candidate that is forced to take the exam but doesn’t care, so just sits there and dozes. Alternatively, it might be a harvester; some who have been assigned to memorize test content, so they spend all the time they can, but only look at half the items so they can focus on memorization.
  5. This examinee had by far the lowest score, and one of the lowest times.  Perhaps they didn’t even answer every question.  Again, there is a motivation/effort issue here, most likely.

How useful is time-score analysis?

Like other aspects of psychometric forensics, this is primarily useful for flagging purposes.  We do not know yet if #4 is a Harvester or just low motivation.  Instead of accusing them, we open an investigation.  How many items did they attempt?  Are they repeat test-takers?  What location did they take the test?  Do we have proctor notes, site video, remote proctoring video, or other evidence that we can review? 

There is a lot that can go into such an investigation.  Moreover, simple analyses such as this are merely the tip of the iceberg when it comes to psychometric forensics.  In fact, so much that I’ve heard some organizations simply stick their head in the sand and don’t even bother checking out someone like #4.  It just isn’t in the budget.

However, test security is an essential aspect of validity.  If someone has stolen your test items, the test is compromised, and you are guaranteed that scores do not mean the same thing they meant when the test was published.  It’s now apples and oranges, even though the items on the test are the same.  Perhaps you might not challenge individual examinees but perhaps institute a plan to publish new test forms every 6 months. Regardless, your organization needs to have some difficult internal discussions and establish a test security plan.

Psychometrics is the cornerstone of any high-quality assessment program.  However, most organizations can’t afford an in-house Ph.D. psychometrician, which then necessitates the search for psychometric consulting.  Most organizations, when first searching, are new to the topic and not sure what role the psychometrician plays. 

In this article, we’ll talk about how psychometricians and their tools can help improve your assessments, whether you just want to check on test reliability or pursue the lengthy process of accreditation.

Why ASC?

Whether you are establishing or expanding a credentialing program, streamlining operations, or moving from paper to online testing, ASC has a proven track record of providing practical, cost-efficient solutions with uncompromising quality. We offer a free consultation with our team of experts to discuss your needs and determine which solutions are the best fit, including our enterprise SaaS platforms, consulting on sound psychometrics, or recommending you to one of our respected partners.

At the heart of our business are our people.

Our collaborative team of Ph.D. psychometricians, accreditation experts, and software developers have diverse experience developing solutions that drive best practices in assessment. This real-world knowledge enables us to consult your organization with solutions tailored specifically to your goals, timeline, and budget.

Comprehensive Solutions to Address Specific Measurement Problems

Much of psychometric consulting is project-based around solving a specific problem.  For example, you might be wondering how to set a cut score on a certification/licensure exam that is legally defensible and meets accreditation standards. 

This is a very specific issue, and the scientific literature has suggested a number of sound approaches.  Here are some of the topics where psychometricians can really help:

  • Test Design: Job Analysis & Blueprints
  • Standard and Cutscore Setting Studies
  • Item Writing and Review Workshops
  • Test and Item Statistical Analysis
  • Equating Across Years and Forms
  • Adaptive Testing Research
  • Test Security Evaluation
  • NCCA/ANSI Accreditation

Why psychometric consulting?

All areas of assessment can be smarter, faster, and fairer.

Develop Reliable and Valid Assessments
We’ll help you understand what needs to be done to develop defensible tests and how to implement them in a cost-efficient manner.  Much of the work revolves around establishing a sound test development cycle.

Increase Test Security
We have specific expertise in psychometric forensics, allowing you to flag suspicious candidates or groups in real-time, using our automated forensics report.

Achieve Accreditation
Our dedicated experts will assist in setting your organization up for success with NCCA/ANSI accreditation of professional certification programs.

Comprehensive Psychometric Analytics
We use CTT and IRT with principles of machine learning and AI to deeply understand your data and provide actionable recommendations.

We can help your organization develop and publish certification and licensure exams, based on best practices and accreditation standards, in a matter of months.

Effective assessments; Best practices

Item and Test Statistical Analysis
If you are doing this process at least annually, you are not meeting best practices or accreditation standards. But don’t worry, we can help! In addition to performing these analyses for you, you also have the option of running them yourself in our FastTest platform or using our psychometric software like Iteman and Xcalibre.

Job Analysis
How do you know what a professional certification test should cover?  Well, let’s get some hard data by surveying job incumbents. Knowing and understanding this information and how to use it is essential if you want to test people on whether they are prepared for the job or profession.

Cut score Studies (Standard Setting)
When you use sound psychometric practices like the modified-Angoff, Beuk Compromise, Bookmark, and Contrasting Groups methods, it will help you establish a cutscore that meets professional standards.

 

It’s all much easier if you use the right software!

Once we help you determine the best solutions for your organization, we can train you on best practices, and it’s extremely easy to use our software yourself.  Software like Iteman and Xcalibre is designed to replace much of the manual work done by psychometricians for the item and test analysis, and FastTest automates many aspects of test development and publishing.  We even offer free software like the Angoff Analysis Tool

However, our ultimate goal is your success: Assessment Systems is a full-service company that continues to provide psychometric consulting and support even after you’ve made a purchase. Our team of professionals is available to provide you with additional support at any point in time. We want to ensure you’re getting the most out of our products!  Click below to sign up for a free account in FastTest and see for yourself.

Sign up for a Free Account

​So, yeah, the use of “hacks” in the title is definitely on the ironic and gratuitous side, but there is still a point to be made: are you making full use of current technology to keep your tests secure?  Gone are the days when you are limited to linear test forms on paper in physical locations.  Here are some quick points on how modern assessment technology can deliver assessments more securely, effectively, and efficiently than traditional methods:

1.  AI delivery like CAT and LOFT

Psychometrics was one of the first areas to apply modern data science and machine learning (see this blog post for a story about a MOOC course).  But did you know it was also one of the first areas to apply artificial intelligence (AI)?  Early forms of computerized adaptive testing (CAT) were suggested in the 1960s and had become widely available in the 1980s.  CAT delivers a unique test to each examinee by using complex algorithms to personalize the test.  This makes it much more secure, and can also reduce test length by 50-90%.

2. Psychometric forensics

Modern psychometrics has suggested many methods for finding cheaters and other invalid test-taking behavior.  These can range from very simple rules like flagging someone for having a top 5% score in a bottom 5% time, to extremely complex collusion indices.  These approaches are designed explicitly to keep your test more secure.

3. Tech enhanced items

Tech enhanced items (TEIs) are test questions that leverage technology to be more complex than is possible on paper tests.  Classic examples include drag and drop or hotspot items.  These items are harder to memorize and therefore contribute to security.

4. IP address limits

Suppose you want to make sure that your test is only delivered in certain school buildings, campuses, or other geographic locations.  You can build a test delivery platform that limits your tests to a range of IP addresses, which implements this geographic restriction.

5. Lockdown browser

 A lockdown browser is a special software that locks a computer screen onto a test in progress, so for example a student cannot open Google in another tab and simply search for answers.  Advanced versions can also scan the computer for software that is considered a threat, like a screen capture software.

6. Identity verification

Tests can be built to require unique login procedures, such as requiring a proctor to enter their employee ID and the test-taker to enter their student ID.  Examinees can also be required to show photo ID, and of course, there are new biometric methods being developed.

7. Remote proctoring

 The days are gone when you need to hop in the car and drive 3 hours to sit in a windowless room at a community college to take a test.  Nowadays, proctors can watch you and your desktop via webcam.  This is arguably as secure as in-person proctoring, and certainly more convenient and cost-effective.

So, how can I implement these to deliver assessments more securely?

Some of these approaches are provided by vendors specifically dedicated to that space, such as ProctorExam for remote proctoring.  However, if you use ASC’s FastTest platform, all of these methods are available for you right out of the box.  Want to see for yourself?  Sign up for a free account!

SIGN UP FOR FREE FASTTEST ACCOUNT

The traditional Learning Management System (LMS) is designed to serve as a portal between educators and their learners. Platforms like Moodle are successful in facilitating cooperative online learning in a number of groundbreaking ways: course management, interactive discussion boards, assignment submissions, and delivery of learning content. While all of this is great, we’ve yet to see an LMS that implements best practices in assessment and psychometrics to ensure that medium or high stakes tests meet international standards.

To put it bluntly, LMS systems have assessment functionality that is usually good enough for short classroom quizzes but falls far short of what is required for a test that is used to award a credential.  A white paper on this topic is available here, but some examples include:

  • Treatment of items as reusable objects
  • Item metadata and historical use
  • Collaborative item review and versioning
  • Test assembly based on psychometrics
  • Psychometric forensics to search for non-independent test-taking behavior
  • Deeper score reporting and analytics

Assessment Systems is pleased to announce the launch of an easy-to-use bridge between FastTest and Moodle that will allow users to seamlessly deliver sound assessments from within Moodle while taking advantage of the sophisticated test development and psychometric tools available within FastTest. In addition to seamless delivery for learners, all candidate information is transferred to FastTest, eliminating the examinee import process.  The bridge makes use of the international Learning Tools Interoperability standards.

If you are already a FastTest user, watch a step-by-step tutorial on how to establish the connection, in the FastTest User Manual by logging into your FastTest workspace and selecting Manual in the upper right-hand corner. You’ll find the guide in Appendix N.

If you are not yet a FastTest user and would like to discuss how it can improve your assessments while still allowing you to leverage Moodle or other LMS systems for learning content, sign up for a free account here.

Desperation is seldom fun to see.

Some years ago, having recently released our online marking functionality I was reviewing some of the functionality in a customer workspace I was intrigued to see “Beyonce??” mentioned in a marker’s comments on an essay. The student’s essay was evaluating some poetry and had completely misunderstood the use of metaphor in the poem in question. The student also clearly knew that her interpretation was way off, but didn’t know how and had reached the end of her patience. So after a desultory attempt at answering, with a cry from the heart, reminiscent of William Wallace’s call for freedom, she wrote “BEYONCE” with about seventeen exclamation points. It felt good to see that her spirit was not broken, and it was a moment of empathy that drove home the damage that standardized tests are inflicting on our students. That vignette is playing itself out millions of time each year in this country, the following explains why.

What are “Standardized Tests”?

We use standardized tests for a variety of reasons, but underlying every reason (curriculum effectiveness, college/career preparedness, teacher effectiveness, etc.) is the understanding that the test is measuring what a student has learned. In order to know how all our students are doing, we give them all standardized tests, meaning every student receives essentially the same set of tests. So, a standardized test is a test where all students take essentially the same test. This is a difficult endeavor given the wide range of students and number of tests, and raises the question “How do we do this reliably and in a reasonable amount of time?”

Accuracy and Difficulty vs Length

We all want tests to reliably measure the students’ learning. In order to make these tests reliable, we need to supply questions of varying difficulty, from very easy to very difficult, to cover a wide range of abilities. In order to reduce the length of the test, most of the questions fall in the medium easy to medium difficulty range because that is where most of the students’ ability level will fall. So the test that best balances length and accuracy for the whole population should be constructed such that the amount of questions of any difficulty is proportionate to the number of students of that ability.

Why are most questions in the medium difficulty range? Imagine creating a test to measure 10th graders’ math ability. A small number of the students might have a couple years of calculus. If the test covered those topics, imagine the experience of most students who would often not even understand the notation in the question. Frustrating, right? On the other hand, if the test was also constructed to measure students with only rudimentary math knowledge, these average to advanced students would be frustrated and bored from answering a lot of questions on basic math facts. The solution most organizations use is to present only a few questions that are really easy or difficult, and accept that this score is not as accurate as they would prefer for the students at either end of the ability range.

These Tests are Inaccurate and Mean Spirited

The problem is that while this might work OK for a lot of kids, it exacts a pretty heavy toll on others. Almost one in five students will not know the answer to 80% of the questions on these tests, and scoring about 20% on a test certainly feels like failing. It feels like failing every time a student takes such a test. Over the course of an academic career, students in the bottom quintile will guess on or skip 10,000 questions. That is 10,000 times the student is told that school, learning, or success is not for them. Even biasing the test to be easier only makes a slight improvement.

Computerized Adaptive Testing, Test Performance with Bell Curve

The shaded area represents students who will miss at least 80% of questions.

It isn’t necessarily better for the top students whose every testing experience assures them that they are already very successful when the reality is that they are likely being outperformed by a significant percentage of their future colleagues.

In other words, at both ends of the Bell Curve, we are serving our students very poorly, inadvertently encouraging lower performing students to give up (there is some evidence that the two correlate) and higher performing students to take it easy. It is no wonder that people dislike standardized tests.

There is a Solution

A computerized adaptive test (CAT) solves all the problems outlined above. Properly constructed, a CAT has the ability to make the following faster, fairer, and more valid:

  • Every examinee completes the test in less time (fast)
  • Every examinee gets a more accurate score (valid)
  • Every examinee receives questions tuned to their ability so gets about half right (fair)

Given all the advantages of CAT, it may seem hard to believe that they are not used more often. While they are starting to catch on, it is not fast enough given the heavy toll that the old methods exact on our students. It is true that few testing providers can enable CATs, but that is simply making an excuse. If a standardized test is delivered to as few as 500 students it can be made adaptive. It probably isn’t, but it could be. All that is needed are computers or tablets, an Internet connection, and some effort. We should expect more.

How can my organization implement CAT?

While CAT used to only be feasible for large organizations that tested hundreds of thousands or millions of examinees per year, a number of advances have changed this landscape.  If you’d like to do something about your test, it might be worthwhile for you to evaluate CAT.  We can help you with that evaluation; if you’d like to chat, here is a link to schedule a meeting. Or contact me if you’d like to discuss the math or related ideas please drop me a note.