The Conference on Test Security is a scholarly conference devoted to the topic of security in educational, credentialing, and employment assessment.   Test fraud/cheating is one of the most salient threats to score validity, and can invalidate the entire exam, defeating the important goals such as ensuring that only qualified people are allowed to practice a profession.  So methods to combat it are extremely important to all of us.

The 2015 Conference on Test Security

The 2015 edition of Conference on Test Security was hosted by the Center for Educational Testing and Evaluation at the University of Kansas.  Assessment Systems had the privilege to present two full sessions: One Size Does Not Fit All: Making Test Security Configurable and Scalable and Let’s Rethink How Technology Can Improve Proctoring.  Abstracts for these are below; if you would like to learn more, please contact us.  Additionally, we had the opportunity for a product demonstration of our psychometric data forensics software, SIFT (more on that below).

The Conference on Test Security kicked off with a keynote on the now-famous Atlanta cheating scandal.  This scandal was unique in that it was systematic and top-down rather than bottom-up.  The follow-up message was a stressing of the difference between assessment and accountability; because a test is tied to accountability standards does not mean the test in itself is bad, or that all testing is bad.

One of the most commonly presented topics at the conference is data forensics.  In fact, the Conference on Test Security used to be call the Conference on Statistical Detection of Test Fraud.  But while there has been research on statistical detection of test fraud for more than 50 years, it is effectively a much younger topic and we are still learning a lot.  Moreover, there are no good software programs that are publicly available to help organizations implement best practices in data forensics. This is where SIFT comes in.

What is Psychometric Data Forensics?

In the realm of test security, data forensics refers to analysis of data to find evidence of various types of test fraud.  There are a few big types, and the approach to analysis can be quite different.  Here are some descriptions, though this is far from a complete treatment of the topic!

• Answer-changing: In Atlanta, teachers and administrators would change answers on student bubble sheets after the test was turned in.  This involves quantification of the changes, and then analysis of right-to-wrong vs. wrong-to-right changes, amongst other things.  This, of course, is primarily relevant for paper-based tests, but some answer-changing can happen on computer-based tests.
• Preknowledge: If an examinee purchases a copy of the test off the internet from an illegal “brain dump” website, they will get a surprisingly high score while taking less time than expected.  This could be on all items or a subset.
• Item harvesting: An examinee is paid to memorize as many items as they can.  They might spend 5 minutes each on the first 15 items and not even look at the remainder.
• Collusion:  The age-old issue of Student A copying off Student B is collusion, but it can also involve multiple students and even teachers or other people.  Statistical analysis looks at unusual patterns in the response data.

How can I improve test security at my organization?

Unfortunately, there is no publicly available software that is adequately devoted to data forensics.  Existing software is very limited in the analysis it provides and/or its usability.  For example, there are some packages available in the R programming language, but you need to learn to program in R, which is notoriously bad.  Therefore Assessment Systems has developed our own system, entitled Software for Investigating Test Fraud (SIFT), to meet this market need.

SIFT will provide a wide range of analysis, including a number of collision indices (see the first six on the left; we will do more!), flagging of possible preknowledge or item harvesting, unusual time patterns, etc.  It will also aggregate the analyses up a few levels; for example flagging test centers or schools that have unusually high numbers of students with unusually high collusion or time pattern flags.

 

Presentation Abstracts

One Size Does Not Fit All: Making Test Security Configurable and Scalable

Development of an organization’s test security plan involves many choices, an important aspect of which is the test development, publishing, and delivery process.  Much of this process is now browser-based for many organizations.  While there are risks involved with this approach, it provides much more flexibility and control for organizations, plus additional advantages such as immediate republishing.  This is especially useful because different programs/tests within an organization might vary widely.  It is therefore ideal to have an assessment platform that maximizes the configurability security.

This presentation will provide a model to evaluate security risks, determine relevant tactics, and design your delivery solution by configuring test publishing/delivery option around these tactics to ensure test integrity.  Key configurations include:

• Regular browser vs. lockdown browser
• No proctor, webcam proctor, or live proctor
• Login processes such as student codes, proctor codes, and ID verification
• Delivery approach: linear, LOFT, CAT
• Practical constraints like setting delivery windows, time limits, and allowing review
• Complete event tracking during the exam
• Data forensics within the system.

In addition, we invite attendees to discuss technological approaches they have taken to addressing test security risks, and how they fit into the general model.

Let’s Rethink How Technology Can Improve Proctoring

Technology has revolutionized much of assessment.  However, a large proportion of proctoring is still done the same way it was 30 years ago.  How can we best leverage technology to improve test security by improving the proctoring of an assessment?  Much of this discussion revolves around remote proctoring (RP), but there are other aspects.  For example, consider a candidate focusing on memorizing 10 items: can this be better addressed by real-time monitoring of irregular response times with RP than by a single in-person proctor on the other side of the room?  Or by LOFT/CAT delivery?

This presentation discusses the security risks and validity threats that are intended to be addressed by proctors and how they might be instead addressed by technology in some way.  Some of the axes of comparison include:

• Confirming ID of examinee
• Provision of instructions
• Confirmation of clean test area with only allowed materials
• Monitoring of examinee actions during test time
• Maintaining standardized test environment
• Protecting test content
• Monitoring irregular time patterns

In addition, we can consider how we can augment the message of deterrence with tactics like data forensics, strong agreements, possibility of immediate test shutdown, and more secure delivery methods like LOFT.

Contact Us To Get Assessment Solution

ASC partners with Escuela Superior de Administración Pública (ESAP) to develop and securely deliver nationwide civil service exams in Colombia as well as admissions exams for educational programs in the field of civil service.  We worked together to provide a comprehensive solution to modernize these assessments when the COVID-19 pandemic hit.  Learn more below!

Requirements For Escuela Superior de Administración Pública

Escuela Superior de Administración Pública (ESAP) is a public Colombian institution which offers educational programs for bachelor and post graduate degrees, but it’s also involved in the development of custom tests for governmental pre-employment programs. That means they are constantly searching for better solutions to meet their assessing needs. The outbreak of the pandemic in 2020 pushed them to look for online solutions for their civil service exams, but just as important, simple and easy to understand solutions to assess all kinds of populations, from prospective students to high level professionals to fill directive governmental positions all across the country.

When they faced this immediate need, ESAP understood that their best bet would be to look for existing solutions rather than developing their own. However, after trying other online services, they discovered they had very specific needs that required an strategic ally instead of a mere deliverer of prebuilt tests.  Requirements includes professional item banking, online test delivery, and modern psychometrics like item response theory and adaptive testing.  It also required a candidate interface in Spanish, and the option for remote proctoring in Spanish.  ASC’s FastTest platform was an ideal fit.

¿Qué Es La Vigilancia (Proctoring) En Línea?

 

Project Approach

Psychometrician and decision maker teams from Escuela Superior de Administración Pública met with our expert and bilingual staff to adequately understand their needs, as they were not fluent English speakers. From there, we could offer the tools that better fit them and offered our support throughout their planning and designing process. When we started working together, ESAP already had upcoming test deliveries and the best option from scratch was to look for configurations as clean and straightforward as possible, given the fact that ESAP constantly assess populations all across the country and their stakeholders always look for successful projects, where absenteeism rate is not a hurdle to their objectives.

Implementation

The key evidence of the successful implementation of the FastTest platform into ESAP’s daily work is that they were able to keep up with their tight agenda, which includes parallel projects and activities of item banking, online test deliveries, scoring and report making. Such experience made ESAP pioneers in Colombia in using tools designed specifically for these purposes, instead of adapting other platforms they previously used and spending valuable time of their team in operative activities. Our support team took ESAP’s needs very seriously, lending a hand to do some of the operative duties when a big number of different tests were being created.

Furthermore, these accomplishments encouraged ESAP to go for more projects, and the satisfaction of their contracting institutions has been the best presentation for new clients to earn their trust to supply their assessing needs.

Regional Challenges

Colombia is a country located in South America, and its only official language is Spanish. Some percentage of their inhabitants in main cities have skills in other languages, but a vast majority of their total population do not have any. This challenge required a service provider with Spanish language support, allowing to present all dialogs and navigation options in Spanish. This strategy also had to go along with a solid communication campaign to make the whole assessing process easy and even familiar to the examinees. This was a key strategy to ensure their best attitude, as online assessments have been used in Colombia in various contexts, but never before in such big and high stakes projects as these civil service exams and admissions tests.

Technical / Digital Challenges

One of the main concerns the Escuela Superior de Administración Pública had when they considered moving to an online solution was if it would be harder for participants to take their exams, given the fact that Colombia is a country with great challenges regarding the internet coverage and a large portion of their population is not used to handling computers. From the very first test delivery, the absenteeism rate was equal to the one observed in previous paper-and-pencil assessments they led before, allowing them to believe that going for this online strategy was easy to understand, thanks to its straightforward user interface.

The support that ASC provided to the ESAP includes bilingual staff to support any kind of questions they may have as administrators or requests from their stakeholders and legal requirements, as well as assistance to those examinees who might need help or guidance with their processes and procedures. We have also set a communication strategy between ESAP and ASC to let us know when the big test deliveries are coming, to adjust our servers and configurations accordingly.

Scale and Complexity

To understand the nature and complexity of ESAP’s activities, we need to mention that they’re usually involved in all kinds of different and parallel projects. That means they must design a workflow to accomplish their goals in projects where they need to assess different traits in different demographic populations. They manage quick projects to select their provisional staff, assessments to select their prospect students and high stakes projects to select the best candidates to fill all kinds of governmental positions.

To select their students, ESAP uses tests that includes general and specific competencies related to the field they intend to study. For the pre-employment programs, ESAP designs and delivers hybrid tests to assess both personal competencies required for the job and behaviors expected from the candidates. The objectives of the assessing are defined by the kind of process, field, and level (or background) required.

Benefits for Escuela Superior de Administración Pública

The flexible online testing solution has allowed Escuela Superior de Administración Pública to successfully deliver exams for top-level positions with only 1 applicant, all the way up to sessions for more than 12 thousand examinees simultaneously.  ASC has been working together with ESAP for two years now, and we have delivered around 90 thousand online tests during that time.  The project was successful enough that other institutions in Colombia have invited ESAP to serve as a consultant or vendor to them.

 

Authors: Nathan Thompson & Leonardo Copete Parra