I recently read a disturbing article in the New York Times regarding the “Opt-Out of Educational Assessment” movement in the State of New York.  I find this downright appalling.  There are many sorts of standardized assessments, and the ones we are talking about here exist to improve the quality of education.  They check to see whether students are learning the curriculum, thereby providing some measure of accountability, as well as an index of improvement, to the teachers, schools, districts, and States.  They are used to help students, not hinder them.  By saying that you do not want your students involved in this, you are saying that you don’t want any accountability, and that you really do not care about education at an aggregate level.  You are happy just to send your kids to school and hope that the teacher teaches them something over the course of the year.

It’s also interesting that the article mentions this being in response, in part, to the fact that test scores “plummeted” after the introduction of Common Core.  The author here is definitely biased and/or uninformed.    Why were scores lower?  Because we no longer allowed States to doctor their data.  Under No Child Left Behind, States were required to assess students but could set the cutscores wherever they wanted.  So the backwoods States obviously set the bar pretty low, and reported that large numbers of their students were proficient.  They were also free to adjust their curriculum, i.e., teach 4th grade math in 5th grade so that, wow… the 5th graders seem to do well on math!  The move to Common Core was in part to prevent these two schemes, and of course such states saw much lower numbers of 5th grade math students score as proficient when being tested on actual 5th grade math, with a standard set that was similar to other states.  The Opt-Out of Educational Assessment movement is fighting this.

What is happening here, then is that people are blaming the test purely out of reactionary tendency.  An analogy might be that you are on a weight loss program and have for years intentionally miscalibrated your bathroom scale to read 10 pounds lower.  You get a fitness coach that calls you on it and forces you to correctly calibrate it, and then to actually weigh yourself once a week to see if you are losing weight.  So you blame 1) the scale, and 2) your fitness coach?  That is probably not going to help improve your fitness.  Or the education system.  But go ahead and opt out of weighing yourself.

The State of Connecticut has enacted a new policy to require all 11th grade students to take the SAT (New York Times, 6 Aug 2015).  As a lifelong advocate of quality assessment, this concerns me.  I consider Instruction and Assessment to be the two primary components of Education; instruction without assessment is like a weight loss program without a scale or mirror.  But the assessment has to be done right.  How do we know what is right?  To evaluate this, we need to couch the discussion in the most important concept in assessment: validity.

Validity

Validity refers to whether the score interpretations being made form a test are those for which the test was intended and designed, and are supported by evidence.  One of my graduate school mentors used to say, “The right tool for the right job.”  You could pound in a nail with a screwdriver, but that is not what it was designed for.  The same goes for assessment.  An actual need has to be identified that requires assessment to be done, and the assessment should be designed for that need.  So let’s look at the situation in Connecticut.

What does Connecticut Need?

Connecticut needs a measure of student success, so that they can evaluate whether a student is learning the curriculum.  A curriculum is designated by the State, and is currently moving towards the Common Core, which is a big improvement over the old days where each State did their own… leading States with poor education systems to simply dumb down their curriculum and lower their standards to make it appear that their students were smart.  A test of what a student has learned is achievement.

For what was the SAT designed?

The Scholastic Aptitude Test (SAT) is, as the name suggests, a test of aptitude, not achievement.  The story behind the SAT: A century ago, colleges and universities had their own entrance/placement exams.  They realized this was stupid and banded together to form the (creatively named) College Board, to make a common exam, as well as other things.  The test was designed to best predict success in college – nothing about how much the student learned in high school.  Here’s an article that discusses that purpose.  The approach they used was to assess cognitive ability via a few areas, currently Critical Reading, Math, and Writing (scores here).  This isn’t a bad idea, given the purpose of the test.  But the SAT definitely does not assess, for example, how much of the Science curriculum the student has learned in high school

Is there too much testing?

Connecticut obviously felt pressure from parents and teachers that there was too much testing for 11th graders; this article in a local paper shows the sentiment.  I don’t want to dispute that.  What concerns me is that Connecticut seemed to have a choice of eliminating their achievement tests or the SAT.  They need an achievement test to track student learning.  Instead, they dropped the achievement test built for that purpose and kept the SAT, which is designed for a completely different purpose.  What irks me even more is that people who should know better, such as the Superintendent in that article, are apparently not even aware of the basic concepts of assessment.  Instead, he is happy to have to go through the work of trying to set new cutscores on a test that is being misapplied.

So how did this happen?

I think the biggest reason for this, and many other misuses of assessment, is that the vast majority of people, including educators and politicians, are psychometrically illiterate.  Educators are incredibly hardworking, but their focus is instruction (and rightly so).  They have never had exposure to basic concepts like validity and reliability.  As a scientific field as well as a business industry, we need to better educate end users and stakeholders.  This is not limited to K12 Education, by the way.

As for Connecticut, I’d suggest one thing to start: follow the money.  That local article says that the SAT is free for all CT students.  We all know there is no such thing as a free lunch. Someone made a very good sale!

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source

Local Newspaper Discusses Test Fraud

The St. Paul Pioneer Press recently published an article on instances of test fraud and security breaches experienced in K-12 assessments in the State of Minnesota (where Assessment Systems is located).  The article does a good job at explaining some of the issues:

 

  • The root issue is that the State needs to protect the massive investment it has made in their assessments, and how the validity of the assessments is crucial in making important decisions.  Test fraud has a negative impact, wasting taxpayer money among other negative outcomes.
  • It is noteworthy that there are no stakes for the students tied to the assessments mentioned; that is, they stand nothing to gain by cheating or stealing.  Of course, the vast majority of student and proctors are upright!
  • When it comes to test security, an ounce of prevention is worth a pound of cure, so MN has focused on the prevention.  I am excited to hear, however, that they are considering the use of statistical indices.

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source

Adaptive testing in medicine is one of the newer applications of this modern psychometric paradigm.  A recent article in the The Journal of Bone & Joint Surgery, Volume 95, Issue 20 evaluates the use of computerized adaptive testing (CAT) approaches in assessment of psychological factors surrounding perceived disability and pain.  Researchers from the Partners health system report favorable findings, namely that the CAT assessments provided in the Patient Reported Outcomes Measurement Information System (PROMIS) correlate highly with older paper-based assessments that will typically take much longer to administer.

Such findings are typical in this sort of research.  For example, the MHCAT project, in which Assessment Systems was involved, published this article which notes that test length was reduced by 95% – yes, 95%! – while maintaining a correlation of the short CAT assessment with the original long assessment of 0.93.  Why would a practitioner keep using an outdated assessment that takes 20 times as long to complete?

The Bone & Joint article was also discussed in the Helio website on medical news.

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.

Newsletter Sign Up
First Name*
Last Name*
Email*
Company*
Market Sector*
Lead Source

 Are states prepared?

Computerized Adaptive Testing (CAT) has been in the news for quite some time as a technology being adopted at the state, and district level to improve the academic assessment performance for K-12 students. Legislation such as NCLB, and a push for a common-core standard have put pressure on many states to find solutions to improve performance and accountability in the classroom. For many state agencies the solution is CAT testing. But are states prepared for this innovation?

In mid-November Governor Malloy of Connecticut appropriated $14 million dollars for the use of CAT testing in Connecticut K-12 schools. This funding will be used to administer CAT end of year tests in grades 3 through 8 and 11. The CAT tests will be aligned to Smarter Balanced Assessments standards used by several other state education agencies across the United States.

Although Connecticut’s commitment to CAT testing will help improve the education performance of Connecticut’s students, many are still concerned about the state’s shortfall in the amount of money being appropriated with computer and bandwidth upgrades. These upgrades are paramount for a successful adoption of CAT testing.  According to WTNH Hartford, the state was only able to distribute $10 million out of the $24 million appropriated for technology upgrades in schools. This poses a central issue, will CAT be implemented successfully?

CAT testing is the innovative technology our country needs to successfully improve our K-12 education. Connecticut’s plan for CAT testing is a great example of what many states will have to deal with very shortly if funding is appropriated for CAT testing.  For many states, significant state-wide upgrades on technology are inevitable for CAT testing to be successfully implemented.

It has already been apparent in many states currently using CAT testing that they are seeing improved performance. What is still not apparent is how CAT testing will be implemented successfully in states where significant technology upgrades are still needed.

Link to WTNH article:

http://www.wtnh.com/news/politics/malloy-14m-in-grant-money-for-schools

The Huffington Post recently published an article discussing the role that computerized adaptive testing (CAT) plays in K-12 educational assessment.  It presents some of the issues in the field of standardized testing, and then discusses how adaptive testing in education can address some of those issues.

This article is a refreshing departure from those typically seen in the mainstream media, which often complain about student assessment without taking the time to become informed on the topic, most notably understanding all the benefits that it provides and advantages over alternatives.  In fact, the article specifically notes that much of the debate going on just completely misses the point.  I recently read an article in a parenting magazine that was on the other end of the spectrum, bordering on fear-mongering sensationalism.

Some of the advantages of adaptive testing in education discussed:

  • CAT helps refocus learning on instruction
  • CAT can reduce the “corruptive pressures” and teaching to the test
  • CAT can enhance security: there are no bubble sheets, which completely eliminates the possibility of teacher modification, and can provide accommodations automatically, greatly reducing the chance of teachers cheating during the accommodation process (e.g., reading the test questions to a student)
  • CAT provides more accurate scores for high-ability and low ability students, and for disabled students
  • CAT tests can cost less and typically take much less time to administer (research suggests 50% less time testing)

Of course, CAT is not a panacea to all the problems facing student assessment, much less our educational system.  However, it has been shown to have numerous benefits, supported by a large body of scientific literature.  The SMARTER Balanced Assessment Consortium has the right idea in utilizing CAT technology to meet the assessment needs of its member states.  I am eagerly looking forward to seeing how those endeavors turn out.

Here is the full article URL: http://www.huffingtonpost.com/darrell-west/how-technology-can-stop-c_b_3784392.html

Interested in adaptive testing in education?

Sign up for our newsletter on the left, or contact us to talk with one of our assessment consultants.

The No Child Left Behind (NCLB) Act is an important piece of US legislation that governs assessment in the K-12 education system.  It is currently up for re-authorization, and language is being considered that will specifically mention computerized adaptive testing (CAT).

“Adaptive testing is proven to be a more effective tool for assessing student performance and competence than standard paper-based testing that only shows whether a student is on grade level.”

-Rep. Tom Petri, R-Wisconsin

Adaptive testing and NCLB work well together, as the advantages of adaptive testing translate well into the classroom as well as to accountability systems.

CAT continues to grow more widespread, especially in relation to the SMARTER Balanced Consortium.  However, most online CAT delivery platforms remain too expensive for many school districts.  FastTest offers an affordable alternative that will deliver CAT assessments which help prepare students for this more sophisticated and precise form of educational assessment.

Read the full article from EdWeek here: http://blogs.edweek.org/edweek/DigitalEducation/2013/07/house_nclb_rewrite_contains_ad.html.

The efforts of the SMARTER Balanced Assessment Consortium are leading to more mentions of computerized adaptive testing in the news.  I recently came across the following article that covers a paper by Mark Reckase, one of the most respected researchers in the field.

http://blogs.edweek.org/edweek/curriculum/2011/05/computer-adaptive_testing_pose.html

Want to improve the quality of your assessments?

Sign up for our newsletter and hear about our free tools, product updates, and blog posts first! Don’t worry, we would never sell your email address, and we promise not to spam you with too many emails.