Ever wanted to take a graduate-level course in assessment and psychometrics? Well, here’s your chance to access that level of learning, but without the price tag of tuition or having to travel thousands of miles.
ASC’s cofounder and CEO, Dr. Nathan Thompson, was invited to serve as an instructor for several years in the Masters in Medical Education program offered by the University of Cincinnati and the Cincinnati Children’s Hospital Medical Center. The program has a very specific and admirable goal: training the next generation of medical educators. Most of the students were professors or deans at prestigious medical schools, extremely accomplished in their field, but wanted to learn more about Education. This particular course was dedicated to the role of assessment and psychometrics in medical education, since the topic is not only vital to student learning, but also to accreditation of the medical school.
The content of the online psychometrics course was tailored to this specific focus. Rather than generally surveying issues in psychometrics, the course focused on what was most relevant in graduate medical education. Entire weeks were devoted to performance testing and simulations, while psychometric topics such as linking and equating were downplayed; these are hugely important in K12 and Certification assessment, but not medical education.
Nevertheless, there is a huge amount of content in this course and the textbook that is relevant to general practice of psychometrics. For this reason, we’ve made the lecture notes available for free. This closely aligns with ASC’s mission: Improving Assessment via Technology and Psychometrics. Our fundamental goal is to improve the quality of assessment around the world, and a core part of that goal is educating more practitioners in the psychometrics that goes into developing a high quality educational assessment.
Because this was an online course, lecture notes were written out rather than drawn on a whiteboard during an in-person class. All the content not covered by the book, as well as some from the book, is in the lecture notes. The course was 10 weeks long, so there are 10 PDF files of notes. Some are short (1-2 pages) but some are lengthy. Here are the topics:
| Week | Topics | Book chapters |
| 1 | Intro; validity | 1,2 |
| 2 | Reliability, Standard error, generalizability theory | 3,4 |
| 3 | Classical test theory (statistics) | 5 |
| 4 | Standard setting | 6 |
| 5 | Written tests and MCQs | 7 |
| 6 | Observational assessment | 8 |
| 7 | Performance tests | 9 |
| 8 | Simulations | 10 |
| 9 | Item response theory | 11 |
| 10 | Advanced topics: adaptive testing, linking, DIF | 12 |
The textbook used for this course was Downing and Yudkowsky (2009). You can preview much of it at Google Books here. You don’t need a copy of the book to gain value from reading the notes, but it will definitely help. However, if you are interested in a more general treatment of psychometrics, you’d probably prefer books like Shultz and Whitney (2005) Measurement Theory in Action (basic level) or Embretson and Riese (2000) Item Response Theory For Psychologists (advanced).
