| The Arpeggio Suite |
| Software for Cognitive Skills Diagnostic Assessment |
Contributors (modeling and/or programming): Dan Bolt, Henry Chen, Lou DiBello, Sarah Hartz, Robert Henson, Louis Roussos, William Stout, Jonathan Templin.
The Arpeggio Suite consists of four cooperating software components with the central purpose of analyzing test data to provide skills diagnoses using examinee test data. Arpeggio classifies each examinee as a master or a nonmaster for each of a set of user-specified skills that are assessed by a test. Additionally, Arpeggio assesses how well the individual items and the test as a whole discriminate between mastery versus nonmastery for each of the skills in the set.
The software facilitates research in two particular areas:
- Research in applied educational measurement and assessment, where research studies of formative assessment, skills diagnosis in the classroom, etc., can now be conducted. Such work may quickly expose important areas of application to educational issues.
- Psychometric research where there are many research questions concerning the modeling of skills and subsequent statistical analysis that can be addressed by use of the Arpeggio system.
Diagnostic Skills
As a prerequisite to using the software, the user must specify the set of skills judged to be important for examinees performing well on the test. This set typically includes 5 to 15 skills. Then, an incidence matrix (typically referred to as the Q matrix) is specified, indicating for each item which of the specified skills that item requires. It is important to note that an item is allowed to require several skills.
There are two distinct testing settings where a skills diagnosis can be appropriate. In the first setting, the user first specifies a set of important skills and then the test is specifically designed to measure examinee mastery versus nonmastery for each of the targeted skills. That is, Arpeggio can be used for tests that have been designed to measure specified skills. Though this has rarely been the case in current testing practice, research is currently underway to demonstrate just how powerful tests can be in discriminating mastery versus nonmastery of skills when they are designed for diagnostic purposes.
The second, and currently much more commonly occurring setting, is that an already existing test that was not specifically designed for skills diagnosis is analyzed to determine if skills diagnostic information can be extracted from it. Then, as a first step, a moderate-sized set of important skills that the test is judged to require is chosen by carefully studying what skills are required to solve each test item, taking into account what skills the test is designed to measure overall.
In both settings, a careful choice of the skills for measurement and reporting is crucial. Most importantly, the skills need to be chosen based on the user’s needs and purposes. For example, Arpeggio can be useful in the classroom where the teacher is attempting, at the individual student level, to assess which of the important concepts of the last instructional unit have been mastered and not mastered, with a view toward then carrying out individualized targeted instruction focusing on each student’s Arpeggio-reported nonmastered skills. In fact, one of the major uses of skills diagnosis is clearly to facilitate such formative testing. Formative testing refers to testing that is used to directly support teaching and learning processes—as opposed to summative testing that occurs after the instruction is completed, which is the purpose of most standardized tests. Arpeggio can be used in both formative testing and summative testing applications.
Several perspectives can, and ideally should, influence the choice of skills for measurement and reporting, including particular psychometric, curricular, cognitive, and instructional considerations.
- Psychometric: Skills should not be too numerous relative to the length of the test, should be distinct from each other, should combine to have reasonable coverage of the concepts desired to be assessed, and should each be measured reasonably well by several items, so that the test can statistically demonstrate reasonable discrimination between mastery and nonmastery for most of the skill/examinee combinations.
- Cognitive: Cognitive science has taught us that practical cognitive principles, such as emphasizing problem solving and de-emphasizing rote memory, to mention a very obvious example, should when possible be brought to bear.
- Curricular: The choice of skills should be aligned with the curriculum being taught, and should support the curricular learning goals.
- Instructional: The information provided by an Arpeggio analysis of the test data should inform instructional decision making and actions.
Program Details
The Arpeggio Suite contains four software components:
- Arpeggio—Estimates model parameters for the IRT Fusion Model.
- SimArpeggulator—Estimates test-taking population distribution of all possible skill vectors.
- Fast Classifier—Efficiently classifies examinees on each skill.
- Tabulator—Estimates correct classification rates on each skill.
Parameter Estimation Phase
As a first step, Arpeggio calibrates the IRT model called the Fusion Model (Roussos et al. 2007b), which models test performance as a function of skill mastery on each of the set of user-specified important skills. The Fusion Model (FM) is a richly yet judiciously parameterized probabilistic version of an underlying conjunctive deterministic modeling viewpoint that presumes that getting an item correct requires mastery of all of the skills the item requires (as specified in the Q matrix). The model also presumes that there are numerous less important and unspecified skills required for getting each item correct. The influence of these less important skills is parsimoniously modeled by a continuous latent trait that also functions conjunctively and which the user has the choice of including or excluding from the analysis, with exclusion being the more common user choice.
The FM is parameterized for each item by an overall item difficulty parameter that is conditional on mastery of all of the item’s required skills, and a set of item/skill-specific discrimination parameters that measure the statistical discriminability of mastery versus nonmastery of the skill for the item. Because of this carefully constructed parameterization, calibrating the FM (i.e., estimating its item parameters) allows the user to assess how well the test measures mastery versus nonmastery for each item/skill combination. Many other skills diagnostic models do not allow this important focusing on each skill/item combination, which allows the user to evaluate test performance at the individual item/skill level.
In some applications, the number of examinees might be large, for example N = 10,000. In such a case the user can avoid excessively time-consuming analyses by first using Arpeggio on a smaller random (or quasi-random) subset of examinees to calibrate the item and population parameters, followed by using the Fast Classifier to classify the larger set of all examinees.
Classification and Tabulation Phase
After item calibration, another component of the Suite, called the Fast Classifier, estimates the mastery/nonmastery status for each examinee on each of the skills, using the calibrated Fusion Model and a likelihood approach. The Fast Classifier depends upon the FM item parameters as estimated by an earlier run of Arpeggio (possibly on a smaller calibration sample, as indicated above) and also on an estimate of the population distribution of all possible skill mastery level vectors. The latter population distribution is estimated by the software using the component called SimArpeggulator. Finally the fourth component called Tabulator can be used to estimate the correct classification rates for each specified skill, indicating how “reliable” the test is at the skills level.
Parameter Estimation Approach
The parameter estimation method is Markov Chain Monte Carlo (MCMC) of the Fusion Model, which is actually a Bayesian version of the reparameterized Unified Model. The parameterization was developed by Sarah Hartz in her Ph.D.dissertation (Hartz. 2002) and is derived from the Unified Model introduced in DiBello, et al (1995). That MCMC approach provides a versatile tool in this situation and, with a little training and experience, is easy to use. At its simplest level, MCMC uses a stochastic sampling method that estimates the conditional distribution of each parameter given the data, where the Bayesian Model portion of the reparameterized Unified Model is chosen to have suitably vague priors. (More info on MCMC)
Similar to the BILOG option where the shape of the underlying ability distribution is estimated from the data and this population information influences each examinee’s ability estimate, estimation information about each examinee’s mastery level has two sources when applying Arpeggio: (1) the performance of the examinee on all the items; and (2) the estimated (posterior) skills vector distribution for the population of all examinees, which is derived from the calibration data. This empirical Bayes approach to estimation is widely used and well established in statistical work.
In addition to dichotomous items, Arpeggio can also be applied to tests in which one or more items have graded response scores. Arpeggio provides the user a choice of two models for polytomously scored items.
Materials and Research
Included on the CD is a strongly recommended manual (Table of Contents, Introduction), as well as a set of the most pertinent papers for studying skills diagnosis, and a file folder that includes all input and output files for a sample Arpeggio analysis, thus allowing users to practice and learn how the software functions (Input File Description, Example Exercises). There are large numbers of papers and reports on skills diagnosis in general and use of the Arpeggio Suite in particular. A good general reference for the potential of skills diagnosis is the recent Journal of Educational Measurement Special Issue, Volume 44, Number 4, Winter 2007, edited by two of the program’s creators (DiBello and Stout) (see References from Manual).
Additional sources regarding skills diagnostics are:
DiBello, L., Stout, W., & Roussos, L. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. Nichols, S. Chipman & R. Brennan (Eds.), Cognitively diagnostic assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.
Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois, Champaign.
Roussos, L., DiBello, L.V., & Stout, W.F. (2007a). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics 26. Amsterdam, The Netherlands: Elsevier,
Roussos, L. A., DiBello, L.V., Stout, W. F., Hartz, S. M., Henson, R. A., & Templin, J. H. (2007b). The fusion model skills diagnostic system. In J. Leighton & M. Gierl (Ed.), Cognitive diagnostic assessment for education: Theory and applications. New York: Cambridge University Press.
The FM and the associated Arpeggio Suite have been a source for high quality research, with an APA-awarded outstanding technical thesis, an NCME best technical research award, an NCME best practical application award, and an NCME best thesis award—all having been awarded for FM-based Arpeggio System research.
System Requirements
Windows 95 or Higher.
|
Order Options and Details
|
 |
| |
Products
Your Account
Your cart
