Measurement applications often entail interval level treatment of score metrics. A seemingly appealing aspect of item response theory (IRT) is a hope that IRT metrics come close to providing a basis for such interval-level interpretations. Successfully fitting a model such as the two-parameter logistic (2PL) model to item response data is sometimes taken as justification for interpreting the latent metric as interval.
While psychometricians caution against such interpretations, the cautions often go unheeded in practice. What may be needed is clearer demonstrations as to how and why metrics can be systematically distorted even when traditional IRT models like the 2PL provide a close fit. We have recently been exploring this issue in relation to IRT models that introduce asymmetries into the item characteristic curves (ICCs). An example is the logistic positive exponent (LPE) model of Samejima (2000), a model that relates asymmetry in the ICC to the underlying complexity of the item. Items that are highly complex (e.g., a math problem involving several cognitive steps to a correct solution) can be expected to be positively asymmetric, while items that are not complex (e.g., a multiple-choice vocabulary item, for which proficiency-based-guessing is also present) are expected to be negatively asymmetric. Positive asymmetry implies the ICC increases more rapidly to the left of the inflection point than it declines to the right; negative asymmetry implies the exact opposite.
Bolt and Liao (2021a; 2021b) show how the presence of ICC asymmetries, when systematic across items, are easily absorbed by the IRT metric, allowing models with symmetric ICCs (e.g., the 2PL) to frequently fit despite the fact that the true interval metric implies items with systematic asymmetries. In both papers we provide results from simulation studies in which items are generated with asymmetries (either positive or negative) and then fit using the symmetric 2PL model. Interestingly, when fitting the 2PL, an item with a strong positive asymmetry does not emerge as misfitting when calibrated with other items that have similarly strong positive asymmetries. By contrast, a true symmetric 2PL item will emerge as misfitting when calibrated with the same positively asymmetric items. The exact same phenomenon occurs for negative asymmetries. The findings can be seen as implying that the latent metric has absorbed the systematic asymmetry. The metric effectively adapts in ways that allow the symmetric model to best fit the data, even though in actuality the true relationship is asymmetric. The same results also explain why it has been found exceedingly difficult to estimate the LPE in empirical applications—there is no way to identify the true nature of asymmetry if the latent metric can simply adapt to accommodate any systematic direction to the asymmetry.
The practical implications can be metrics in which the latent units are either shrunken at the upper end (Bolt and Liao, 2021b) in the presence of systematic positive asymmetry, or at the lower end (Bolt and Liao, 2021a) in the presence of systematic negative asymmetry. Most importantly, these asymmetries have clear psychological interpretations; Lee and Bolt (2018) demonstrated by simulation how the nature of the underlying latent response subprocesses in items can often render substantial and predictable asymmetries in ICCs (see also Bolt and Liao, 2021a).
This set of results has been used to provide possible explanations for a number of phenomena seen in measurement practice. For example, the likely presence of positive asymmetry in ICCs may well explain the frequently seen score deceleration phenomenon in math, where year-to-year growth at higher grade levels is often a small fraction of what is seen at early grade levels (Bolt, Deng and Lee, 2014). A similar phenomenon may explain fadeout effects seen for interventions in the area of math (Wan, Bond, Lang, Clements, Sarama and Bailey, 2021), where the initial gains seen from an educational intervention rapidly decline over time. The existence of negative ICC asymmetries, such as due to proficiency-based guessing, may well explain the observation of Matthew effects in reading (Protopapas, Parrila and Simos, 2016). The presence of negative asymmetry has also been argued to explain the positive correlation often seen between differential item functioning and item difficulty on verbal tests (Bolt and Liao, 2021a). Either form of systematic asymmetry could also contribute to treatment effect heterogeneity (Schochet, Puma and Deke, 2014) when comparing subpopulations of different mean proficiency levels.
Similar considerations can be given to metrics constructed from IRT models for polytomously scored items. As there are many more IRT models that can be considered in such contexts, the potential for metric distortions depending on which models are chosen as the basis for modeling is a real possibility (Lyu and Bolt, in progress).
More generally, we believe this work highlights the importance of thinking carefully about response process as a contributing factor to the interpretation of latent metrics generated in IRT. There is generally no good reason to believe that such processes are best understood through the traditional logistic (or normal ogive) functions used to relate the latent metric to expected item performance. Selection of an IRT model should be viewed as being as much about defining a latent metric as it is about defining a relationship between the latent trait/proficiency and item score.
References
Bolt, D.M. and Liao, X. (2021a, November). On the positive correlation between DIF and difficulty: A new theory on the correlation as methodological artifact. Journal of Educational Measurement.
Bolt, D.M. and Liao, X. (2021b; under review). Item complexity: A neglected psychometric feature of test items?
Bolt, D. M., Deng, S. and Lee, S. (2014). IRT model misspecification and measurement of growth in vertical scaling. Journal of Educational Measurement, 51(2), 141-162.
Lee, S. and Bolt, D. M. (2018a). Asymmetric item characteristic curves and item complexity: Insights from simulation and real data analyses. Psychometrika, 83(2), 453-475.
Protopapas, A., Parrila, R. and Simos, P. G. (2016). In search of Matthew effects in reading. Journal of Learning Disabilities, 49(5), 499-514.
Samejima, F. (2000). Logistic positive exponent family of models: Virtue of asymmetric item characteristic curves. Psychometrika, 65, 319-335.
Schochet, P. Z., Puma, M. and Deke, J. (2014). Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods. NCEE 2014-4017. National Center for Education Evaluation and Regional Assistance.
Soland, J. (2017). Is teacher value added a matter of scale? The practical consequences of treating an ordinal scale as interval for estimation of teacher effects. Applied Measurement in Education, 30(1), 52-70.
Wan, S., Bond, T. N., Lang, K., Clements, D. H., Sarama, J. and Bailey, D. H. (2021). Is intervention fadeout a scaling artefact? Economics of Education Review, 82, 102090.