This column provides a brief synopsis and reference information for select recent journal publications. Note that this is only a short sampling of publications and is not intended as a comprehensive listing.
Words or Numbers? Communicating Probability in Intelligence Analysis
Following the major intelligence failure that led to the 2003 war in Iraq, intelligence organizations implemented policies for communicating probability in their assessments. They focused on the use of standardized linguistic lexicons that involve the use of an ordered set of probability terms (e.g., highly likely) that are associated with numeric ranges. The benefits and drawbacks of this approach are discussed drawing on psychological research on probability communication and the effectiveness of standardized lexicons.
Dhami, M. K., & Mandel, D. R. (2021). Words or Numbers? Communicating Probability in Intelligence Analysis. American Psychologist, 76, 549-560. http://dx.doi.org/10.1037/amp0000637
An Iterative Parametric Bootstrap Approach to Evaluating Rater Fit
To detect problematic scoring patterns, two rater fit statistics, the infit and outfit mean square error (MSE) statistics are routinely used. A common practice is that researchers employ established rule-of-thumb critical values to interpret infit and outfit MSE statistics. Prior studies have shown that these rule-of-thumb values may not be appropriate in many empirical situations. Parametric bootstrapped critical values for infit and outfit MSE statistics provide a promising alternative approach to identifying item and person misfit in item response theory (IRT) analyses. In this study, a bootstrap procedure is illustrated that researchers can use to identify critical values for infit and outfit MSE statistics, and a simulation study is used to assess the false-positive and true-positive rates of these two statistics. We observed that the false-positive rates were highly inflated, and the true-positive rates were relatively low. Thus, an iterative parametric bootstrap procedure to overcome these limitations was proposed.
Guo, W., & Wind, S. A. (2021). An iterative parametric bootstrap approach to evaluating rater fit. Applied Psychological Measurement, 45, 315-330.
Computerized Adaptive Testing for Testlet-based Innovative Items
Increasing use of innovative items in operational assessments has shed new light on the polytomous testlet models. Several scoring models are examined when polytomous items exhibit random testlet effects: The partial credit model (PCM), testlet-as-a-polytomous-item model (TPIM), random-effect testlet model (RTM), and fixed-effect testlet model (FTM). The performance of the models was evaluated in two adaptive test situations where testlets have nonzero random effects. The outcomes of the study suggest that, despite the manifest random testlet effects, PCM, FTM, and RTM perform comparably in trait recovery and examinee classification. The overall accuracy of PCM and FTM in trait inference was comparable to that of RTM. TPIM consistently underestimated population variance and led to significant overestimation of measurement precision, showing limited utility for operational use.
Kang, H., Han, S., Betts, J., & Muntean, W. (2021). Computerized adaptive testing for testlet-based innovative items. British Journal of Mathematical and Statistical Psychology, Doi: https://doi.org/10.1111/bmsp.12252
Clarifying Causal Mediation Analysis for the Applied Researcher: Defining Effects Based on What We Want to Learn
The goal of this article is to improve the understanding and application of causal mediation analysis. It describes the difference between causal inference and traditional mediation analysis, and emphasizes the need for explicit causal thinking and the causal inference approach in mediation analysis. Existing effect types are explained paying special attention to motivating these effects with different types of research questions and with concrete examples. Information is provided about analyses that are explanatory in nature as well as those that are interventional.
Nguyen, T. Q., Schmid, I., & Stuart, E. A. (2021). Clarifying causal mediation analysis for the applied researcher: defining effects based on what we want to learn. Psychological Methods, 26, 255-271. http://dx.doi.org/10.1037/met0000299
Guidelines for the “Publication Corner” column
Have you seen a recent publication that you think would be of great interest to Div. 5 membership? Please feel free to share suggestions for publications to include in this column. Please send your ideas to The Score Editor Gill Sitarenios. Please be sure to include full reference details and a short synopsis (two to three sentences) of the publication. Use “Publication Corner” in the subject line of the email.