This column provides a brief synopsis and reference information for select recent journal publications. Note that this is only a short sampling of publications and is not intended as a comprehensive listing.
Managing Validity Versus Reliability Trade-Offs in Scale-Building Decisions
The goal of this article is to make plain that the source of validity versus reliability tradeoffs is systematic error that contributes to item communality. This article highlights common tradeoffs in six contexts: item content, item construction, item difficulty, item scoring, item order, and item analysis. Five recommendations are provided for managing tradeoffs. Reviewers should require scale builders to declare how validity and reliability will be prioritized and penalize those who resolve tradeoffs in goal-inconsistent ways.
Clifton, J.D. W. (2020). Managing Validity Versus Reliability Trade-Offs in Scale-Building Decisions. Psychological Methods, 25, 259 –270. ISSN: 1082-989X https://dx.doi.org/10.1037/met0000236
Will the Questions Ever End? Person-Level Increases in Careless Responding During Questionnaire Completion
This article reports on the results from two studies that examine the extent to which careless responding occurs in responding to self-report questionnaires. The results showed consistent evidence that participants responded more carelessly as they progressed further into a questionnaire. There was mixed evidence that participants who were warned that carelessness would be punished displayed smaller increases in carelessness, and mixed evidence that increases in carelessness were greater within an unproctored online study than within a proctored laboratory study. These findings help pinpoint when careless responding is most likely to occur, and the authors suggest some possible effective preventive strategies.
Bowling, N. A., Gibson, A. M., Houpt, J. W., & Brower, C. K. (2020). Will the Questions Ever End? Person-Level Increases in Careless Responding During Questionnaire Completion
Organizational Research Methods, 1-21. https://doi.org/10.1177%2F1094428120947794opens in new window
Quantity Over Quality? Reproducible Psychological Science from a Mixed Methods Perspective
This paper addresses (a) how and to what extent existing quantitative reproducibility guidelines apply to mixed methods study designs; and (b) whether existing reproducibility guidelines be improved by incorporating best practices from qualitative research and epistemology. The authors offer 10 key recommendations for use within and outside of mixed methods research. Finally, the authors argue that good science and good ethical practice are mutually reinforcing and lead to meaningful, credible science.
Reischer, H. N., & Cowan, H. R. (2020). Quantity Over Quality? Reproducible Psychological Science from a Mixed Methods Perspective. Collabra: Psychology, 6(1), 26. DOI: https://doi.org/10.1525/collabra.284
Evaluating Equivalence Testing Methods for Measurement Invariance
Measurement Invariance (MI) is often assessed using a chi-square difference test, or by examining change in goodness-of-fit indices (ΔΔGOFs). Both of these commonly used methods have limitations. An equivalence test (EQ) is one possible alternative to the chi-square difference test. Due to concerns with the statistical power of the EQ test, an adjusted version (EQ-A) has been proposed. The current study evaluated the Type I error and power of both the EQ and EQ-A, and compared their performance to that of the traditional chi-square difference test and ΔΔGOFs. The EQ was the only procedure that maintained empirical error rates below the nominal alpha level. The EQ requires larger sample sizes than traditional difference-based approaches or using equivalence bounds based on larger than conventional RMSEA values (e.g., > .05) to ensure adequate power rates.
Counsell, A., & Cribbie, R. (2020, March 25). Evaluating Equivalence Testing Methods for Measurement Invariance. https://doi.org/10.1080/00273171.2019.1633617
Guidelines for the “Publication Corner” column
Have you seen a recent publication that you think would be of great interest to Div. 5 membership? Please feel free to share suggestions for publications to include in this column. Please send your ideas to The Score editor, Gill Sitarenios . Please be sure to include full reference details and a short synopsis (2–3 sentences) of the publication. Use “Publication Corner” in the subject line of the email.