What is the role of advanced statistics in experimental research?

The authors solicit feedback from Div. 3 members about the role of advanced statistical analyses in research based on a true experimental design.

By Renee Cloutier, MS, and Ellyn Bass, MA

Greetings. We, , Renee Cloutier, MS (University of North Texas) and Ellyn Bass, MA (University of Nebraska Omaha), are reaching out to Div. 3 in hopes of starting a discussion on the role of advanced statistical analyses in research based on a true experimental design (e.g., IV manipulations). There appears to be a call for researchers to employ advanced statistical techniques such as structural equation modeling and hierarchical/multilevel modeling) for analyzing experimental data (e.g., MacCallum & Austin, 2000; Russell et al., 1998). We wonder, do the members of Div. 3 believe that these advanced statistical techniques advance experimental research above and beyond that which could be achieved with simpler techniques? In other words, do the gains of these analytic approaches (e.g., detection of nuanced, non-linear and/or small effects) outweigh the practical costs in overall interpretation of the findings? For researchers who have analyzed data in this way, how have you balanced the costs of running an experiment in terms of time, money and materials, with the need for large samples when using more advanced statistical techniques? Finally, are there any specific statistical techniques that Div. 3 members perceive to be underutilized in psychological research broadly or within their field more specifically?

There will likely be some diversity in responses depending on each member’s substantive area of research so we welcome all responses to any of the listed questions either through the newsletter or privately via emailing Renee or Ellyn. Thank you so much for taking the time to read this, and we look forward to hearing from you.

References & Resources

Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390-412.

MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.

Muthén, B., & Curran,P. (1997). General longitudinal modeling of individual differences in experimental designs: A latent variable framework for analysis and power estimation. Psychological Methods, 2, 371-402.

Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209.

Russell, D. W., Kahn, J. H., Spoth, R., & Altmaier, E. M. (1998). Analyzing data from experimental studies: A latent variable structural equation modeling approach. Journal of Counseling Psychology, 45(1), 18.