Author Note: Correspondence concerning this article should be addressed to Jolynn Pek, PhD, Department of Psychology, The Ohio State University, 228 Lazenby Hall, 1827 Neil Avenue, Columbus, OH 43210. E-mail Jolynn Pek.
The American Psychological Association (APA) describes quantitative psychology as a field that “develops methods and techniques for the measurement of human behavior and attributesopens in new window.” The focus is on methods, which broadly concerns tool development or the “what” of quantitative psychology. Not emphasized is methodology, which focuses instead on the motivation and justification of tools and their implementation, or the “why” and “how” aspects of quantitative psychology. Focused on methods over methodology, much of the research conducted by quantitative psychologists consists of complex technical refinements that are relatively inaccessible to substantive researchers. Unfortunately, this narrow focus has led research in quantitative psychology to develop relatively separately from “substantive research”, that is, research within content areas of psychology. Not surprisingly, quantitative psychologists also tend to have limited involvement in substantive psychological research projects (Wijsen et al., 2022), with the most common role (in our experience) being to bless a proposed study design or analysis (i.e., sanctifying data; Tukey, 1969), or answer the occasional question about model specification or software implementation. This situation is counterproductive, given the essential role of methodology in advancing psychological science. We thus observe that current quantitative developments tend to go against Tukey’s (1962, p. 13–14) maxim, “Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.” Methods research tends to increase the precision of existing solutions to old problems, often with diminishing returns, whereas methodology often leads to approximate solutions to important new problems.
Expanding on this distinction (see also Beaujean, 2021 and Pek & Bauer, 2023), a method refers to a tool or technique (e.g., PROCESS macro; Hayes, 2022), whereas methodology combines method with logos (i.e., discourse), becoming “knowledge about, or a reasoned argument for, doing something a certain way” (Beaujean, 2021). Thus, methodology refers to reasoned argument about using a method. For example, a formula for computing reliability is a method whereas discerning which measure of reliability is pertinent to a particular design reflects methodology (see Cronbach & Shavelson, 2004; Shrout & Fleiss, 1979). Another example which distinguishes method from methodology is the calculation of effect size. Here, method reflects formulas for unstandardized versus standardized effect sizes whereas methodology maps onto the justifications required for using unstandardized over standardized effect sizes (see Baguley, 2006; Pek & Flora, 2018; see also King, 1986).
The methodological developments of many early psychologists were highly impactful. For example, according to Google Scholar, Thurstone’s (1927) law of comparative judgment as a mathematical formulation of discriminating between two choices (methodology) has been cited 8,052 times as a precursor to modern approaches to cognitive modeling. Another example is Cronbach’s (1951) paper on coefficient alpha as a quantification of reliability (cited 60,463 times). Cronbach (1951) is cited for providing a closed form equation to coefficient alpha (method) but also for its contribution to classical test theory (methodology). A more modern example is Baron and Kenny’s (1986) paper on mediation and moderation (cited 117,726 times), which did not introduce a method but a refreshing perspective on how to think about examining underlying mechanisms in a process (a methodology). What is common among these methodological papers is that they highlight issues concerning how best to approach and conduct research, which remains a common language across disciplines in psychological science.
Given the high impact of many methodological contributions, why has contemporary research in quantitative psychology instead favored methods? We highlight two systemic constraints that seem to motivate the narrowed focus on developing methods (i.e., tools in a statistical toolbox) over methodology (i.e., justifications for applying statistical tools in particular research contexts). The first is that there is a limited number of graduate programs in quantitative psychology. Specifically, there are fewer than 30 quantitative doctoral programs compared to almost 300 psychology PhD programs in the U.S. and only 8% of job postings requiring expertise in quantitative psychology were housed within a quantitative psychology program (Aiken, 2020). While counts from 2007 to 2017 based on the National Science Foundation Survey of Earned Doctorates indicate a growing number of quantitative psychology graduates over time (see Aiken, 2020), the opportunities for these graduates to develop a career focused on methodology are limited. This is the second constraint: quantitative psychologists are typically recruited into faculty positions to teach tools rather than to develop new methodological approaches. In a systematic study of faculty job postings that involve teaching quantitative methods from 2017 to 2018, Aiken (2020) found that 92% of these postings named at least one substantive area of specialization. Implicitly, these job descriptions regarded quantitative expertise as secondary to candidates’ substantive areas of research, valuing method application without regarding methodological development as a discipline of research in its own right. In earlier work, Aiken et al. (2008) highlighted this continuing trend for “twofer” (two for the price of one) faculty positions that seek faculty who have two responsibilities: (a) teach and conduct research in a substantive specialty area and (b) teach advanced graduate-level quantitative methods courses while providing quantitative consulting. It is unclear how faculty in such positions are evaluated for tenure in terms of their scholarship (e.g., substantive research versus method application and development), but Aiken et al. (2008) note that early career faculty hired into such positions historically had difficulty achieving tenure.
These structural constraints reflect a misalignment of what constitutes quantitative expertise (method versus methodology) among psychologists with substantive expertise versus quantitative psychologists, which likely influences incentives for promotion and tenure for quantitative psychologists. The focus on method over methodology incentivizes publishing highly technical and narrow work (e.g., deriving second-order derivatives for a standard error in a finite mixture structural equation model; Pek et al. 2011) over methodological work (e.g., test theory by Lord & Novick, 1968) that has inspired the development of new methods and methodology. Unfortunately, in the current research climate, highly technical methods work seems to be regarded as a mark of intellectual prowess, in contrast to the development and dissemination of widely relevant and applicable methodology (Fabrigar et al., 1999; Harlow, 2017). From time to time, quantitative psychologists, from graduate students to senior faculty, seem to emphasize the technical level of their work like peacocks displaying their feathers to impress, with other psychologists finding this posturing largely incomprehensible and bewildering. Given this value system, scholarly work on methods instead of methodology is more straightforward to publish. Highly technical method developments are primarily evaluated for errors in their derivation or implementation with little controversy. However, methodological work can be contentious and take a long time to publish. For example, Baron & Kenny (1986) was initially rejected from Psychological Bulletin before being published in Journal of Personality and Social Psychology. More recently, Flake et al. (2022) was rejected from Psychological Bulletin and Perspectives in Psychological Science before appearing in American Psychologist. Some method papers are also highly cited (e.g., Preacher & Hayes, 2004, cited 20,328 times), motivating such publications. Finally, because most substantive experts do not have the training to fully evaluate technical methods developments, they may tend to emphasize quantity over quality of publications when evaluating their quantitative colleagues. Taken together, systemic structures and incentives have likely promoted method over methodological development, leading to the increasing separation of quantitative psychology from content in psychology that produces piecewise knowledge on methods that contributes little to improving psychological science. Tukey (1962, p. 7–8) regarded this pursuit of “apparent neatness” in optimizing methods an “ossification” in contrast to “real research“ in methodology.
How can we refocus our research on methodology rather than methods? Let us begin by redefining quantitative psychology to be a field that prioritizes methodological over method development (cf. APA’s description of quantitative psychology that highlights technical developments). Like statisticians and engineers, quantitative psychologists have done well in developing and refining tools (Thissen, 2001). Like data scientists, quantitative psychologists have applied methods to increasingly complex data. To distinguish ourselves from statisticians and data scientists, quantitative psychologists should attend more to the application of our tools to substantive psychological problems and embrace the uncertainty and approximate nature of their results (see also Tukey, 1962). Our scholarly activities should better align with our redefined identity as methodological methodologists. Instead of sanctifying data with cookbook methods, we should emphasize our roles as data detectives by advancing methodology (Tukey, 1969). Additionally, the quantitative psychologist should be an integral part of the scientific team tackling a substantive research problem, not a contractor-for-hire to merely approve or apply methods to data. Often, it is the substantive research problem that begets methodological developments and not the converse. For example, dyadic data analysis was borne out of studying interpersonal phenomena (e.g., Kenny, Kashy, & Cook, 2020), integrative data analysis (IDA) was devised to pool multiple data sets together to study developmental psychopathology over the life course (e.g., Curran & Hussong 2009), and moderated nonlinear factor analysis was developed to take into account measurement invariance when harmonizing responses to distinct but related measures across multiple data sets (Bauer & Hussong, 2009). As these examples illustrate, only by being engaged in substantive research can quantitative psychologists become aware of the methodological challenges that must be overcome to meaningfully advance the science. Re-aligning research practices, values, and incentives in these and other ways is necessary to support increased engagement in impactful methodological work.
This tension between method and methodology is not new, and the preference for refining tools is longstanding. It is not that developing methods should be devalued, but that developing methodology needs new emphasis. Imperfect solutions to important new scientific problems in psychology are worthy of recognition, funding, publication and citation, and have the potential to drive whole new areas of research. We must thus align training programs, faculty positions, and incentives for research that advances methodology. We must fully embrace Tukey’s (1962, p.65) call to quantitative methodologists: “That remains to us, to our willingness to take up the rocky road of real problems in preference to the smooth road of unreal assumptions, arbitrary criteria, and abstract results without real attachments. Who is for the challenge?”
References
Aiken, L. S. (2020). Not the what of quantitative training but the who. In J. L. Rodgers (Ed.,) Teaching statistics and quantitative methods in the 21st century (pp. 55—69). Routledge.
Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training in statistics, measurement, and methodology in psychology: Replication and extension of Aiken, West, Sechrest, and Reno’s (1990) survey of PhD programs in North America. American Psychologist, 63(1), 32–50. https://doi.org/10.1037/0003-066X.63.1.32
Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603–617. https://doi.org/10.1348/000712608X377117
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173–1182. https://doi.org/10.1037/0022-3514.51.6.1173
Bauer, D. J., & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101–125. https://doi.org/10.1037/a0015583
Beaujean, A. A. (2021, October). Message from the new president. The Score, the Division 5 (Quantitative and Qualitative Methods) Newsletter of the American Psychological Association. https://www.apadivisions.org/division-5/publications/score/2021/10/message-president
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/bf02310555
Cronbach, L. J., & Shavelson, R. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64(3), 391–418. https://doi.org/10.1177/0013164404266386
Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100. https://doi.org/10.1037/a0015914
Flake, J. K., Davidson, I. J., Wong, O., & Pek, J. (2022). Construct validity and the validity of replication studies: A systematic review. American Psychologist, 77(4), 576–588. https://doi.org/10.1037/amp0001006
Hayes, A. F. (2022). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach (2nd ed.). Guilford Press.
Kenny, D. A., Kashy, D. A., & Cook, W. L. (2020). Dyadic data analysis. Guilford Publications.
King, G. (1986). How not to lie with statistics: Avoiding common mistakes in quantitative political science. American Journal of Political Science, 30(3), 666–687. https://doi.org/10.2307/2111095
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Information Age Publishing.
Pek, J., & Bauer, D. J. (2023). How can we move advanced methodology into practice more effectively? Policy Insights from the Behavioral and Brian Sciences, 10(1), 3–10. https://doi.org/10.1177/23727322221144649
Pek, J., & Flora, D. B. (2018). Reporting effect sizes in original psychological research: A discussion and tutorial. Psychological Methods, 23(2), 208–225. https://doi.org/10.1037/met0000126
Pek, J., Losardo, D., & Bauer, D. J. (2011). Confidence intervals for a semiparametric approach to modeling nonlinear relations among latent variables. Structural Equation Modeling: A Multidisciplinary Journal, 18(4), 537–553. https://doi.org/10.1080/10705511.2011.607072
Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior research methods, instruments, & computers, 36, 717-731. https://doi.org/10.3758/BF03206553
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
Thissen, D. (2001). Psychometric engineering as art. Psychometrika, 66, 473–485. https://doi.org/10.1007/BF02296190
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–286. https://doi.org/10.1037/h0070288
Tukey, J. W. (1962). The future of data analysis. Annals of Mathematical Statistics, 33, 1–67. doi:10.1214/aoms/1177704711
Tukey, J. W. (1969). Analyzing data: Sanctification or detective work? American Psychologist, 24(2), 83–91. https://doi.org/10.1037/h0027108
Wijsen, L. D., Borsboom, D., & Alexandrova, A. (2022). Values in psychometrics. Perspectives on Psychological Science, 17(3), 788–804. https://doi.org/10.1177/17456916211014183