Scientific inquiry is an uncertain endeavor and identifying a signal, ascertaining its replicability and generalizability are challenging. Ongoing discussions on how to improve the dependability of psychological research reflects the difficulty in working with uncertainty (e.g., Fabrigar et al., 2020; Pashler & Wagenmakers, 2012; Shrout & Rodgers, 2018). Results from any data analytic endeavor encompass uncertainty such as sampling error, measurement error, and model error, among others, pointing to two issues which motivate my research. The first issue focuses on considering, quantifying, and potentially addressing different sources of uncertainty. The second issue centers on moving developments about uncertainty quantification into practice by bridging the gap between methodological developments and their use by substantive (non-methods) researchers.
There is much interest in the role of power analysis in improving the conduct and outcomes of scientific research (e.g., Giofrè et al., 2017; Giofrè et al., 2022), and my work examines the role of uncertainty and moving methods into practice in the context of power analysis. I begin by reviewing concepts about power analysis, highlighting how different sources of uncertainty influences calculated power and determined sample sizes (determined 𝑁). Finally, I describe implications of considering uncertainty in power analysis in designing and evaluating substantive research. In general, (a) there is much more uncertainty inherent in power analysis than researchers realize, (b) despite such uncertainty power analysis continues to be useful in the pre-study phase of design, and (c) power analysis is irrelevant in the post-study phase of research.
Power is the frequentist probability that a statistical test (procedure) correctly rejects the null hypothesis (𝐻0) over repeated samples when there is an effect (i.e., 𝐻1 is true). Thus, power is a property of a test (procedure) when it is applied to an infinite number of samples. The calculated value of power (or determined 𝑁) is a theoretical abstraction that quantifies “what if” scenarios that are absent of empirical data (Judd et al., 2017, p. 68). Thus, values obtained using power analysis (e.g., determined 𝑁) must be treated as inexact, approximate, and uncertain (Kraemer & Blasey, 2015).
Let us express power as a function of its inputs: (1 − 𝛽) = 𝑓(𝛼, 𝑁, 𝜽). We denote power by (1 − 𝛽), which is the mathematical complement of the Type II error rate, 𝛽. Then, 𝛼 represents the Type I error rate, 𝑁 is the sample size of the studies, and 𝜽 is a vector of model (population) parameters that contains the effect size of interest. The function 𝑓(⋅) represents the model in which the test is nested within. For example, an 𝐹-test for an interaction is nested within an ANOVA model. Highlighting these inputs required in power analysis points to different types of uncertainties that can influence the outputs of power analysis and their use in substantive research.
Uncertainties in power analysis
There are several different types of uncertainties that influence power analysis. We focus on sampling variability, uncertainty about the population effect size, random population effects, and model error.
Sampling variability. Sampling variability can affect power calculations in two ways. First, holding all other aspects in power analysis constant, increasing 𝑁 increases power by reducing sampling variability. Second, because the population effect size of interest is unknown, researchers often make use of effect size values estimated from data as input for power analysis (Hoisington-Shaw et al., in preparation). These estimates come with estimate imprecision (i.e., uncertainty due to sampling; see Perugini et al., 2014; McShane & Böckenholt, 2016) and publication bias (Anderson, et al., 2017; Taylor & Muller, 1996).
In general, incorporating estimate imprecision into power analysis results in much uncertainty about calculated power while holding 𝑁 fixed (Yuan & Maxwell, 2005). This uncertainty also manifests in determined 𝑁 when holding power fixed.
Consider estimating a population Cohen’s 𝛿 = 0.8 using 𝑁 = 198. In a simulation, using these estimated Cohen’s 𝑑, power for a two-sided 𝑡-test from a two-condition within-subjects design with 𝑁 = 198 ranged from .05 to 1.00, with 95% of power values falling between .57 and .94 (Pek et al., in press). Alternatively, power analyses using d estimates from previous studies of size 𝑁 = 198 with population 𝛿 = 0.8 had determined sample sizes ranging from 𝑁 = 79 to 1,099. When a more precise d estimate from a meta-analysis is used as input in power analysis, uncertainty in calculated power or determined 𝑁 continues to abound (McShane et al., 2020). This is because between-study variability adds a different source of uncertainty while sampling variability is reduced with larger combined 𝑁 (see also McShane & Böckenholt, 2014). Furthermore, correcting for publication bias in a meta-analytic effect size input tends to increase uncertainty about calculated power (McShane et al., 2020; and by analogy, determined 𝑁). Taken together, after incorporating estimate precision in power analysis, the high variability in calculated power (or determined 𝑁) should limit confidence in power-based choices when planning future studies.
Unknown population effect size. Instead of estimating the unknown population effect size and incorporating estimate imprecision into power analysis, a Bayesian approach that quantifies (subjective) uncertainty could be an alternative workaround. Here, a design prior (distribution) that represents the extent of uncertainty a researcher has about the unknown effect size is applied to (classical) power calculations to determine 𝑁. Because design priors take into account all the information the researcher has about the effect size including risks of implementing the study, resources, and expected outcomes, the prior can be highly informative unlike diffuse analysis priors (O’Hagan et al., 2005). Despite using informative priors, Pek and Park (2019) showed that there remains much uncertainty in calculated power under this Bayesian-classical hybrid approach. Thus, like incorporating estimate imprecision in power analysis, incorporating subjective uncertainty due to not knowing what the population effect size also results in power values so uncertain to be of little use for planning future studies.
Random population effect size. Our consideration of power analysis thus far assumed that the unknown effect size was a fixed and single unknown value. However, large scale replication efforts observe much heterogeneity between studies (Klein et al., 2014; Klein et al., 2018; see also Hedges & Schauer, 2018). The variability in estimated effect sizes between studies reflect both sampling variability and population effect size heterogeneity (also termed a random population effect size). While sampling variability is a form of epistemic uncertainty that can be reduced with more information (e.g., increasing 𝑁, improved measurement), heterogeneity at the level of the population is a form of aleatory uncertainty that cannot be reduced (e.g., collecting more studies or increasing study 𝑁 does not reduce between-study variability). In general, the larger the between-study heterogeneity, the more uncertain power calculations are, requiring much larger 𝑁 (McShane and Bökenholt, 2014). In a simulation, Pek et al. (in press) demonstrate that having a random population effect size causes some values of its estimates to take on a sign in the opposite direction of the population value, because sampling variability and heterogeneity compound, making calculated power values meaningless.
In sum, when considering uncertainty due to sampling and heterogeneity in the population effect size, the downstream variability in calculated power (and determined 𝑁) is so high that these values provide little guidance in planning future studies. By considering different sources of uncertainty, we are relaxing the assumptions underlying classical power analysis (cf. Cohen, 1988). These assumptions include a known population effect size (cf. sampling error or subjective uncertainty of not knowing the population value) and a fixed population effect size (cf., a heterogeneous population effect). Recall that power is a function of inputs: 𝑓(𝛼, 𝑁, 𝜽). With sampling error, we consider 𝜽⋀ (estimates) in place of 𝜽 (population parameters). With a heterogenous population effect, we change the form of the model 𝑓(⋅) from fixed effects to random effects. Below, we further consider model error in which it is explicitly acknowledged that models just cannot fully represent complexities of the real word and provide, at best, a mere approximation (MacCalllum, 2003). Acknowledging model error emphasizes the “what if” nature of power calculations (and determined 𝑁) and the importance of selecting a model with inputs that well approximate the phenomena under study.
Model error. Researchers typically develop theories as a conceptual representation of phenomena. These theories are formalized as statistical models that are fit to data from a population. Model error can be conceptualized as the distance between the population from which observations are drawn and the model that represents the theory that is hypothesized (Wu & Browne, 2015). Cudeck and Henly (1991) term this distance discrepancy of (model) approximation, which is consistent with Meehl’s (1990) notion that a theory (and its formalization as a statistical model) is incomplete relative to complexities in the real world. Thus, it should be explicitly recognized that power analysis is conducted with fallible and incomplete models such that its outputs (i.e., power values, determined 𝑁) are mere approximate “what if” scenarios that continue to be useful for designing studies (Kraemer & Blasey, 2015; Judd et al., 2017). Although model error has not been systematically examined in the context of power analysis, it would be reasonable to assume that when model error is considered, calculated power values and determined 𝑁 would exhibit comparable levels of uncertainty as when sampling error, uncertainty about the unknown effect size, and population heterogeneity are considered. Given such uncertainty, how should researchers use power?
Implications for practice
Our review highlights power as a pre-study concept that is useful for designing studies. Power, however, should not be used to evaluate completed studies because power is a property of a (pre-study) procedure applied to future studies and not of (post-study) data (see also Pek et al., 2022). The pre-study Type I error rate, 𝛼, is irrelevant relative to the 𝑝-value when study results are interpreted. By analogy, the pre-study concept of power, (1 − 𝛽), is irrelevant relative to the statistical information communicated by confidence intervals post-study.
Power analysis is central to designing a study akin to budgeting for a vacation. Planning improves outcomes of the study and the vacation. When the study is implemented or when the vacation is experienced, power analysis or budgeting is predictive of but does not guarantee positive results. Just as budgets can be underspent or exceeded during the vacation, uncertain events during a study can cause divergence between the planned vs. implemented study. Even though calculated values of power and determined 𝑁 are highly uncertain, engaging in power analysis is essential to motivating “statistical thinking” (Chance, 2002). Statistical thinking involves considering different designs, manipulations, measures, sampling frames, developing a data analysis plan, and providing reasons for study elements. These elements strengthen the validity of study findings. Power-based decisions in study design should thus acknowledge different sources of uncertainty, playing a less deterministic role in sample size justifications. Sample sizes are better justified in the larger context of the population, accessibility to samples, and resource constraints in conjunction with “what if” determined 𝑁.
Researchers rightly presume that conducting power analysis is a hallmark of high-quality research (e.g., Funder et al., 2013) because power analysis indicates that the study is well-placed within a context of prior research and theory (Wilkinson & the Task Force for Statistical Inference, 1999) and current resources. However, it is a gross overextension to treat power as important in evaluating results of completed studies for several reasons. First, power is a pre-study concept that refers only to future studies. Applying power to a completed study is “analogous to giving odds on a horse race after seeing the outcome” (Greenland, 2012, p. 364). Because power is a probability over random (hypothetical) data, applying it to fixed (observed) data is a fatal ontological error (Goodman & Berlin, 1994). Second, even if one ignores this ontological issue, values of power cannot be treated as exact and free of error given their highly approximate nature once realistic sources of uncertainty are incorporated into power calculations. Finally, applications of power in the context of making statements of published research rely on implicit (fallible) models about the nature of research findings. These applications include linking power to the false finding rate (FFR; Ioannidis, 2005; Pashler & Harris, 2012), the probability of a statistically significant finding being a Type I error, in which competing alternative perspectives (models) for considering power in relation to evaluating results have been proposed (see Wegener et al., 2022). Other applications involve “statistical forensics,” in which statistics are used to discover anomalies or detect questionable research practices (e.g., Brunner & Schimmack, 2020; Francis, 2013; Ioannidis & Trinkalinos, 2007; see also Pek et al., 2022). In these applications, the models about the nature of research findings are treated as exactly correct – a brazen and implausible assumption. Thus, statements derived from applying power to observed results should be also treated as approximate “what if” scenarios.
In sum, power calculations are pertinent only to the pre-study phase of research. When different sources of uncertainty are considered in power analysis, calculated power values (or determined 𝑁) exhibit much uncertainty reflecting their “what if” nature. However, like budgeting for a vacation, conducting power analysis situates the study in the context of prior research and theory while providing scaffolding to ‘statistically’ think through the design and implementation of the study, which predict but does not guarantee positive results. Power, however, is irrelevant in the post-study phase (Cohen, 1973; Greenland, 2012; Senn, 2002), and results should instead be evaluated according to the broader context of statistical information (e.g., effect sizes, 𝑝-values, and confidence intervals) and other forms of validity (see Shadish et al., 2002).