The last decade has seen influx of research on integrating machine learning (i.e., artificial intelligence, data mining, statistical learning) with traditional psychometric methods. One of the difficulties in this is defining what methods qualify as machine learning, and which as traditional statistics or psychometrics. One algorithm that exemplifies this is lasso regression. A form of regularization, lasso regression incorporates a penalty on the regression coefficients, often resulting in a solution that has set multiple coefficients to zero, thus performing variable selection. On the one hand, in lasso regression, the regression coefficients are shrunken towards zero in relation to the linear or logistic regression solution; thus, it is a more conservative approach. However, lasso regression is often applied in an atheoretical manner, often when the number of variables is large (often hundreds), thus necessitating variable selection. Ultimately, the distinction between machine learning and traditional statistics is best personified in the discrepancy between how the methods are applied, considering the level of theoretical input, and whether the goal is closer to prediction or explanation (Yarkoni & Westfall, 2017). The motivation behind the methods we highlight is in extending traditional forms of modeling to new types of data, often larger in either the number of observations, time points or variables (Adjerid & Kelley, 2018). By capitalizing on the central component of psychometrics, namely the modeling of measurement error, these new forms of modeling can better alter machine learning algorithms for the inherent difficulties in psychological data. Our goal in this article is to give a brief overview of the developments in this area, focusing on two statistical algorithms, regularization and trees. We end with a discussion of current limitations and future directions.
Regularization
The application of structural equation modeling (SEM) is typically thought to be based on theory, although in practice, there is often a mixture of theoretical formulation and uncertainty. When theoretical specification fails, leading to poor model fit, it is often necessary to perform model modification. To overcome the limitations of modification indices, heuristic search algorithms were proposed around 20 years ago to efficiently identify an optimal set of models (e.g., Marcoulides & Drezner, 2003; Marcoulides, Drezner, & Schumacker, 1998). More recently, a similar method, termed regularization, has been integrated into SEM (Jacobucci, Grimm, & McArdle, 2016), item response theory (e.g., Sun, Chen, Liu, Ying, & Xin, 2016), mixed effects models (e.g., hierarchical or multilevel; Schelldorfer, Bühlmann, & van de Geer, 2011), exploratory factor analysis (Jennrich, 2006; Scharf & Nestler, 2019), and network models (Epskamp, Rhemtulla, & Borsboom, 2017). The most common purpose in applying regularization in these forms of models is to reduce the complexity of the model by selecting or removing paths or variables out. A second, and related purpose in applying regularization is to overcome problems in estimation, such as non-positive definite matrices through incorporating ridge penalties (e.g., Yuan & Chan, 2016), which can be traced back to its application in regression to tackle similar problems Hoerl & Kennard (1970).
Trees
Identifying heterogeneity in the population has become an increasingly popular form of statistical application in the last 20 years, spurred on by the integration of mixture models with more traditional modeling procedures. The use of mixture models represents a form of modeling more along the lines of machine learning, given the lack of theoretical input representative in most research applications. A similar form of algorithm that explicitly identifies heterogeneity, with less of the parametric assumptions, is decision trees. Decision trees have been paired with structural equation models (Brandmaier, von Oertzen, McArdle, & Lindenberger, 2013), as well as mixed effects models (Abdolell, LeBlanc, Stephens, & Harrison, 2002; Stegmann, Jacobucci, Serang, & Grimm, 2018) to explicitly capture heterogeneity with respect to model parameters. This may be best captured in the context of longitudinal data (see Jacobucci, Grimm, & McArdle, 2017), where growth mixture models identify multiple unobserved classes that demonstrate differing trajectories of change, while trees assign class membership derived from observed covariates. This process has been extended to building large number of trees to increase both prediction and the stability of inferences. With a single outcome, this algorithm is referred to as random forests (Breiman, 2001), and has been extended to modeling a SEM as an outcome (Brandmaier, Prindle, McArdle, & Lindenberger, 2016; see Miller, Lubke, McArtor, and Bergeman (2016) for the application of a similar algorithm for multivariate outcomes).
Limitations and Future Directions
Given that these forms of research are still in their infancy, a great deal of research is needed. Specifically, very little research has examined how model misspecification can influence the resulting conclusions (see Usami, Hayes, & McArdle, 2017 for an example of this type of research). Additionally, more research is needed to identify data characteristics that lend themselves to the incorporation of regularization (e.g., the assumption of sparsity), and scenarios that do not (e.g., see Williams, Rhemtulla, Wysocki, & Rast, 2019). Simply put, regularized estimation in SEM is not a cure all — maximum likelihood estimation works just as well and in most research scenarios, better. Finally, tree models are inherently unstable, leading to uncertainty into whether the tree structure and resulting group assignment is likely to generalize to alternate samples.
Although these new forms of modeling allow for increasing the dimensionality of our models, we need to be clearer about the purpose of these analysis. For instance, one could fit a multiple indicator, multiple causes model with 1,000 predictors and a sample size of 200 by incorporating shrinkage priors (Bayesian form of regularization, see Van Erp, Oberski, & Mulder, 2019) in Bayesian estimation software such as Stan (Gelman, Lee, & Guo, 2015). However, we need to ask ourselves what practical problems do psychological researchers have that lend themselves to this form of modeling? It is important to not lose sight of our goal of improving psychological research, and not get caught up in the novelty of new, shiny algorithms. To push this area of research forward, we need to better identify and engender research applications that have both the questions and data that lend themselves to extremely complex models that cannot be estimated (at all or well) with traditional software.
References
Abdolell, M., LeBlanc, M., Stephens, D., & Harrison, R. V. (2002). Binary partitioning for continuous longitudinal data: categorizing a prognostic variable. Statistics in Medicine, 21, 3395-3409.
Adjerid, I., & Kelley, K. (2018). Big data in psychology: A framework for research advancement. American Psychologist, 73, 899.
Brandmaier, A. M., von Oertzen, T., McArdle, J. J., & Lindenberger, U. (2013). Structural equation model trees. Psychological Methods, 18, 71.
Brandmaier, A. M., Prindle, J. J., McArdle, J. J., & Lindenberger, U. (2016). Theory-guided exploration with structural equation model forests. Psychological Methods, 21, 566.
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32.
Epskamp, S., Rhemtulla, M., & Borsboom, D. (2017). Generalized network psychometrics: Combining network and latent variable models. Psychometrika, 82, 904–927.
Gelman, A., Lee, D., & Guo, J. (2015). Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics, 40, 530-543.
Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Applications to nonorthogonal problems. Technometrics, 12, 69–82.
Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2016). Regularized structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23, 555-566.
Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2017). A comparison of methods for uncovering sample heterogeneity: Structural equation model trees and finite mixture models. Structural Equation Modeling: A Multidisciplinary Journal, 24, 270-282.
Jennrich, R. I. (2006). Rotation to simple loadings using component loss functions: The oblique case. Psychometrika, 71, 173-191.
Marcoulides, G. A., & Drezner, Z. (2003). Model specification searches using ant colony optimization algorithms. Structural Equation Modeling: A Multidisciplinary Journal, 10, 154-164.
Marcoulides, G. A., Drezner, Z., & Schumacker, R. E. (1998). Model specification searches in structural equation modeling using Tabu search. Structural Equation Modeling: A Multidisciplinary Journal, 5, 365-376.
Miller, P. J., Lubke, G. H., McArtor, D. B., & Bergeman, C. S. (2016). Finding structure in data using multivariate tree boosting. Psychological Methods, 21, 583.
Scharf, F., & Nestler, S. (online first). Should Regularization Replace Simple Structure Rotation in Exploratory Factor Analysis? Structural Equation Modeling: A Multidisciplinary Journal, 1-15.
Schelldorfer, J., Bühlmann, P., & van de Geer, S. (2011). Estimation for high-dimensional linear mixed-effects models using l1-penalization. Scandinavian Journal of Statistics, 38, 197- 214.
Stegmann, G., Jacobucci, R., Serang, S., & Grimm, K. J. (2018). Recursive Partitioning with Nonlinear Models of Change. Multivariate Behavioral Research, 53, 559-570.
Sun, J., Chen, Y., Liu, J., Ying, Z., & Xin, T. (2016). Latent variable selection for multidimensional item response theory models via L1 regularization. Psychometrika, 81, 921–939.
Usami, S., Hayes, T., & McArdle, J. (2017). Fitting structural equation model trees and latent growth curve mixture models in longitudinal designs: The influence of model misspecification. Structural Equation Modeling: A Multidisciplinary Journal, 24, 585-598.
Van Erp, S., Oberski, D. L., & Mulder, J. (2019). Shrinkage priors for Bayesian penalized regression. Journal of Mathematical Psychology, 89, 31-50.
Williams, D. R., Rhemtulla, M., Wysocki, A. C., & Rast, P. (online first). On nonregularized estimation of psychological networks. Multivariate behavioral research, 1-23.
Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12, 1100-1122.
Yuan, K. H., & Chan, W. (2016). Structural equation modeling with unknown population distributions: Ridge generalized least squares. Structural Equation Modeling: A Multidisciplinary Journal, 23, 163-179.
