Factor analysis has a notably long history in psychology. It begins in the early 19th century when Charles Spearman proposed to decompose observed score into a true score plus error, leading to a two-factor theory (Spearman, 1904a). Then it culminates in the 1930s, when L.L. Thurstone derived common factor analysis (Thurstone, 1931) and undertook a massive Primary Mental Ability study. In that study, 240 college students volunteered to take a 56-test cognitive battery. 12 factors were extracted from the item correlation matrix, among which 7 were considered well-defined scientific constructs of ability (Thurstone, 1935, 1936a, 1936b, 1938). It is also from the same analysis that Thurstone recognized the arbitrariness of any factor locations. Factor rotation, and exploratory factor analysis, soon came to flourish.
Thurstone is unarguably the most important single contributor to the development of factor analysis after Spearman, and uncoincidentally, his 1925 legacy paper entitled “A method of scaling psychological and educational tests” (Thurstone, 1925) probably laid out the first intimations of item response theory (IRT). In that paper, he provided a solution to put the items of the Binet and Simon test of children’s mental development on an age-graded scale. His analysis shares basic features of IRT, such as the monotonically increasing S-shaped item characteristic curve as well as the crucial concept of scaling, among others. A profound difference, though, is that instead of the observed chronological age which governs children’s probability of success on an item, IRT assumes that the probability of correct response is controlled by a latent variable. Indeed, latent variable is a central notation of IRT, and it refers to the hypothetical construct the test intends to measure, such as ability, proficiency, attitude, etc. Since 1925, several factors play in concert to spur the formal creation of IRT. From application perspective, the success of group testing of military recruits based on Army Alpha Test greatly popularizes standardized paper-and-pencil test, followed by the momentum gathered through the inclusion of multiple-choice items (for objective scoring) in College Entrance Exam and Scholastic Aptitude Test (Bock, 1997). Meanwhile, statistical methodology has advanced rapidly to offer necessary infrastructure for IRT, including for instance, the local independence assumption formalized by Lazarsfeld (1950), the maximum likelihood estimation, Fisher information, as well as logistic model by Lord (1980). Since then, the era of IRT began.
The fields of factor analysis and IRT continue to develop in parallel, and involve the same notion of latent variables. This parallel development is understandable because factor analysis focuses on uncovering meaningful causal constructs to account for the correlations between variables, and IRT from its inception is to model item responses as a function of a pre-specified latent variable. Takane and de Leeuw (1987) formally established the equivalency between IRT and confirmatory factor analysis. Despite of the similarity between IRT and factor analysis, there is one interesting note to make. In factor analysis, the number of factors is almost never restricted to one. From early trans-Atlantic debate between Spearman and Thorndike regarding a single general factor of cognitive ability vs. multi-factors of intellect (Thorndike, Lay, & Dean, 1909), to a later conceptualization of hierarchical factor structure of human abilities (Humphreys, 1962), factor analysis seems to handle multiple factors skillfully. In contrast, the unidimensionality assumption (i.e., a single latent trait) has dominated IRT for decades. One central reason is the computational challenge involved in the marginal maximum likelihood estimation (MMLE), the status quo estimation method for IRT. This challenge is also known as the curse of dimensionality.
However, the unidimensionality assumption may not always hold. It is common for educational and psychological tests to be inherently multidimensional, meaning that they measure more than one trait, skill, or dimension. Multidimensional IRT (MIRT) naturally relaxes this assumption and allows for multiple correlated latent traits. MIRT can not only help better interpret the interactions between item characteristics and persons’ behaviors, but also provide more diagnostic information, such as a whole profile of an individual on the measured traits. It is in recent decades that MIRT starts to gain its momentum, mostly resulting from the advent of more powerful and flexible estimation methods, such as the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm (e.g., Cai, 2010), or the Bayesian Markov chain Monte Carlo (MCMC) algorithm (Patz & Junker, 1999). More importantly, with the availability of the dedicated software packages such as IRTPRO, flexMIRT, and ‘mirt’ R package, MIRT has become a methodology that is now within the reach of psychologists.
My first foray into MIRT research, in 2009, is an application of MIRT in computerized adaptive testing (CAT), sometimes called multidimensional CAT (MD-CAT). CAT is considered one prominent application of IRT: it allows substantial reductions in testing time while maintaining a predetermined level of measurement precision. When a test intends to measure a repertoire of correlated constructs, such as science test measuring life, physical, and chemistry science, MD-CAT can select items to balance content coverage and yield increased measurement precision than administering several unidimensional CATs (Segall, 1996). The argument of increased precision has been supported by several simulation studies imitating applications in education and health measurements (e.g., Paap, Born, & Braeken, 2019; Paap, Kroeze, Terwee, et al., 2017). Because MD-CAT provides more information than unidimensional CAT, the adaptive programming is far more complex and can be unstable. My research on MD-CAT has focused on proposing statistically optimal and computationally efficient item selection algorithms (e.g., Wang & Chang, 2011; Wang, Chang, & Boughton, 2011), interim latent trait update method (Wang, 2015), as well as stopping criteria (Wang, Chang, & Boughton, 2013; Wang, Weiss, & Shang, 2018). It has been a quite active research topic from which many of my earlier publications are generated, until very recently when the first MD-CAT becomes live to measure physical functioning in post-acute care hospital settings (Cheville, Wang, Weiss, et al., 2021). By then, many of the important technical and methodological barriers of MD-CAT will have been successfully overcome.
Fast forward, contemporary extensions of IRT include multilevel, multiple-group, and mixture IRT models besides MIRT (Cai, Choi, Hansen, & Harrell, 2016). Therefore, IRT can be used in a far wider array of applications. For instance, multilevel IRT models can routinely handle nested data structures, such as students nested within classrooms or repeated measures nested within person. Some of my more recent work has been devoted to developing IRT models to handle longitudinal data (Wang, Kohli, & Henn, 2016; Wang & Nydick, 2020), as well as evaluating intra-individual change (Wang & Weiss, 2018; Wang, Weiss, & Suen, 2020). Mixture IRT model, broadly conceived, includes either a mixture of discrete and continuous latent variables (Hong, Wang, Lim, & Douglas, 2015) or a mixture of different types of underlying behaviors that govern item responses. Taking the latter scenario as an example, I have developed mixture IRT models to differentiate problem-solving behavior from aberrant behavior such as rapid guessing or cheating (Wang & Xu, 2015; Wang, Xu, & Shang, 2018; Wang, Xu, Shang, & Kuncel, 2018).
Now, as many assessments move online, especially amid social distancing, collecting data on large sample and many variables become much less costly than in-person surveys. Moreover, the specter of “log everything” looms large in the pursuit of big data. Even the sophisticated MH-RM or MCMC algorithms cannot seem to keep on par with the growing data analysis demand. With the slow, yet continuous penetration of modern machine learning methods, psychometricians and quantitative psychologists start to embrace new techniques such as variational methods or deep learning methods. Some recent studies, including my own, show that these advanced methods can handle MIRT with large number of factors (e.g., Cho, Wang, Zhang, & Xu, 2020; Urban & Bauer, 2020), large sample size, and long test length quite efficiently. Although it seems tempting to try these methods on big data and let the algorithm do its job, users of MIRT should not get lost in the weeds of data. Instead, psychological theory should still play a central role to govern the analysis and guide the interpretation of the results.
References
Bock, D. (1997). A brief history of Item Response Theory. Educational Measurement: Issues and Practice, 16, 21-33.
Cai, L. (2010). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307-335.
Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of statistical software, 48(6), 1-29.
Cheville, A., Wang, C., Weiss, D., Yost, K., Teresi, J., Ramirez, M., Ocepek-Wilson, K., Ni, P., Marfeo, E., Keeney, T., & Jette, A. (2021). Improving the Delivery of Function-Directed Care During Acute Hospitalizations: Methods to Develop and Validate the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT). Archives of Rehabilitation Research and Clinical Translation.
Cho, A. E., Wang, C., Zhang, X., & Xu, G. (2020). Gaussian variational estimation for multidimensional item response theory. British Journal of Mathematical and Statistical Psychology.
Hong, H., Wang, C., Lim, Y. S., & Douglas, J. (2015). Efficient models for cognitive diagnosis with continuous and mixed-type latent variables. Applied psychological measurement, 39(1), 31-43.
Humphreys, L. G. (1962). The organization of human abilities. American Psychologist, 17, 475-482.
Lazarsfeld, P. F. (1950). The logical and mathematical foundation of latent structure analysis. In S. A. Stouffer, L. Guttman, E. A. Suchman, P. F. Lazarsfeld, S. A. Star, & J. A. Clausen, Measurement and prediction (pp. 362-412). Princeton, NJ: Princeton University Press.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
Paap, M. C., Born, S., & Braeken, J. (2019). Measurement efficiency for fixed-precision multidimensional computerized adaptive tests: Comparing health measurement and educational testing using example banks. Applied psychological measurement, 43(1), 68-83.
Paap, M. C., Kroeze, K. A., Terwee, C. B., van der Palen, J., & Veldkamp, B. P. (2017). Item usage in a multidimensional computerized adaptive test (MCAT) measuring health-related quality of life. Quality of Life Research, 26(11), 2909-2918.
Patz, R. J., & Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24(4), 342-366.
Segall, D. O. (1996). Multidimensional adaptive testing. Psychometrika, 61(2), 331-354.
Spearman, C. (1904a). The proof and measurement of the association between two things. American Journal of Psychology, 15, 72-101.
Takane, Y., & De Leeuw, J. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52(3), 393-408.
Thorndike, E. L., Lay, W. & Dean, P. R. (1909). The relation of accuracy in sensory discrimination to general intelligence, American Journal of Psychology, 20, 364-369.
Thurstone, L. L. (1925). A method of scaling psychological and educational tests. Journal of Educational Psychology, 16, 433-451.
Thurstone, L. L. (1931). Multiple factor analysis. Psychological Review, 38, 406-427.
Thurstone, L. L. (1935). The vectors of mind, University of Chicago Press, Chicago.
Thurstone, L. L. (1936a). A new conception of intelligence and a new method of measuring primary abilities. Educational Record, 17, 124-138.
Thurstone, L. L. (1936b). A new conception of intelligence. Educational Record, 17, 441-450.
Thurstone, L. L. (1938). Primary mental abilities. Psychometric Monographs, No 1.
Urban, C. J., & Bauer, D. J. (2020). A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis. arXiv preprint arXiv:2001.07859.
Wang, C., & Chang, H.-H. (2011). Item selection in multidimensional computerized adaptive testing—Gaining information from different angles. Psychometrika, 76(3), 363-384.
Wang, C., Chang, H.-H., & Boughton, K. A. (2011). Kullback–Leibler information and its applications in multi-dimensional adaptive testing. Psychometrika, 76(1), 13-39.
Wang, C., Chang, H.-H., & Boughton, K. A. (2013). Deriving stopping rules for multidimensional computerized adaptive testing. Applied psychological measurement, 37(2), 99-122.
Wang, C., Kohli, N., & Henn, L. (2016). A second-order longitudinal model for binary outcomes: Item response theory versus structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(3), 455-465.
Wang, C., & Nydick, S. W. (2020). On longitudinal item response theory models: A didactic. Journal of Educational and Behavioral Statistics, 1076998619882026.
Wang, C., & Weiss, D. J. (2018). Multivariate hypothesis testing methods for evaluating significant individual change. Applied psychological measurement, 42(3), 221-239.
Wang, C., Weiss, D. J., & Shang, Z. (2018). Variable-Length Stopping Rules for Multidimensional Computerized Adaptive Testing. Psychometrika, 1-23.
Wang, C., Weiss, D. J., & Suen, K. Y. (2020). Hypothesis Testing Methods for Multivariate Multi-Occasion Intra-Individual Change. Multivariate behavioral research, 1-17.
Wang, C., & Xu, G. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456-477.
Wang, C., Xu, G., & Shang, Z. (2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83(1), 223-254.
Wang, C., Xu, G., Shang, Z., & Kuncel, N. (2018). Detecting Aberrant Behavior and Item Preknowledge: A Comparison of Mixture Modeling Method and Residual Method. Journal of Educational and Behavioral Statistics, 1076998618767123.