For admissions officers, college registrars, and college applicants in the United States, storm clouds have been gathering, as two weather fronts threaten to collide. On the one front, diversity and equitable access to higher education represent a compelling state interest. No less a body than the Supreme Court of the United States (SCOTUS) noted that “student body diversity promotes learning outcomes, and “better prepares students for an increasingly diverse workforce and society, and better prepares them as professionals,”” citing benefits in education, business, the military, and as a training ground for the Nation’s future leaders (Grutter v. Bollinger, 2003). On another front, groups like Students for Fair Admissions (2023) maintain that “a student’s race and ethnicity should not be factors that either harm or help that student to gain admission to a competitive university.” Mindful of constitutional protection of individual rights, SCOTUS has allowed affirmative action processes but only within tight limits, defined to include the necessity to withstand strict scrutiny, be narrowly tailored, encompass multiple elements of diversity, consider available alternatives, occur in good faith, and be time-limited (Regents of University of California v. Bakke, 1978; Gratz v. Bollinger, 2003; Grutter v. Bollinger, 2003; Fisher v. University of Texas at Austin, 2013, 2016). Commenting in 2003, exactly 25 years after the first SCOTUS affirmative action ruling in Bakke, Supreme Court Justice Sandra Day O’Connor stated, “the court expects that 25 years from now, the use of racial preferences will no longer be necessary to further the affirmative action interest approved today.” Her words never held any practical force, and she would later walk them back, stating “that may have been a misjudgment,” (Thomas, 2019) but even so her words reflected discomfiture with the existing conditions.
With arguments presented before SCOTUS by the plaintiff Students for Fair Admissions and defendants Harvard University and University of North Carolina, and with publication of judgment anticipated later this spring, clouds do indeed seem to gathering (Strauss, 2023). The crux of the initial lawsuit against the universities was their focus on personal metrics to the detriment of quantitative academic measures like test scores, grade point average (GPA), and academic honors. Had standardized test scores and grade point average been used exclusively for admissions decisions, fewer Asian American applicants would have been denied entry. They are nonplussed by dilution of standardized tests by seemingly less objective mission-aligned holistic metrics, and prefer reliance on traditional standardized tests, despite their well-documented moderate to large group differences (Davis et al., 2013). They point to authoritative review largely confirming these tests to be unbiased (Sackett et al., 2013). Two of the strongest pieces of that evidence are that the SAT does not underpredict future performance by members of less advantaged groups, and that the lower proportion of disadvantaged groups selected to colleges is strongly influenced by the lower applicant proportions from those groups.
Both these arguments can be reasonably challenged. The human resources literature clearly demonstrates that perceptions of procedural unfairness have a deleterious impact on likelihood of members of disadvantaged groups to apply (Schmit & Ryan, 1997). As for predicting future performance of members of disadvantaged groups, MCAT does not under-predict their successful graduation at four years and national licensure success on first attempt but does under-predict for their ultimate success at five years and by second attempt, respectively (Davis et al., 2013). Despite coming from less privileged environmental and educational backgrounds, they ultimately have the same success rates (Davis et al., 2013). They have the equal ability to succeed, merely needing time to close the achievement gap. Should SCOTUS rule as anticipated, the two storm fronts of diversity and traditional test dependence threaten to converge. Of three priorities – diversity, legality, and use of traditional standardized tests – admissions officers and college registrars can choose two, but not all three. Seeing the oncoming collision, they are increasingly choosing diversity and legality, and making applications test-optional (FairTest, 2023), releasing themselves from those tests’ dependence on achievement and moderate to large group differences.
This Hobson’s choice is, however, artificial. The authors of this paper have collaborated with others to create several tests with small or negligible group differences. Successful creation of group-neutral tests is dependent upon avoidance of construct-irrelevant contaminations that introduce group differences. Within a conceptual framework of test content, test format, and test delivery, we present an approach to contamination avoidance for those interested in creating tests with small to negligible group differences.
Test Content
In the first decade of the twentieth century, in response to a call to identify French schoolchildren in need of greater educational supports, Alfred Binet and Theodore Simon created tests of intelligence (Binet, 1905). Between 2005 and 2011 test versions, Binet was successful in excluding questions dependent upon educational and environmental factors, but derivatives through Lewis Terman, Arthur Otis and David Wechsler, and through to contemporary tests, reintroduced questions focusing on knowing rather than thinking. We have, as a result, superb tests of achievement, but not of ability. Uncontaminated measurement of ability requires avoidance of content dependent upon educational and environmental factors. Accordingly, test questions on a test of cognitive ability should not require academic knowledge. Verbal and quantitative test questions should be solvable using any language. Content should be carefully reviewed to ensure avoidance of cultural or linguistic bias. Equitable assessment can be achieved if we rely on tests that measure how well a student thinks in a way that is not confounded by what a student knows (Naglieri, Brulles, & Lansdowne, 2022; Naglieri & Bardos, 1997; Wechsler & Naglieri, 2006).
Test Format
In tests like GAMA (Naglieri & Bardos, 1997), Wechsler Nonverbal (Wechsler & Naglieri, 2006), and NNAT 3 (Naglieri, Brulles, & Lansdowne, 2022), non-verbal approaches to both stems and responses sidestep the unintended contributions of verbal skills, resulting in absence of group differences (Selvamenan et al., 2020). Similar findings were seen with Casper AVR (Dore et al., 2009; Dore et al., 2017), a constructed-response situational judgment test adapted to include audiovisual as well as typed responses. Health professional school applicants were required to complete the typed responses to a dozen scenarios, of which eight were presented as video stems and four as written stems. At the conclusion of the test, volunteers self-identified their race/culture and provided audiovisual responses to different, but similarly styled scenarios. Group differences for typed response Casper (moderate and small Cohen’s d for Black and Hispanic/Latino respectively) were smaller than those typically found in standard achievement tests (large and moderate Cohen’s d respectively) (Juster et al., 2019), but dropped to negligible with audiovisual responses, regardless of whether raters scored based upon the audiovisual responses themselves or their auto-transcriptions (Reiter et al., 2021). The negligible group differences with both auto-transcription and audiovisual responses suggested that the reduction of group differences could be readily explained by avoidance of writing/typing skills and could not be readily explained by rater over-compensation for bias, as had been suggested elsewhere (Lievens et al., 2019).
Test Delivery
For many aspects of the Covid-19 era, the pandemic has acted as an accelerant rather than as a precipitant. The world was already slowly moving towards online testing, due to its singular benefit of improved access, as defined by ease and cost of attendance (avoidance of requisite travel to computer on public access test centers) and reduction in cost of test administration and test result provision (Naglieri et al., 2004). These benefits are further driven by enhanced scalability. For those groups disproportionately affected by less advantageous environmental factors, online testing can markedly decrease differences in access. Appropriate attention must be given to issues of test-taker authentication and monitoring; scalability of these processes is critical to safeguard low test cost and hence protect hard-won access. The same is true where constructed-response testing is used. The group difference reductions seen with audiovisual responses could potentially be undercut by either poorly scalable ratings expenditures driving up test administration costs, or by dependence on auto-transcriptions for test-taker groups with multilingual backgrounds. Both concerns, fortunately, are likely to diminish over time given rapidly advancing technologies.
It is critical to realize that avoidance of any one concern described above is insufficient. All must be addressed. The size of the group differences will be driven by the most rate-limiting step. As described above (Juster et al., 2019), typed-response Casper had eight video stems and four typed stems. One might reasonably expect that the group differences would be less for the portion of the test (video stems) which had avoided the need for reading comprehension. Alas, that was not the case, as the group differences were identical for both test portions. Regardless of the stem format, all responses for both portions were typed, dependent upon writing skill.
A further caution should be given, in the event of successful entry of significant numbers of highly able but lower achievement-level trainees, as this will place a marked burden on programs to provide resources to “level up” those students. Such efforts already exist, in small numbers, which strain limited resources (Thomson et al., 2003). As interested parties of post-secondary education, retrospectively we might ask ourselves to what extent our willingness historically to use standardized tests with large group differences was fueled by fear of that burden. Prospectively, we should ask ourselves how we can better scale that levelling up process, so that we can more aggressively embrace emerging standardized tests that carry small or negligible group differences.
References
Binet, A. (1905). New methods for the diagnosis of the intellectual levels of subnormals. L’Annee Psychologique, 12, 191-244.
Davis, D., Dorsey, K., Franks, R.D., Sackett, P.R., Searcy, C.A., & Zhao, X (2013). Do racial and ethnic group differences in performance on the MCAT exam reflect test bias? Academic Medicine, 88(5):593-602. doi: 10.1097/ACM.0b013e318286803a
Dore, K. L., Reiter, H. I., Eva, K. W., Krueger, S., Scriven, E., Siu, E., Hilsden, S., Thomas, J., & Norman, G. R. (2009). Extending the interview to all medical school candidates – computer-based multiple sample evaluation of non-cognitive skills (CMSENS). Academic Medicine, 84, 9-12. doi: 10.1097/ACM.0b013e3181b3705a.
Dore, K. L., Reiter, H. I., Kreuger, S., & Norman, G. R. (2017). CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores. Advances in health sciences education: theory and practice, 22(2), 327–336. https://doi.org/10.1007/s10459-016-9739-9opens in new window
FairTest: The National Center for Fair & Open Testing (2023). FairTest.org. https://Fairtest.org/test.optional.list/opens in new window
Fisher v. University of Texas at Austin, 579 U.S. (2016). https://supreme.justia.com/cases/federal/us/579/14-981/opens in new window
Fisher v. University of Texas at Austin, 579 U.S. (2013). https://supreme.justia.com/cases/federal/us/570/297/opens in new window
Gratz v. Bollinger, 539 U.S. 244 (2003). https://supreme.justia.com/cases/federal/us/539/244/opens in new window
Grutter v. Bollinger, 539 U.S. 306 (2003). https://supreme.justia.com/cases/federal/us/539/306/opens in new window
Juster, F. R., Baum, R. C., Zou, C., Risucci, D., Ly, A., Reiter, H., Miller, D. D., & Dore, K. L. (2019). Addressing the Diversity-Validity Dilemma Using Situational Judgment Tests. Academic medicine: journal of the Association of American Medical Colleges, 94(8), 1197–1203. https://doi.org/10.1097/ACM.0000000000002769opens in new window
Lievens, F., Sackett, P. R., Dahlke, J. A., Oostrom, J. K., & De Soete, B. (2019). Constructed response formats and their effects on minority-majority differences and validity. The Journal of applied psychology, 104(5), 715–726. https://doi.org/10.1037/apl0000367opens in new window
Naglieri, J.A. (2015) Naglieri Nonverbal Ability Test – Third Edition (NNAT 3). Pearson, London
Naglieri, J. A., Brulles, D. M., & Lansdowne, K. (2022). Naglieri General Ability Tests: Technical Manual. Multi Health Systems. Toronto.
Naglieri, J. A., & Bardos, A. N. (1997). General Ability Measure for Adults (GAMA). Pearson. London.
Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., & Velasquez, R. (2004). Psychological testing on the internet: new problems, old issues. American Psychologist, 59, 150-62. doi: 10.1037/0003-066X_59_3_150
Reiter, H., Hussin, A., Lawoyin, K., Henderson, A., Derby, J., & Dore, K. L. (2021, August 27-31). For constructed-response situational judgement tests, does response format moderate group differences? [Conference Presentation]. AMEE 2021 Virtual Conference, Dundee, United Kingdom.
Regents of University of California v. Bakke, 438 U.S. 265 (1978). https://supreme.justia.com/cases/federal/us/438/265/opens in new window
Sackett, P.R., Kuncel, N.R., Beatty, A.S., Rigdon, J.L., Shen, W., & Kiger, T.B. (2012). The Role of Socioeconomic Status in SAT-Grade Relationships and in College Admissions Decisions. Psychological Science, 23, doi: 10.1177/0956797612438732
Schmit, M.J., & Ryan, A.M. (1997). Applicant withdrawal: the role of test-taking attitudes and racial differences. Personnel Psychology, 50, 855-876.
Selvamenan, M., Paolozza, A., Solomon, J., Naglieri, J. A., & Schmidt, M. T. (2020). Race, ethnic, gender, and parental education level differences on verbal, nonverbal, quantitative Naglieri General Ability Tests: Achieving Equity. Submitted for publication.
Strauss, V. (2023, January). The Element of Suspense in Supreme Court’s 2023 Affirmative Action Ruling. Washington Post. https://www.washingtonpost.com/education/2023/01/16/suspense-supreme-court-affirmative-action/opens in new window
Students for fair admissions (2023, March). Help us eliminate race and ethnicity from college admissions. https://Studentsforfairadmissions.orgopens in new window.
Thomas, E. (2019, March). Why Sandra Day O’Connor Saved Affirmative Action. The Atlantic. https://www.theatlantic.com/ideas/archive/2019/03/how-sandra-day-oconnor-saved-affirmative-action/584215/opens in new window
Thomson, W. A., Ferry, P. G., King, J. E., Martinez-Wedig, C., & Michael, L. H. (2003). Increasing access to medical education for students from medically underserved communities: one program's success. Academic medicine: Journal of the Association of American Medical Colleges, 78(5), 454–459. https://doi.org/10.1097/00001888-200305000-00006opens in new window
Wechsler, D., & Naglieri, J. A. (2006). Wechsler Nonverbal Scale of Ability. San Antonio, TX: Pearson.