The use of high-stakes educational assessments is increasingly in hot water. In the past year, criticism against standardized testing has coalesced around the issue of test inequality—namely that educational assessments used here in North America and around the world (Miranda, 2020) systematically discriminate against marginalized groups of students, such as Black, Latino, and low income students (Koljatic et al., 2021; Sireci, 2021). In response to this issue the University of California System decided this year to no longer make use of the ACT and SAT for admission purposes (Rio, 2021).
As educational testing professionals, we have a responsibility to address these concerns through quality improvements to educational assessments, making them as fair and impartial as possible, to reduce and attempt to eliminate the test score gap that is consistently seen between privileged and marginalized groups of students (Jencks & Phillips, 1998). While we cannot fully eliminate disparities in test-taking populations that are a result of centuries of systemic inequality (Lucey and Saguil, 2020), we can, and should, make use of statistical and psychometric tools to identify test items that contribute to the problem and remove them. Fortunately, such tools exist, one of them being differential item functioning (DIF) analysis.
Pioneered by the Educational Testing Service in the 1980s (Zieky, 2011), DIF allows researchers to determine whether score differences between groups on a given test item stems from true differences in ability on the construct that the item is intended to measure (e.g. verbal reasoning ability), or construct-irrelevant differences (e.g., cultural, historical, or scientific knowledge on an item intending to measure verbal reasoning). In the case of the latter, the item may be biased against a particular group of students and should be removed.
DIF analysis is a more psychometrically acceptable approach than simply eliminating items which exhibit large mean overall differences between two groups. Instead, DIF compares performance on a test item between a reference (majority) group and a focal (marginalized) group, while controlling for performance on the test as a whole (an estimate of the true difference in the construct-relevant ability) (Holland and Wainer, 1993; Zumbo, 1999). DIF analysis has been successfully used to detect problematic items in large-scale assessments such as the SAT (Sehmitt and Dorans, 1990; Zenisky et al., 2003) and it has been advocated for use in the development and testing of high-stakes assessments (Martinková et al., 2017).
Methods and considerations when implementing DIF
Over the past 40 years, a variety of different methods have been developed to implement DIF analysis. Many of these methods are available in easy-to-use packages for open source software such as R (Y.-J. Choi and Asilkalkan, 2019). These methods range from the relatively straightforward Mantel-Haenszel statistic, a nonparametric test that can be used on dichotomous test items (graded as correct or incorrect), to more complex methods used such as logistic regression and Item Response models, which have been adapted to handle test items with ordinal grading scales (French and Miller, 1996; Zumbo, 1999).
Even after selection of a DIF methodology there are a number of additional decisions around implementation to consider. First, selecting an appropriate sample of the test-taking groups. While there are no hard rules on the minimum sample size needed to run DIF analysis (Scott et al., 2010) it has been observed that unequal sample sizes can lead to reduced statistical power (Kristjansson et al., 2005). For parametric methods such as logistic regression it has been suggested that 200 is the minimum number required for each group (Scott et al., 2010). Second, deciding what statistical threshold must be crossed before an item is flagged as exhibiting DIF. This involves:
- selection of a suitable significance level (alpha) for the test statistic used,
- deciding whether to apply a correction for multiple comparisons, and
- selection of an effect size measure (e.g., R2, ETS Delta) to determine if the magnitude of the DIF found is practically meaningful (Jodoin and Gierl, 2001).
Finally, if DIF is detected, it can be either positive (the item advantages a focal group), negative (advantages the reference group), or mixed (nonuniform DIF where, for example, when comparing high-performing students across the two groups, the reference group is advantaged, while when comparing low-performing students across the two groups, the reference group is disadvantaged). It is a judgement call whether to remove an item if it exhibits positive DIF, but as both negative and positive DIF indicate that an item is measuring construct-irrelevant knowledge, it is considered best practice to remove all items exhibiting a large magnitude of DIF regardless of the direction (Zieky, 2011).
The reader is encouraged to explore the many reviews of existing DIF methodology available which discuss the advantages and disadvantages of various methods and the details of their implementation (Kristjansson et al., 2005; Martinková et al., 2017; Teresi and Fleishman, 2007). In the next section, I’ll be discussing my involvement with a recent application of DIF analysis to a large-scale educational assessment.
DIF analysis of a high-stakes educational assessment
As of March 2021, our research team at Altus Assessments began applying DIF analysis to our situational judgement test Casper. Casper is designed to evaluate the personal and professional characteristics of individuals, primarily applicants to medical education programs. While Casper shows smaller demographic differences as compared to scores for other commonly used admissions criteria, such as MCAT or GPA (Juster et al., 2019), we applied DIF analysis to continue our team’s pursuit of reducing demographic differences on our assessments (Casper Demographic Differences 2019-2020, 2020).
Our team applied DIF analysis to 168 unique Casper test sittings that occurred between 2018–2021, containing over 2,000 test items. Using demographic information provided by our applicants in a voluntary exit survey, we compared the performance on these items along gender (self-identified as male vs. female) and racial lines (White vs. Black or African American/Hispanic, Latino, or Spanish origin/Indigenous/Asian). To enable us to perform DIF analysis at-scale we automated the process, making use of two R packages, lordif (S. W. Choi et al., 2011) and difNLR (Hladká and Martinková, 2020), specifically chosen as they provided ordinal logistic regression functions compatible with the ordinal grading system used by Casper. Based on Jodoin and Gierl’s (2001) effect size criteria, which categorizes the magnitude of DIF into negligible, moderate and large, none of the items we tested exceeded a negligible amount of DIF. However, we flagged any items which exhibited even a negligible amount of DIF in at least two different test sittings, and we are currently performing qualitative review of these test items to look for any obvious signs of demographic bias in question wording. However, not all bias inherent in item content will be readily apparent, and it is therefore recommended that cultural experts be used whenever possible to examine items flagged as showing DIF.
Conclusion
DIF analysis is but one approach that the testing industry should use to minimize demographic differences on high-stakes educational assessments. As Koljatic et al. (2021) put it “rather than simply claim they follow established statistical procedures to screen items for differential item functioning (DIF), testing companies should commit to invest in exploring new ways of testing to effectively reduce performance gaps”. Indeed, Altus Assessments is currently testing an alternative response format for Casper designed to significantly reduce demographic differences (Dore, 2021). The psychometric community has the means to reduce inequities in testing while maintaining high standards of psychometric reliability and validity.
In the face of criticism over standardized testing researchers would be wise to heed another quote, this one from Douglas Adams’ The Hitchhiker’s Guide to the Galaxy: “Don’t Panic”.
Conflict of interest: Potential or perceived conflicts of interests may arise in that the author is a full-time scientist employed by Altus Assessments.
References
Casper Demographic Differences 2019-2020. (2020). Altus Assessments.
Choi, S. W., Gibbons, L. E., and Crane, P. K. (2011). lordif: An R Package for Detecting Differential Item Functioning Using Iterative Hybrid Ordinal Logistic Regression/Item Response Theory and Monte Carlo Simulations. Journal of Statistical Software, 39(8). https://doi.org/10.18637/jss.v039.i08opens in new window
Choi, Y.-J., and Asilkalkan, A. (2019). R Packages for Item Response Theory Analysis: Descriptions and Features. Measurement: Interdisciplinary Research and Perspectives, 17(3), 168–175. https://doi.org/10.1080/15366367.2019.1586404opens in new window
Dore, K. (2021). For Constructed-Response Situational Judgement Tests: Does Response Format Moderate Group Differences? AAMC Learn Serve Lead.
French, A. W., and Miller, T. R. (1996). Logistic Regression and Its Use in Detecting Differential Item Functioning in Polytomous Items. Journal of Educational Measurement, 33(3), 315–332. https://doi.org/10.1111/j.1745-3984.1996.tb00495.xopens in new window
Hladká, A., and Martinková, P. (2020). difNLR: Generalized Logistic Regression Models for DIF and DDF Detection. R Journal, 12(1).
Holland, P. W., and Wainer, H. E. (1993). Differential item functioning. Lawrence Erlbaum.
Jencks, C., and Phillips, M. (1998). The Black-White Test Score Gap: Why It Persists and What Can Be Done. Brookings Institute. https://www.brookings.edu/articles/the-black-white-test-score-gap-why-it-persists-and-what-can-be-done/opens in new window
Jodoin, M. G., and Gierl, M. J. (2001). Evaluating type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329–349.
Juster, F. R., Baum, R. C., Zou, C., Risucci, D., Ly, A., Reiter, H., Miller, D. D., and Dore, K. L. (2019). Addressing the Diversity–Validity Dilemma Using Situational Judgment Tests. Academic Medicine, 94(8), 1197–1203. https://doi.org/10.1097/ACM.0000000000002769opens in new window
Koljatic, M., Silva, M., and Sireci, S. G. (2021). College Admission Tests and Social Responsibility. Educational Measurement: Issues and Practice. https://doi.org/10.1111/emip.12425opens in new window
Kristjansson, E., Aylesworth, R., Mcdowell, I., and Zumbo, B. D. (2005). A Comparison of Four Methods for Detecting Differential Item Functioning in Ordered Response Items. Educational and Psychological Measurement, 65(6), 935–953. https://doi.org/10.1177/0013164405275668opens in new window
Lucey, C. R., and Saguil, A. (2020). The consequences of structural racism on MCAT scores and medical school admissions: The past is prologue. Academic Medicine, 95(3), 351–356.
Martinková, P., Drabinová, A., Liaw, Y.-L., Sanders, E. A., McFarland, J. L., and Price, R. M. (2017). Checking Equity: Why Differential Item Functioning Analysis Should Be a Routine Part of Developing Conceptual Assessments. CBE—Life Sciences Education, 16(2), rm2. https://doi.org/10.1187/cbe.16-10-0307opens in new window
Miranda, N. A. R. (2020, January 6). Chilean university admissions tests hit by fresh protests. Reuters. https://www.reuters.com/article/us-chile-protests-university-idUSKBN1Z522Iopens in new window
Rio, G. M. N. del. (2021, May 15). University of California Will No Longer Consider SAT and ACT Scores. The New York Times. https://www.nytimes.com/2021/05/15/us/SAT-scores-uc-university-of-california.htmlopens in new window
Scott, N. W., Fayers, P. M., Aaronson, N. K., Bottomley, A., de Graeff, A., Groenvold, M., Gundy, C., Koller, M., Petersen, M. A., and Sprangers, M. A. (2010). Differential item functioning (DIF) analyses of health-related quality of life instruments using logistic regression. Health and Quality of Life Outcomes, 8(1), 1–9.
Sehmitt, A. P., and Dorans, N. J. (1990). Differential item functioning for minority examinees on the SAT. Journal of Educational Measurement, 27(1), 67–81.
Sireci, S. G. (2021). NCME Presidential Address 2020: Valuing Educational Measurement. Educational Measurement: Issues and Practice, 40(1), 7–16.
Teresi, J. A., and Fleishman, J. A. (2007). Differential item functioning and health assessment. Quality of Life Research, 16(S1), 33–42. https://doi.org/10.1007/s11136-007-9184-6opens in new window
Zenisky, A. L., Hambleton, R. K., and Robin, F. (2003). Detection of differential item functioning in large-scale state assessments: A study evaluating a two-stage approach. Educational and Psychological Measurement, 63(1), 51–64.
Zieky, M. J. (2011). The origins of procedures for using differential item functioning statistics at Educational Testing Service. In Looking Back (pp. 115–127). Springer.
Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF). Directorate of Human Resources Research and Evaluation, Department of National Defense, Ottawa.