Mental health and substance use screening measures can be instrumental in ensuring that individuals in need of care are referred to the appropriate resources (SAMHSA, 2017). For example, assessment specialists could refer respondents who score 16 or higher on the CES-D (Center of Epidemiological Studies – Depression) scale to a diagnostic interview (Vilagut, Forero, Barbaglia, & Alonso, 2016), or a psychologist might conduct further cognitive assessment on children who score 76 or higher on the AQ-Child (Autism Spectrum Quotient – Children’s Version; Auyeung, Baron-Cohen, Wheelwright, & Allison, 2008). Screening measures are important in health and clinical research because they are typically shorter, cheaper and less invasive than comprehensive assessments.
Before screening measures are used on a mass scale with a common cutoff score, it is imperative to examine whether the screener performs similarly across priority groups. In the case of psychological measures, priority groups are defined by background characteristics that may affect an individual’s responding behavior, such as gender, ethnicity, race, age group, language of origin, among others. For example, Hispanics may be more likely to endorse items related to crying in a depression scale than members from other ethnic groups because crying behavior is more culturally accepted in Hispanic cultures than in other cultures (Azocar, Arean, Miranda, & Muñoz, 2001). Consequently, if two respondents, one from a Hispanic background and another from a different ethnic group, have the exact same level of depression but they interact differently with the crying item, then the Hispanic respondent might have a higher likelihood to be above the cutoff score on the depression scale. In other words, the item response does not only reflect depression, but also the respondent’s background characteristics – which is not the intention of screening measures.
Items are said to exhibit measurement bias when they are prone to differential responding due to a background characteristic, so the presence of items with measurement bias in the screener could differentially affect who gets flagged by the screener. It is hypothesized that a lack of cultural equivalence in screening measures (and generally, in any health or clinical assessment) leads to minority health disparities and a misallocation of resources (Meredith & Teresi, 2006). Therefore, it is important to evaluate if the screening measures used in the field are fair and equitable, so that priority groups are not adversely affected on resource allocation (e.g., referred to a diagnostic interview, further assessment, etc.) before or after we evaluate respondents on the outcome of interest. Although there are methodological tools to detect if items exhibit bias (Millsap, 2011), more guidance is needed to determine if the presence of measurement bias materializes in meaningful differences in the screener’s performance across priority groups. Our research group is currently attempting to address this problem.
Screening measures are typically administered via self-report, and the item responses are then added up to a total score, which is compared to a cut off score. The cut score functions as a binary decision rule by selecting individuals who are most at risk while filtering others. Cut scores are derived by using a gold-standard criterion (such as a diagnosis made by a clinician) and determining which score maximizes the number of individuals who are identified as being at risk by the screener. For example, if we are interested in screening for depression, we want our screening measure to have a cut score, which can capture individuals who are truly high enough on the construct (i.e., depression) to receive a diagnosis and benefit from treatment (i.e., true positives), while ruling out individuals who would not receive a diagnosis (i.e., true negatives). Two common metrics to assess screening performance are sensitivity and specificity. Sensitivity is the probability that the screening measure will flag a respondent who is a true positive (i.e., respondent who truly has depression). Specificity is the probability that the screening measure will rule out a true negative (i.e., respondent who truly does not have depression). A screening measure that performs perfectly would have 100 percent specificity and sensitivity, but this is rarely the case — item responses can contain both measurement error as well as systematic error related to respondents’ background characteristics, as mentioned above. Research in the area of measurement invariance and differential item functioning (Millsap, 2011) has been useful in identifying items that exhibit measurement bias via statistical significance tests, but these methods fail to clarify the extent to which measurement bias would affect the practical utility of a measure.
Suppose a researcher has found statistically significant measurement bias in several items on a measure. The logical question to ask is whether that measurement bias should be cause for alarm. As is the case in many applications of statistical methods, there is no simple rule of thumb for distinguishing between statistical and practical significance when measurement non-invariance is identified. Effect sizes are intended to make the assessment of practical significance easier, but the standardized metric of many effect sizes can obscure their interpretation. For instance, it is not often clear what a mean difference of half a standard deviation unit means for test fairness and equity. To address the question of whether a screening measure is fair and equitable, our research group based in the L. L. Thurstone Psychometric Laboratory at the University of North Carolina at Chapel Hill has recently developed tools that clarify the impact of bias in a language and metric that is familiar to applied researchers working with screening measures. This language is that of sensitivity and specificity and turns the question into, “What proportion of individuals in this group will my measure fail to identify if I use the measure as is?”.
The technical details for the procedure are as follows. A user first provides group specific parameters (for noninvariant items) obtained from their data. The procedure uses these parameters and Monte Carlo simulation methods to generate artificial data, which contain the “true” scores (i.e., latent variable scores) as well as observed sum scores. Datasets are generated for two different assessment scenarios: one in which items with measurement bias are accommodated and the other in which items with measurement bias are ignored (Gonzalez, Pelham III, & Georgeson, in press). Using the dataset in which measurement bias is accommodated, a preestablished cut score is used to find the corresponding true cut score, and these scores are used to separate the bivariate data into quadrants. Using methods similar to those employed by Millsap and Kwok (2004), we use the proportions of datapoints within the quadrants to estimate the expected sensitivity and specificity across assessment scenarios to determine if ignoring measurement bias materially affects the sensitivity and specificity of the screener across groups.
Ultimately, the output of our procedure communicates how sensitivity and specificity of the screeners are affected if measurement bias is ignored. For instance, the output could report that a test is 8 percent less sensitive in one group, meaning that 8 percent of people who truly meet the criteria are missed by the screening measure when non-invariance is not accounted for. Researchers could then decide if the change in sensitivity and specificity adversely affects a group, or if the screener is still fit for purpose. By making the impact of measurement bias on screening performance more transparent, our work empowers the assessment specialist to determine if they could tolerate the presence of measurement bias in the assessment. If the currently accepted cut score will result in missing a substantial portion of individuals in a priority group, then recommendations can be made about how to better use the screener in different populations.
We have extended the procedure to work with continuous, binary and ordinal data, and are currently working on developing an R package that will be available on the CRAN repository. The functions we have developed, (which are either in press or forthcoming), were designed to be flexible so that researchers can enter the parameters obtained from Mplus or IRTPRO by hand, or use output from popular latent variable modeling packages such as mirt and lavaan. Additionally, the procedure does not require that the measure has a predefined cut score, which widens the applicability of the procedure to other scenarios.
While the inspiration and application of our work has focused on mental health screening, the applications of the methods we are developing are not limited to those screening scenarios. Tests and assessments with cut scores have been used in other environments to make decisions, such as admission practices, hiring or licensing. Future directions of our work include quantifying how measurement bias affects selection accuracy and make the effects of measurement bias more transparent. Finally, we would like to thank Roger Millsap, Oscar’s late mentor at Arizona State University and past president of APA Div. 5 (2004), for inspiring this research.
References
Auyeung, B., Baron-Cohen, S., Wheelwright, S., & Allison, C. (2008). The autism spectrum quotient: Children’s version (AQ-Child). Journal of Autism and Developmental Disorders, 38, 1230-1240.
Azocar, F., Arean, P., Miranda, J., & Muñoz, R. F. (2001). Differential item functioning in a Spanish translation of the Beck Depression Inventory. Journal of Clinical Psychology, 57, 355-365.
Gonzalez, O., Pelham, W. E. III, & Georgeson, A. R. (in press). Impact of measurement bias on screening measures. In Quantitative Psychology: 84th Annual Meeting of the Psychometric Society, Santiago, Chile, 2019.
Meredith, W., & Teresi, J. A. (2006). An essay on measurement and factorial invariance. Medical Care, S69-S77.
Millsap R.E. (2011). Statistical approaches to measurement invariance. New York: Routledge. Millsap, R. E., & Kwok, O. M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93-115.
Substance Abuse and Mental Health Services Administration (SAMHSA). (2017). Key substance use and mental health indicators in the United States: Results from the 2016 National survey on drug use and health (HHS Publication No. SMA 17-5044, NSDUH Series H-52). Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration.
Vilagut, G., Forero, C. G., Barbaglia, G., & Alonso, J. (2016). Screening for depression in the general population with the Center for Epidemiologic Studies Depression (CES-D): a systematic review with meta-analysis. PLOS ONE, 11, e0155431.