This article discusses the role of holistic review in identifying applicants to Health Professions Education (HPE) programs with a particular focus on the value of Situational Judgement Tests (SJTs) at the initial screening stage of admissions selection. The societal and institutional imperatives for effective selection are outlined first. Namely, training cohorts of practitioners who possess strong technical skills, are socially intelligent, professional, and diverse with respect to race, ethnicity, gender, and socioeconomic status will meaningfully reduce the administrative costs of remediation throughout training, and more importantly, improve the quality and satisfaction of care for the patients. While initial results of holistic admissions selection are promising, particularly with respect to gaining more diverse cohorts, mechanisms for efficient and effective initial screening have not yet been widely adopted. SJTs are well positioned to fill this gap, offering efficient assessments of interpersonal skills and professionalism that also relate to performance during training. A 2020 systematic review and meta-analysis on the association between SJT scores at admission and scores on in-person individual assessments in-program along with clinical performance showed a moderate effect (pooled correlation coefficient = 0.32, 95%CI = 0.26 - 0.39, p < .0001) (Webster et al., 2020). Of note, open-response SJTs using behavioral tendency questions have most recently gained wider adoption due to their promising predictive validity throughout training and smaller performance differences across demographic groups.
The importance of balanced selection
Health Professions Education (HPE) programs have a responsibility to participate in and improve care. Accordingly, these programs must find and train individuals who will exhibit the technical mastery, social intelligence, and professionalism needed to deliver high-quality, cost-effective, and equitable care (Ellaway et al., 2017; Reeve et al., 2017). Training individuals with this balance of attributes has shown to benefit society. For instance, higher self-reported physician empathy on the Jefferson Scale of Empathy has been associated with higher after-care patient-to-provider satisfaction (Wang et al., 2018). Physicians are more likely to work in communities of people with common identity characteristics (such as areas traditionally underserved and underrepresented) and in these instances patients report greater satisfaction and adherence to treatment (Goodfellow et al., 2016; Talamantes et al., 2019).
Selecting trainees who exhibit social intelligence and professionalism can also reduce HPE program’s administrative costs. In a study of surgery residency, the cost of just selecting residents ranged between USD$45,000 to $148,000 per program annually. Then remediation, predominantly attributable to non-technical abilities, cost an additional USD$3,400 to $5,300, with 30% of residents requiring these interventions (Gardner et al., 2018). The financial cost of failure across HPE programs has been estimated at US$9,371 (Foo et al., 2019). These costs have been shown to persist throughout training. A case-control study of 108 trainees who required professionalism-related remediation during undergraduate medicine showed that these individuals were five times more likely to require disciplinary review in residency, the next phase of their training. And later in clinical practice, these individuals were twice as likely to be sanctioned, sued or make review appearances (Krupat et al., 2020). Aside from the ultimate quality of care, given the direct and indirect resources required to review and remediate professionalism lapses, it is clearly essential to identify prospective trainees who are extremely unlikely to undergo remediation or fail.
Holistic review
Considering the institutional and societal benefits of selecting HPE applicants with strong social intelligence and professionalism skills, it is not only logical but imperative to emphasize these characteristics throughout the admissions process. In fact, holistic review in admissions has been widely adopted over the past two decades. Guiding definitions, principles and advantages are outlined by major governing bodies. The Association of American Medical Colleges (AAMC) describes holistic review as: “mission-aligned admissions or selection processes that take into consideration applicants’ experiences, attributes, and academic metrics as well as the value an applicant would contribute to learning, practice, and teaching. Holistic Review allows admissions committees to consider the “whole” applicant, rather than disproportionately focusing on any one factor.” (AAMC, 2021)
One of the first large-scale evaluations of holistic review (Urban Universities for HEALTH, 2014) showed promising results with most HPE programs seeing increased diversity in gender, race and socioeconomic status while maintaining GPA and standardized testing scores of incoming cohorts. Despite these affirmations, there was a wide variety of holistic reviews in use and the effectiveness of each was largely unknown. The majority of programs had little evidence evaluating non-academic admissions attributes with respect to success in-program despite a core principle of holistic review from the AAMC being that it should be “supported by student performance data that show that certain experiences or characteristics are linked to that individual’s likelihood of success as a student and/or physician” (AAMC 2021). The next section discusses contemporary and promising methods in implementing effective, equitable, and efficient holistic review.
Note, also important to holistic review, but not addressed in this article is the implementation of structured, mission-aligned multiple mini-interviews, implicit bias, and holistic review training for admissions committees (AAMC 2021). A number of resources on these subjects exist and interested readers are encouraged to seek out these tools.
Situational judgement tests
Critical to effective, equitable and efficient holistic review is adopting an intentional strategy throughout the selection process. Artinian et al., 2017 suggest including and weighting non-academic attributes specifically at the initial screening stage. Programs that balanced ‘academic’ and ‘non-academic’ screening measures showed greater diversity in interview and matriculant pools compared to programs who relied on GPA and standardized academic measures alone (e.g. MCAT and DAT). While some of this information (e.g., race, ethnicity, socioeconomic indicators) can be gathered through self-reported applications, it is logistically infeasible to invite and manually review personal statements, essays, and reference letters of all applicants initially screened. Further, these written statements are particularly poor indicators of predicting future success (Patterson, Knight, et al., 2016).
Recently, Situational Judgement Tests (SJTs) have been adopted to gauge interpersonal, intrapersonal, and professionalism-related skills at initial screening. SJTs provide test takers with a series of hypothetical scenarios and assess the individual’s response to the situation either in a closed-response (multiple choice) or open-response (free-text, audio, visual) format. The evaluations of SJTs are usually centralized, allowing all programs to review candidates’ performance across the cohort (as opposed to program-specific tests). Accordingly, SJTs are more efficient compared to reviewing personal statements (Patterson et al., 2016).
Perhaps most importantly, SJTs show good predictive validity. A 2020 systematic review and meta-analysis on the association between SJT scores at admission and ‘ratings at face to face, one-off, assessments’ in-program along with ‘evaluations of longer term clinical training or work performance’ showed a moderate effect (pooled correlation coefficient = 0.32, 95%CI = 0.26 - 0.39, p < .0001) (Webster et al., 2020). However, SJTs vary with respect to targeted aspects and question format. Closed-response (multiple choice) formats for instance, invoke a form of procedural knowledge wherein applicants identify what should be done. While of some value, this ability does not necessarily indicate how an applicant will behave in the future. In contrast, some open-response SJTs use behavioral tendency questions to ask the applicants what they would do in a given situation (Corstjens et al., 2017). This is an advantageous approach since it correlates more strongly with assessments of personality as compared to cognitive abilities (McDaniel et al., 2007). Behavioral tendency questions produce smaller performance differences across demographic groups relative to knowledge type questions (Whetzel et al., 2008). Additionally, open-response format SJTs tend to produce lower demographic differences relative to close-ended response options (Lievens et al., 2019).
While broad adoption is nacient (only since 2015) there have been positive associations between the most widely used open-response SJT for HPE admissions selection and in-program or clinical measures of social intelligence and professionalism skills. This relationship has been seen throughout training from the interview stage through to residency selection. In medicine, Casper (an open-response SJT) score at time of entry is associated with increased odds of obtaining the highest Standardized Patient Experience score at interview (OR= 1.31, 95%CI = 1.22 - 1.42) (Altus Assessments, 2021). Then, in preclinical training, Casper scores have been significantly associated with Professional and Skill Development sub-scores from the Objective Structured Clinical Exam (β =0.53, p=.02) (Yan et al., 2021). Additionally, there has been evidence of associations with professional identity (r= 0.10, p<. 05) and moral reasoning (r=0. 09, p<. 05) (Yingling et al., 2018). During clerkship, the odds of obtaining the highest advisor rating also increased with each point increase in Casper (OR = 1.34, 95%CI = 0.98 - 1.84) (Woodson et al., 2022). Finally, Casper scores have shown moderate correlation (r=0.30 to 0.50, p<0.05) with Canadian medical licensing exam scores in sub-sections related to 1) cultural-communication, legal, ethical, and organizational aspects of the practice of medicine (CLEO) and 2) population health and ethical, legal, and organizational aspects of medicine (PHELO) (Dore et al., 2017), along with being a predictor of residency match success (OR= 2.01, 95%CI: 1.08 - 3.75) (Burgess et al., 2020).
While the magnitude of these results vary depending on the study sample and alignment between selection measure and outcome criterion, the association between open response SJT scores and measures of interpersonal and professionalism skills throughout training indicate it's potentially instrumental role in the future success of holistic review.
Conclusion
This article discusses the societal and institutional imperatives for identifying applicants to HPE programs. Namely that training a cohort of practitioners who possess strong technical skills, are socially intelligent, professional, and diverse with respect to race, ethnicity, gender, and socioeconomic status will meaningfully reduce the administrative burden of remediation throughout training, and will also improve the quality and satisfaction of care for the patients. Holistic admissions has been a pillar in the last 2 decades towards developing this balanced health profession generation. While initial results of holistic admission are promising, mechanisms for efficient and effective initial screening have not yet been widely adopted. SJTs are well positioned to fill this gap, offering efficient assessments of interpersonal skills and professionalism that also relate to performance during training. Of note, open-response SJTs using behavioral tendency questions have most recently gained wider adoption due to their promising predictive validity through training and smaller performance differences across demographic groups.
References
Artinian, N. T., Drees, B. M., Glazer, G., Harris, K., Kaufman, L. S., Lopez, N., Danek, J. C., & Michaels, J. (2017). Holistic admissions in the health professions: Strategies for leaders. College and University : The Journal of the American Association of Collegiate Registrars, 92(2), 65–68.
Burgess, R., Barber, C., Mountjoy, M., Whyte, R., Vanstone, M., & Grierson, L. (2020, September 7). Evaluating the validity evidence of admissions and medical school performance variables on student outcomes at McMaster University. AMEE 2020.
Corstjens, J., Lievens, F., & Krumm, S. (2017). Situational Judgement Tests for Selection. In H. W. Goldstein, E. D. Pulakos, J. Passmore, & C. Semedo (Eds.), The Wiley Blackwell Handbook of the Psychology of Recruitment, Selection and Employee Retention (1st ed., pp. 226–246). Wiley. https://doi.org/10.1002/9781118972472.ch11
Dore, K. L., Reiter, H. I., Kreuger, S., & Norman, G. R. (2017). CASPer, an online pre-interview screen for personal/professional characteristics: Prediction of national licensure scores. Advances in Health Sciences Education, 22(2), 327–336. https://doi.org/10.1007/s10459-016-9739-9
Foo, J., Cook, D. A., Walsh, K., Golub, R., Abdalla, M. E., Ilic, D., & Maloney, S. (2019). Cost evaluations in health professions education: A systematic review of methods and reporting quality. Medical Education, 53(12), 1196–1208. https://doi.org/10.1111/medu.13936
Gardner, A. K., Grantcharov, T., & Dunkin, B. J. (2018). The science of selection: Using best practices from industry to improve success in surgery training. Journal of Surgical Education, 75(2), 278–285. https://doi.org/10.1016/j.jsurg.2017.07.010
Goodfellow, A., Ulloa, J. G., Dowling, P. T., Talamantes, E., Chheda, S., Bone, C., & Moreno, G. (2016). Predictors of primary care physician practice location in underserved urban and rural areas in the United States: A systematic literature review. Academic Medicine : Journal of the Association of American Medical Colleges, 91(9), 1313–1321. https://doi.org/10.1097/ACM.0000000000001203
Holistic Admissions Diversifies Health Pro Schools | Feature Story. (2014). National Inistitute of Health. https://www.nimhd.nih.gov/news-events/features/training-workforce-dev/holistic-admissions.html
Holistic Review. (2021). AAMC. https://www.aamc.org/services/member-capacity-building/holistic-review
Krupat, E., Dienstag, J. L., Padrino, S. L., Mayer, J. E., Shore, M. F., Young, A., Chaudhry, H. J., Pelletier, S. R., & Reis, B. Y. (2020). Do professionalism lapses in medical school predict problems in residency and clinical practice? Academic Medicine, 95(6), 888–895. https://doi.org/10.1097/ACM.0000000000003145
Lievens, F., Sackett, P. R., Dahlke, J. A., Oostrom, J. K., & De Soete, B. (2019). Constructed response formats and their effects on minority–majority differences and validity. Journal of Applied Psychology, 104(5), 715–726. https://doi.org/10.1037/apl0000367
McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & GRUBB III, W. L. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60(1), 63–91.
Patterson, F., Knight, A., Dowell, J., Nicholson, S., Cousans, F., & Cleland, J. (2016). How effective are selection methods in medical education? A systematic review. Medical Education, 50(1), 36–60. https://doi.org/10.1111/medu.12817
Patterson, F., Zibarras, L., & Ashworth, V. (2016). Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100. Medical Teacher, 38(1), 3–17. https://doi.org/10.3109/0142159X.2015.1072619
Russo, A. (2021, June 16). Research shows Casper can support holistic admissions at Texas A&M College of Medicine. Altus Assessments. https://altusassessments.com/casper/research-shows-casper-can-support-holistic-admissions-at-texas-am-college-of-medicine/
Talamantes, E., Henderson, M. C., Fancher, T. L., & Mullan, F. (2019). Closing the gap—Making medical school admissions more equitable. New England Journal of Medicine, 380(9), 803–805. https://doi.org/10.1056/NEJMp1808582
Wang, H., Kline, J. A., Jackson, B. E., Laureano-Phillips, J., Robinson, R. D., Cowden, C. D., d’Etienne, J. P., Arze, S. E., & Zenarosa, N. R. (2018). Association between emergency physician self-reported empathy and patient satisfaction. PLOS ONE, 13(9), e0204113. https://doi.org/10.1371/journal.pone.0204113
Webster, E. S., Paton, L. W., Crampton, P. E. S., & Tiffin, P. A. (2020). Situational judgement test validity for selection: A systematic review and meta-analysis. Medical Education, 54(10), 888–902. https://doi.org/10.1111/medu.14201
Whetzel, D. L., McDaniel, M. A., & Nguyen, N. T. (2008). Subgroup differences in situational judgment test performance: A meta-analysis. Human Performance, 21(3), 291–309. https://doi.org/10.1080/08959280802137820
Woodson, M., Hussin, A., & MacIntosh, A. (2022, April). Evaluating medical school admissions screening measures to promote diversity while relating to future clerkship performance. American Educational Research Association. 2022 AERA Annual Meeting, San Diego, CA.
Yan, X. W., Wood, T. J., & Lemay, G. (2021). Canadian Conference on Medical Education 2021 Abstracts. Canadian Medical Education Journal, 12(2), OF1-3–2, 60601. https://doi.org/10.36834/cmej.72402
Yingling, S., Park, Y. S., Curry, R. H., Monson, V., & Girotti, J. (2018). Beyond cognitive measures: Empirical evidence supporting holistic medical school admissions practices and professional identity formation. MedEdPublish, 7(4). https://doi.org/10.15694/mep.2018.0000274.1
Conflict of interest: Potential or perceived conflicts of interests may arise in that the authors are full-time scientists employed by Altus Assessments.