The existence of a college admissions cheating scheme has drawn weeks of media attention. Shockingly, this cheating service seemed like an auto-detailing shop with different option packages including cheating on standardized tests and the VIP fake athletic admissions treatment. It revealed an unsupervised side door into colleges and the fact that test administrators are often the weakest link in test security.
However, I may be one of the few people who see this scandal as largely a good sign and not the main problem with admissions. Cheating is clearly bad, but our reaction to it was good. Any system with valued rewards is going to have cheaters. What matters is how a society handles this reality. In this case, law enforcement took tips seriously, conducted a lengthy investigation, and this reaction lead to charges and prosecution. In addition, there was national outrage and cheating parents have been publicly shamed. We will see what the sentences are, but so far, I think we are in a good place so long as we continue to be vigilant.
Each of the students who benefited from the cheating displaced another student who was legitimately deserving of the spot. This is a serious and real cost, especially if this robbed someone of entry into their dream school. At the same time, if the displaced students would have been admitted to one of these prestigious schools, they were probably very competitive applicants to begin with. The reality is that the displaced students were probably admitted to good colleges and are doing well. I certainly hope so.
A bigger problem are the tools and processes used by many schools. Colleges want to evaluate the whole student, and this has led to more weight on interviews, personal statements, accomplishments and letters of recommendation. Schools want to know about a host of important characteristics including resilience, leadership, empathy, drive and curiosity. These are all important but are difficult to measure accurately under the best circumstances. Many admissions systems are a long way from state of the art. This is an opportunity for Div. 5 because a lot of work needs to be done and colleges need to become aware of the need.
We have conducted meta-analyses examining how many of these measures correlate with success and the results are unsettling. Student essays and personal statements are nearly uncorrelatedopens in new window with success (Murphy, Klieger, Borneman, & Kuncel, 2009). Other studies even demonstrate that they aren’t even good measures of a student’s writing skill. Undergraduate letters of recommendation are weak predictorsopens in new window and heavily overlap with prior grades and tests (Kuncel, Kochevar, & Ones, 2014). It appears that letter writers mostly reflect the students’ academic record along with their own idiosyncratic rating and writing foibles.
Interestingly, in some new workopens in new window, we found good evidence that professionally developed measures of background and accomplishments (often called biodata) turn out to be pretty good predictors of success across multiple domains. But my professional experience is that these characteristics are usually captured informally and then evaluated judgmentally by admissions committees. This raises a lot of basic questions. Is team captain more valuable than patrol leader for a Scout troop? How are science awards related to social class? Does the reviewer’s background influence how they evaluate an accomplishment in the arts? We just don’t know.
Finally, research on admissions interviews, focused mostly on medical admissions, has revealed that traditional interviews don’t work very wellopens in new window and are prone to bias (Basco, Lancaster, Gilbert, Carey, & Blue, 2008). It seems that an informal, ill-trained alumni interview is mostly garbage and probably influenced by the attractivenessopens in new window of the student and their similarity to the interviewer (Gilmore, Beehr, & Love, 1986). Who would have imagined?
All of this is coupled with a decision-making process that is sometimes terribly flawed. It’s been known for decadesopens in new window that decision makers struggle to make the most of numerous pieces of information while trying to maximize multiple goals (Kuncel, Klieger, Connelly, & Ones, 2013). These goals can include yield, diversity, academic preparedness, campus climate, geographic representation and excellence in the arts, sciences and athletics. Admissions personnel are being asked to do an overwhelming cognitive task. For this kind of task, the literature clearly shows that experts are inconsistent in how their use information and this inconsistency leads to lower accuracy.
To their credit, a number of colleges are taking this research gap seriously and are investing in upgrading their tools and processes. They have extensive archival data that can answer some of the most pressing questions. An ongoing challenge is the intersection of science and practice. Some evidence-based practices are simply not feasible in college admissions. For example, it is better to ask the same questions in every interview. But the reality is that this question list would end up on the Internet in a flash. Figuring out how to develop and then evaluate a feasible high-quality system is what we do well, and good work is happening in some places but not everywhere. Some administrators seem happier not knowing. Given the threat of lawsuits, I understand this impulse but also believe that proactively addressing the problem can help restore people’s faith in the system.
The current state of admissions represents an opportunity and maybe even a duty for Div. 5. Better assessments, paired with methodologically sound evaluation, are precisely what is needed. We already have better tools and processes on the shelf; they just need to be implemented. It is my hope that groups of colleges will partner with our members to modernize admissions. This will lead to better and fairer matching of students to schools.
References
Basco, W. T., Lancaster, C. J., Gilbert, g. E., Carey, M. E., & Blue, A. V. (2008). Medical School Application Interview Score Has Limited Predictive Validity for Performance on a Fourth Year Clinical Practice Examination. Advances in Health Sciences Education, 13(2), 1510162. https://doi.org/10.1007/s10459-006-9031-5opens in new window
Gilmore, D. C., Beehr, T. A., & Love, K. G. (1986). Effects of applicant sex, applicant physical attractiveness, type of rater and type of job on interview decisions. Journal of Occupational and Organizational Psychology, 59(2), 103-109. https://doi.org/10.1111/j.2044-8325.1986.tb00217.xopens in new window
Kuncel, N. R., Kochevar, R. J., & Ones, D. S. (2014). A Meta‐analysis of Letters of Recommendation in College and Graduate Admissions: Reasons for hope. International Journal of Selection and Assessment, 22(1), 101-107.https://doi.org/10.1111/ijsa.12060opens in new window
Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98(6), 1060-1072. https://dx.doi.org/10.1037/a0034156opens in new window
Murphy, S. C., Klieger, D. M., Borneman, M. J., & Kuncel, N. R. (2009). The Predictive Power of Personal Statements in Admissions: A Meta-Analysis and Cautionary Tale. College & University, 84(4), 86-86,88.