Call to action for ethical AI in education

Motivation

Artificial intelligence (AI) algorithms are being used increasingly across various fields (Tsamados et al., 2022), including education (US Department of Education, 2023) and healthcare (Alowais et al., 2023).  In fact, as observed with the launch of ChatGPT and ongoing discussions around its impact (Montenegro-Rueda et al., 2023), AI has already changed the landscape of education as we know it. As AI use continues to grow, so do the associated risks and concerns. One major risk has to do with fairness: AI algorithms can be trained on biased data, which leads to inequitable and unreliable outcomes. AI bias has been documented in tools used in recruitment (Lambrecht & Tucker, 2019), in healthcare (Kiyasseh et al., 2023), as well as in the case of large language models like ChatGPT, where evidence of bias was found both in terms of race (Omiye et al., 2023) and gender (Kotek et al., 2023). 

Mission

Our responsibility as members of the education community is to ensure that the AI tools we use are designed in alignment with human values (Han et al., 2022), and that they consider both ethical implications (Ammanath, 2022), as well as legal and governmental requirements, such as the Blueprint for an AI Bill of Rights (U.S. Office of Science and Technology Policy, 2022). This responsibility does not rest only with those in the EdTech sector, but with everyone who interacts with AI in Education. 

One way to harness AI for the greater good is to create and uphold ethical frameworks governing the design, development, and implementation of AI models in education. Such frameworks consist of a set of actionable ethical principles (Ammanath, 2022) which can be applied at different stages of AI development by considering key questions (e.g., Who will be impacted by the algorithm?) with the purpose of mitigating potential risks. In line with leading tech organizations and higher education institutions, at Acuity Insights we have been developing our own set of guiding principles and considerations as we tread the waters of society’s increasing use of AI both in day-to-day activities as well as in admissions (Lira et al, 2023) and higher education more broadly. However, such AI principles tend to overlap significantly across the different sectors (Prem, 2023) and may risk being too high-level to monitor and implement (Munn, 2023). For this reason, in the following section, we outline the Ethical AI principles Acuity Insights subscribes to and list some key practical questions any AI user (educators, students, developers, etc.) can consider to guide and assess their responsible use of AI in education.

Ethical AI principles

Over the past year, we have reviewed recommendations for responsible AI spanning various organizations, from governing bodies, the technology sector, to higher education institutions, and have summarized these insights into 5 core principles (see Figure 1). In a nutshell, both EdTech companies and the larger education community should strive to ensure that:

• AI tools are fair and potential biases and inequities can be mitigated;
• Results are reliable and AI tools perform as intended;
• Strategies are in place to responsibly assess and address any potential risks;
• User data is secure and security guidelines are upheld;
• Users have access to clear explanations of how the AI tools work.

Figure 1. Ethical AI principles and key questions

In our endeavor to document the various facets we need to consider in our AI journey, we have compiled a list of questions that can guide us when assessing whether the AI tools we use and develop meet our Ethical AI principles. What follows is a brief summary of each of these principles, as well as a few examples of the relevant questions any AI user can ask themselves when evaluating these criteria. 

Fair, equitable. Detect, document, and mitigate biases in the training and output data, and ensure the algorithms apply and uphold standards for fairness.

• Is personal data being used as input? Can and should this be avoided?
• How does your institution assess whether the results of the AI tool are biased in any way?
• How does the AI tool help mitigate any existing biases? What additional mitigation strategies can be employed? 

Reliable, rigorous. Ensure that the AI system is robust and can perform reliably, as intended, even in less-than-ideal conditions.

• Given that AI tools may produce unreliable results, how will you or your institution verify the accuracy of the tool’s output?
• What steps will be taken when reliability issues and concerns are encountered?
• What are the limitations and assumptions of the tool? What real-world factors outside of the training data might lead to different outcomes? 

Responsible, accountable. Ensure that the AI system is aligned with human values, and that proper governance and mitigation strategies are in place for potential unintended outcomes.

• What are the tool’s objectives and desired outcomes? How are they aligned with your institution’s mission and values? Do the benefits outweigh the risks?
• What are some potential risks and unintended consequences of using this tool? What are your mitigation plans should these risks occur?
• Who is accountable for the implementation of the tool’s recommendations and results? 

Secure, confidential. Respect data privacy, abide by data protection regulations, and uphold data security measures to prevent and mitigate data breaches.

• Does the AI tool’s organization have publicly available privacy and security policies? Does your institution have privacy and security policies?
• What data are you sharing with the organization that developed the AI tool? What rights do they have to this tool? Can users opt out?
• What measures are in place to ensure the security of the collected data and who is responsible for evaluating compliance with data privacy rules? How is the AI system safeguarded against any potential cyberattacks and exploits? 

Transparent, explainable. Provide users with meaningful and clear information to nurture general understanding around how the algorithm works.

• At a high level, do users have access to documentation outlining the key factors that drive the AI tool’s decisions and the needed input data?
• Can users and affected parties explain how the tool works? How is high-level information being shared with users?
• How is feedback being collected from all the relevant end users, stakeholders, and affected parties? 

Working with the education community 

While the AI principles and questions we drafted were based on a literature review of existing guidelines, they were largely built from the perspective of EdTech researchers and developers. As AI affects the broader education community, we want to ensure that the guidelines we recommend and employ capture a variety of perspectives (students, educators, advisors, etc.). For instance, in a workshop we facilitated alongside faculty in Education and Physical Therapy (Ivan et al., 2023), one common theme the audience highlighted was the need for educational institutions (e.g., universities, schools) to develop their own policies around the use of AI. To this end, we hope that the brief overview we offer in this paper can provide an example of what applicable Ethical AI principles could look like and how one might assess them.

Motivated by our research and discussions in this field, we believe all members of the education community have the right and responsibility to not only define what Ethical AI in education means, but also to ensure these defined goals are met. Subscribing to this spirit of transparency and accountability, we would love to hear from you, learn more about your journey with AI in the education sphere and adapt our responsible AI recommendations and principles. Please do reach out if you would like to provide feedback, share insights, find out more, or collaborate with us!

References

Acuity Insights Ethical AI in Education. (2023). https://acuityinsights.com/ethical-ai-in-education Last accessed December 12, 2023.

Alowais, S.A., Alghamdi, S.S., Alsuhebany, N. et al. (2023). Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Medical Education 23, 689. https://doi.org/10.1186/s12909-023-04698-z

Ammanath, B. (2022). Trustworthy AI. Wiley.

Han, S., Kelly, E., Nikou, S. et al. (2022). Aligning artificial intelligence with human values: reflections from a phenomenological perspective. AI & Society 37, 1383–1395. https://doi.org/10.1007/s00146-021-01247-4

Ivan, R., A. MacIntosh, L. Traywick, C. Herringer, G. Musolino, P. A. Salamh, S. F. Maher, , O. Bulut, J. Shin, M. Condo. (2023). “Ethical AI in Health Professions Education: Co-creating an ethical framework to ensure responsible AI use”. Workshop at the 18th Annual Physical Therapy Educational Leadership Conference. October 13-15, 2023, Philadelphia, Pennsylvania, USA. https://view-su2.highspot.com/viewer/6570ade2a27219a272119583

Kiyasseh, D., Laca, J., Haque, T.F. et al. (2023). Human visual explanations mitigate bias in AI-based assessment of surgeon skills. npj Digit. Med. 6, 54. https://doi.org/10.1038/s41746-023-00766-2

Kotek, H., R. Dockum, D. Sun. (2023). Gender bias and stereotypes in Large Language Models. In Proceedings of The ACM Collective Intelligence Conference (CI '23). Association for Computing Machinery, New York, NY, USA, 12–24. https://doi.org/10.1145/3582269.3615599

Lambrecht, A., C. Tucker. (2019). Algorithmic Bias? An Empirical Study of Apparent Gender-Based Discrimination in the Display of STEM Career Ads. Management Science 65 (7): 2966-2981. https://doi.org/10.1287/mnsc.2018.3093

Lira, B., et al. (2023). Using artificial intelligence to assess personal qualities in college admissions. Science Advances Vol 9, issue 41. DOI:10.1126/sciadv.adg9405

Montenegro-Rueda, Marta, José Fernández-Cerero, José María Fernández-Batanero, and Eloy López-Meneses. 2023. "Impact of the Implementation of ChatGPT in Education: A Systematic Review" Computers 12, no. 8: 153. https://doi.org/10.3390/computers12080153

Munn, L. (2023). The uselessness of AI ethics. AI Ethics 3, 869–877. https://doi.org/10.1007/s43681-022-00209-w

Omiye, J.A., Lester, J.C., Spichak, S. et al. (2023) Large language models propagate race-based medicine. npj Digital Medicine. 6, 195. https://doi.org/10.1038/s41746-023-00939-z

Prem, E. (2023). From ethical AI frameworks to tools: a review of approaches. AI Ethics 3, 699–716 (2023). https://doi.org/10.1007/s43681-023-00258-9

Tsamados, A., Aggarwal, N., Cowls, J. et al. (2022). The ethics of algorithms: key problems and solutions. AI & Society 37, 215–230. https://doi.org/10.1007/s00146-021-01154-8

U.S. Department of Education, Office of Educational Technology. (2023). Artificial Intelligence and Future of Teaching and Learning: Insights and Recommendations, Washington, DC, 2023.

U.S. Office of Science and Technology Policy. (2022). Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People, at 3 (Oct. 2022), www.whitehouse.gov/ostp/ai-bill-of-rights.