My APA colleagues are often surprised to learn that my background and training are not in psychology, but in education. Although I recently retired from the Department of Graduate Psychology at JMU, I spent much of my career in the Colleges of Education at the Universities of Nebraska and Georgia. Because of my background, I regularly attend the annual meetings of both APA and AERA, the American Educational Research Association, as well as that of the National Council on Measurement in Education (NCME). I bring all this up because NCME has recently hosted several webinars and conference sessions on the use of culturally relevant and justice-oriented test items in educational testing that I thought might be of interest to members of Division 5. I made that research the topic of my recent presidential address at the APA Convention and summarize it in this newsletter article.
Given the current political climate in which states such as Florida have implemented the so-called “Don’t Say Gay” bill and removed the College Board’s Advanced Placement course on African American History from the state’s curriculum, it may surprise some to learn that other school districts and K-12 testing organizations in the U.S. are experimenting with test items that focus on cultural relevance and social justice. Culturally relevant, or culturally responsive, test items have grown out of the movements for culturally relevant pedagogy (Ladson-Billings, 1994; 1995; 2014) and culturally responsive teaching (Gay, 2002; 2018). Ladson-Billings (1995) describes culturally relevant pedagogy as “…a theoretical model that not only addresses student achievement but also helps students to accept and affirm their cultural identity while developing critical perspectives that challenge inequities that schools (and other institutions) perpetuate” (Ladson-Billings, 1995, p. 469). Culturally responsive teaching incorporates the experiences, perspectives, and cultural milieu of ethnically diverse students into all aspects of teaching with the goal of creating a learning experience that is more personally relevant, meaningful, interesting, and thus more easily learned. It is thought that such a learning experience will, in turn, result in higher academic achievement.
The development of culturally relevant and justice-oriented test items is based on three tenets of culturally relevant pedagogy: student achievement, cultural competence, and sociopolitical consciousness. In culturally relevant pedagogy, measurement of student achievement should be designed to reveal each student’s growth, rather than comparing students to each other. Teachers are encouraged to use different types of assessments, reflecting the understanding that students learn in different ways. Finally, culturally relevant teaching is designed to infuse aspects of students’ cultural backgrounds into instruction through culture-based examples and references, and by asking questioning that require students to consider situations from the perspectives of those with different sociopolitical viewpoints. Teachers can also enhance students’ sociopolitical consciousness through assignments requiring critiques of social inequities.
Antiracist assessment practices Randall (2021; Randall et al. 2022) are based on the argument that traditional approaches to achievement testing perpetuate racism through their exclusive focus on white content and ways of knowing. As Randall notes (p. 5), “Issues of inequity and injustice are not to be resolved in assessment simply by increasing ‘access’ to whiteness (i.e., opportunity to learn) to marginalized students,” because such content ignores the culture and experiences of large segments of the student population. Randall advocates the development of test items that are explicitly designed to disrupt or expose racist ideas. Such test items might, for example, confront negative stereotypes about minorities or be based on contexts that highlight inequities. Examples of antiracist items are not yet widely available, but recent NCME webinars and conference presentations, as well as one recent doctoral dissertation have provided examples of culturally relevant and justice-oriented items and students’ reactions to these. Although I cannot do justice to this research in this short article, I hope the examples and student reactions shared here will generate interest. Steedle and Anguiano-Carrasco (2023) provided the example below of a culturally relevant test item developed at the ACT:
The lyrics of Lin-Manuel Miranda’s Hamilton soundtrack are sung or rapped at an average rate of 144 words per minute. Given that the Hamilton soundtrack is 2 hours 23 minutes long, which of the following is closest to the total number of words in the Hamilton soundtrack?
- 2,500
- 3,500
- 19,000
- 20,500
- 32,000
Patterson (2023) created the justice-oriented item below for his doctoral dissertation research on college students’ perceptions of such items in the context of an undergraduate assessment of information literacy:
Shanice Richards wants to raise awareness of housing segregation. To inform her actions, she needs to research the history of housing segregation in the United States. Which of the following sources would be most useful for her to use?
A) A website by the Shenandoah Valley Black Heritage Project that features housing segregation
B) A website by a JMU professor that studies history
C) A chapter from The Color of Law—a book written by scholars in race and housing
D) A brochure from a realtor on houses in Harrisonburg
Randall (2021, p. 2) provides the example below of an antiracist math item:
Marcellus is cooking hot meals to hand out to a small group of twelve Black Lives Matters protesters demonstrating against separating families held at the U.S./Mexico border. He is making a meal of rice, cornbread, and red beans. He wants to make enough red beans for each person to have more than ¾ cup.Determine whether each inequality or number line correctly models c, the number of red beans Marcellus needs to make.
Select Yes or No for each option.
One question of interest to many researchers is how students react to these types of items. Do such items result in the hoped-for perceptions of more personal relevance, meaningfulness, interest, motivation, and engagement? If so, do these benefits result in higher academic achievement? Before attempting to answer these questions, I want to note that at a recent conference there was considerable pushback on my suggestion that more research was needed on these questions. Several audience members pointed out that we should not need research to support the inclusion in assessments of all members of society. Instead, the need for such inclusion should be taken as self-evident, thus negating the need for justification. I completely agree, but unfortunately, given today's political climate it seems inevitable that such questions will be asked.
To date, research on these questions has been qualitative, consisting mainly of cognitive interviews and focus groups with students of various ages. This research is focused on students’ perceptions of the items rather than students' achievement on such tests. The quotations below are taken from the NCME conference and webinar presentations I alluded to previously. When available, I included information on student ethnicity and grade level.
Overall, students appear to have positive reactions to the items, as evidenced in the quotations below from research conducted at WestEd and reported by Faulkner-Bond (NCME, 2022) as part of the NCME Equity Webinar series:
“I’m not sure if everybody had similar questions to look at, but I really did appreciate the diversity. I don’t want to say ‘I’...but most of the time people don’t really get to see a whole bunch of [representation] and just thought it was really cool to include everybody instead of just having one specific group of people.” (8th grade student)
Such support was voiced by both students of color and white students, but younger students seemed less aware of the items’ cultural content. The comment below about a culturally relevant item was made by a white undergraduate student in Patterson’s (2023) dissertation study:
“Okay. I liked it. I feel like I’m getting a sense of someone’s life that’s not mine. And it's interesting just seeing all of these different sources that all relate to the same topic but can go in different ways of how you perceive it and the different information that it gives about the item.”
Although students did appreciate the diversity of the items, many students seemed to care more about the personal interest items held for them than their diversity per se. The quotation below is from a middle school student in one of Faulkner-Bond's (NCME, 2022) focus groups:
“I don’t know if this is, like, the theme or not but I would not, like, change how interesting the questions are for diversity. I would rather read something more interesting than something I was included in. It’s a lot more enjoyable for me to read something that’s interesting and less lengthy.”
The reference to question length in the quotation above was a common theme. Students tended to prefer shorter questions, even if this was at the expense of diversity. The quotation below is from Steedle and Anguiano-Carrasco’s (2023) research on ACT items incorporating cultural relevant items:
“…if it's like a two sentence question or a four sentence cultural question I’d say go for the two sentence question…but if the questions are going to be the same length either way, I don’t think the cultural content is what is slowing us down.” (white 12th grade student)
Another student in Steedle and Anguiano-Carrasco’s (2023) study addressed the motivational value of culturally relevant items:
“I think because…assuming what...the mindset of students who aren't too fond of taking the test is, I feel like they are just more like ‘oh why is a richer you know Caucasian people making these tests for like me?’ I feel like them knowing...people who are higher up care about their culture, will like motivate them to wanna do better for their own culture, like seeing their culture represented…can be very eye opening. It can be...life changing for a student to see...a question like that, and then for them to be like ‘omg...they care about my culture...let me do this for my culture and for myself.’” (Latinx 11th grade student)
In his dissertation study, Patterson (2023) created both traditional and culturally relevant or justice-oriented versions of test items for an undergraduate test of information literacy. He then conducted focus groups with undergraduate students in which he asked about their perceptions of the items. One student commented that the items made them feel “seen:”
“[These culturally relevant items] made me feel represented in testing, which is something I didn’t think that I wanted until I got it." (Mixed race, transgender college student).
The qualitative research to date suggests that culturally relevant and justice-oriented items are generally perceived positively by students, with older students expressing more support for such items than younger students. White students were not always as enthusiastic about the new item types as were students of color and at least one student expressed difficulty in answering items with which they did not have direct experience, as evidenced by the comment below from a white student in Patterson’s (2023) dissertation study:
“I think it’s harder to answer these questions. I’m saying this because I’ve never experienced it. So, it’s just harder to, I guess, answer them sometimes.”
In Patterson’s (2023) dissertation study, some students had trouble with diverse names for people and places. This comment illustrates the possibility that inclusion of such items could result in construct irrelevant variance due to unfamiliar terminology. However, as Randall, Slomp, Poe, and Oliveri (2022, p. 4) note that “In some respects, then, it can be said that what is construct-relevant behavior is dependent on the eye of the beholder. What is construct irrelevant for the assessment designer, may very well be construct relevant for the examinee.” Construct relevance depends on the definition of the construct. If this definition includes students’ awareness of cultural factors or social justice issues, such material becomes construct relevant. A related issue is whether culturally relevant or justice-oriented contexts make items more cognitively complex. Such content can make items longer, as noted by Steedle and Anguiano-Carrasco (2023) and Patterson, among other researchers. Emotionally laden contexts might also be the source of construct irrelevant variance. Patterson investigated this issue and found that some students did express anger or other negative emotions on items related to unjust situations. For example, in the context of an item about the death or disappearance of indigenous women, one student stated:
“The emotion that I felt was anger because, again, it's not something that’s talked about. It's because it's a group of minorities, and like, at least in this country, unless it's a white person it's not really looked into.” (Latina undergraduate student)
As a measurement professional who supports greater equity in assessment, I applaud the efforts of those developing culturally relevant and justice-oriented assessments and look forward to wider development and adoption of these. My hope is that, in the not-too-distant future, such assessments will be taken as a matter of course by both students and the public.
References
Gay, G. (2002). Preparing for culturally responsive teaching. Journal of Teacher Education, 53(2), 106-116. https://journals.sagepub.com/doi/pdf/10.1177/0022487102053002003
Gay, G. (2018). Culturally responsive teaching: Theory, research, and practice (3rd ed.). Teachers College Press.
Ladson-Billings, G. (1994). The dreamkeepers: Successful teaching for African-American students. Jossey-Bass.
Ladson-Billings, G. (1995). Toward a theory of culturally relevant pedagogy. American Educational Research Journal, 32(3), 465-491. https://doi.org/10.3102/00028312032003465
Ladson-Billings, G. (2014). Culturally relevant pedagogy 2.0: A.k.a. the remix. Harvard Educational Review, 84(1), 74-135. https://doi.org/10.17763/haer.84.1.p2rj131485484751
Patterson, C. R. (2023). Observing how college students process culturally responsive test items. [Unpublished doctoral dissertation]. James Madison University.
Randall, J. (2021). Color-neutral is not a thing: Redefining construct definition and representation through a justice-oriented critical antiracist lens. Educational Measurement: Issues and Practice, 40(4), 82-90.
Randall, J., Slomp, D., Poe, M., & Oliveri, M.E. (2022). Disrupting white supremacy in assessment: Toward a justice-oriented, antiracist validity framework. Educational Assessment, 27(2), 170-178. https://doi.org/10.1080/10627197.2022.2042682
Steedle, J., & Anguiano-Carrasco, C. (2023, March 30). Developing culturally relevant math and science items: Lessons learned and student reactions. [Conference presentation]. Annual Meeting of the National Council on Measurement in Education, Chicago, IL.