In This Issue

Evaluation of English learners: Issues in measurement, interpretation and reporting

Current issues and recommendations for improvement in the assessment of English learners.

By Samuel O. Ortiz

One of the fastest growing segments of the current U.S. population is made up of individuals for whom the primary or native language is not English (U.S. Census Bureau Statistics, 2015). The substantive changes in the population suggest that psychologists will increasingly find themselves in the position of needing to conduct evaluation of intelligence, cognitive abilities, neuropsychological domains and the like using tools and instruments that may not have been specifically validated for these populations.

Even when the difficulties in conducting such evaluations have been addressed to the extent possible, there is a need to communicate findings in a manner that conveys the requisite meaning, inferences and diagnostic impressions with confidence that they represent defensible and valid conclusions. In evaluations with native-English speakers, the process is one that is very familiar to all applied psychologists and poses little difficulty on any of the various aspects involved (e.g., assessment philosophy, evaluation framework, legal requirements, tool/instrument selection, validity, interpretation). Much of this process is well outlined and clearly delineated in a variety of sources, in particular the Standards for Educational and Psychological Testing (AERA, APA & NCME, 2015).

The same cannot be said, however, for evaluation of English learners, especially since there do not exist any accepted or consensus standards for evaluation. Apart from general guidance on ensuring that evaluations are conducted with some sensitivity or attention to issues of cultural, linguistic and other types of diversity (e.g., APA, 2002; 1990), specifics are difficult to come by. The absence of any such framework is perilous because the fact that cultural and linguistic variables are present at all in the context of evaluation, and to whatever degree, always means they have the potential to invalidate the collected data, invoke additional legal and ethical requirements, and may well alter the manner in which the evaluation is conducted as well as the way in which it is documented (Ortiz, 2014). 

The role that linguistic differences play in affecting the process of evaluation can be seen in school-based evaluations conducted under the auspices of the Individuals with Disability Education Act (IDEA; 2004) which outlines the process of conducting evaluations designed to investigate suspected disabilities that may otherwise qualify a student for special education services. Both IDEA and its attendant regulations (i.e., Code of Federal Regulations 34 CFR) note specifically that a student cannot be determined as having a disability if the primary or “determinant” factor for the conclusion is related to “limited English proficiency.” The statute and regulations are even more prescriptive than other requirements that may govern such evaluations to the extent that they provide additional requirements regarding the process and materials used in the assessment to ensure that they: 

(i) are selected and administered so as not to be discriminatory on a racial or cultural basis; (ii) are provided and administered in the language and form most likely to yield accurate information on what the child knows and can do academically, developmentally, and functionally, unless it is not feasible to so provide or administer; (iii) are used for purposes for which the assessments or measures are valid and reliable; (IDEA, 2007). 

The corresponding federal regulations reinforce these notions and add even more specificity to the requirements of assessments within individuals who are not native English speakers to ensure that they are “provided and administered in the child’s native language or other mode of communication … materials and procedures used to assess a child with limited English proficiency are selected and administered to ensure that they measure the extent to which the child has a disability and needs special education, rather than measuring the child’s English language skills” (34 CFR § 300.532).

What is clear about these regulations is that they point toward the issue of validity. For example, evaluators are required by law to make a determination regarding the extent to which cultural and linguistic factors may have influenced the validity of the assessment data, which would, of course, influence subsequent interpretation of the data. Therefore, the most critical section in any report is a “determination,” that is, a discussion of the relevant cultural and linguistic factors and the extent to which these factors might be the primary cause of the individual’s observed learning difficulties rather than the presence of a disability. Although empirical investigation in this area is sorely lacking, at least one study appears to suggest that the frequency and degree to which reports of psychological evaluation actually document adherence to the requirements and discuss the manner of determination is nearly zero.

According to Figueroa & Newsome (2006), of the 19 psychological reports they reviewed, and which were conducted specifically under the provisions of IDEA, none of them provided any mention of the determination that the diagnosed disability was not due primarily to limited English proficiency. Moreover, none of them addressed whether the materials employed in the assessment were in fact valid for the purpose for which they were being used, or how validity may have been affected by use of an interpreter (as was used in six cases), or any discussion regarding whether the diagnosis might not have instead been a reflection of the normal process of second language acquisition rather than a handicapping condition (Figueroa & Newsome, 2006). In virtually all reports, a ubiquitous disclaimer was used that resembled wording to the effect, “the disability is not the result of limited English proficiency” with no explanation regarding how this was determined, or “the discrepancy is due to a disorder…and is not the result of environmental, cultural or economic disadvantage” (again, without any explanation as to how this was ascertained), and most often, “these results should be interpreted with extreme caution” which was always followed by interpretation that looked suspiciously like interpretation accomplished without any caution.

Simply ignoring the need to consider factors of diversity, such as limited English proficiency, or dispensing with their influence and potential adverse effect on construct validity with hollow assertions that rely on the reader’s pure trust, is poor and indefensible practice. Because the written report is merely a reflection of the evaluation that was conducted, paying attention to what should be included in the report serves the dual purpose of guiding the evaluations in the first place. To ensure full legal and ethical compliance, some simple recommendations are provided here that, if followed, can improve the quality of evaluations, assist in establishing the validity of the obtained results, and create the necessary foundation for drawing defensible and valid inferences and interpretations of the data. These guidelines include, at a minimum: 

  • Not a simple list of procedures or tests — but instead, a clear description of the process used in evaluation, a rationale for its selection, and the ways in which it helped to ensure fairness and responsiveness to the cultural and linguistic factors present in the case.
  • Not a generic background section — but instead, an extended discussion and analysis of the salient cultural and linguistic developmental experiences and how they are either the primary explanation for the observed academic problems (no disability) or are only contributory factors (leaves open possible disability).
  • Not a disclaimer about validity — but instead, a statement that clearly describes the process by which the assessment data were determined to be valid or invalid and which can defend and support subsequent interpretation. 

In addition, there are other considerations that can improve evaluations and their concomitant reports. One of these relates to readability and language. For example, most reports seem to be technical and written for other psychologists, not lay people. Parents of culturally and linguistically diverse students may have limited education and literacy. Simple, plain English reports are most likely to be helpful to parents and educators in making programmatic decisions and are more easily translated. Below is an illustration of this issue: 

  • Typical: “The Processing Speed Index measures Emilia’s ability to quickly identify, register and implement decisions based on visual stimuli. Her overall score was 111, and is considered to be in the high average range. The two subtests that measure this are Symbol Search and Coding. Emilia’s scores on these two subtests were in the average/ high average range with an 11 in Coding and a 13 in Symbol Search. Based on her scores, Emilia should be able to perform simple cognitive tasks fluently and automatically. These tasks include finishing tests and exams in the time given, and completing classroom assignments on time.”
  • Better: “Emilia is able to complete her school work quickly and without any problems.” 

Another slight modification to typical psychological reports that can help bolster their application and relevance to intervention with individuals who are from diverse linguistic and cultural backgrounds can be made in the manner in which the results themselves are described. Typically, results and scores are presented in terms of a categorical classification (see Table 1) as opposed to a functional classification (Table 2). Note that a categorical system often implies that the individual in that category is only as capable as the classification implies — as if the measured level of ability were only innate and not necessarily circumstantial. In contrast, a functional classification system provides a clearer idea regarding instructional level and promotes a more positive view of ability and development.

Table 1. Typical Classifications (Categorical)

Category Description

Standard Score

Percentile Rank

Superior          

130+

98th +

High average

110−129

76th − 97th

Average

90−109

25th − 75th

Low average 

80−89

9th − 24th

Below average

70−79

2nd − 8th

Extremely below average

< 70

< 2nd

Table 2. Better Classifications (Functional)

Functional Category Description

Standard Score

Percentile Rank

Highly proficient — independent functioning, rarely needs help          

111+

76th +

Proficient — consistent functioning, occasionally needs help

90−110

25th − 75th

Emergent — inconsistent functioning, often needs help

80−89

9th − 24th

Problematic — difficulty in functioning, always needs help 

80−89

9th − 24th

Below average

70−79

2nd − 8th

Extremely below average

< 79

<8th

Use of the latter system retains significantly more relevance for instruction and intervention than the former, irrespective of whether the individual is identified as having a disability. In fact, it is possible to improve reports and evaluations further by adopting not only a functional, “plain English” approach to describing the results, but also by keeping the report and evaluation focused on intervention. The purpose of any evaluation, particularly those conducted for diagnostic purposes, is to provide information that assists with efforts to intervene and remediate. This is particularly true for school-based evaluations, especially in cases where the identified deficits are rooted in linguistic/cultural differences that are evidence of systemic and environmental effects, as opposed to being intrinsic markers of lack of ability. Because evaluation and intervention should always be linked, a useful report should provide information and data in a manner that directly informs intervention, as illustrated in the example below: 

  • Typical: “The multidisciplinary team should review this report and integrate the findings and recommendations into the educational program,” or, “It is recommended that these results should be considered by the Committee on Special Education.”
  • Better: “Emilia has adequate capacity for school-based learning; however, her proficiency in English is only at the intermediate level. This suggests that classroom instruction needs to be sheltered sufficiently so that she is able to fully comprehend her school assignments.” 

In summary, evaluations conducted on individuals who are linguistically and culturally diverse require attention to variables that affect both the process of the evaluation as well as the manner in which the evaluation is documented and reported. Perhaps because of lack of training, education and expertise, or their sheer complexity and daunting nature, cultural and linguistic influences are often minimized, dismissed or ignored in both respects. Yet, even seemingly minor cultural and linguistic differences in developmental experience are sufficient grounds to warrant a comprehensive, nondiscriminatory evaluation with full attention to all the necessary components of an appropriate report. Ultimately, an effective report about multicultural individuals will depend on the implementation of an effective evaluation. Without such a foundation, an appropriate report cannot be written. In the end, describing the “determination” regarding the validity of assessment data with true discretion will prove far superior to issuing any hollow echoes of false caution. 

References 

American Psychological Association (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, 1060-1073. 

American Psychological Association (1990). Guidelines for Providers of Psychological Services to Ethnic, Linguistic, and Culturally Diverse Populations. Washington DC: American Psychological Association. 

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2015). Standards for Educational and Psychological Testing. Washington, D.C.: Author. 

Figueroa, R. & Newsome, P. (2006). The Diagnosis of LD in English Learners: Is It Nondiscriminatory? Journal of Learning Disabilities, 39, 206-214. 

Individuals with Disabilities Education Act, 20 U.S.C. § 1400 (2004).

Ortiz, S.O. (2014). Best Practices in Nondiscriminatory Assessment. In P. Harrison & A. Thomas (Eds.) Best Practices in School Psychology VI: Foundations (pp. 61-74), Bethesda, Maryland: National Association of School Psychologists. 

U.S. Census Bureau (2015). Current Population Survey, released 2015. Available at http://www.census.gov/cps/. Last retrieved, Oct. 22, 2016.