In this issue

Examining the correspondence between a direct and an indirect measure of executive functions: Implications for school-based assessment

Multiple measures of what seems to be the same construct may not be.

By D. Jake Follmer and Candice R. Stefanou

In recent years, there has been increasing interest in executive functions as they relate to learning, behavioral, and emotional control. Seen as increasingly important in our understanding of adaptive mechanisms of learning, the incorporation of executive functioning has demonstrated utility and value in shaping assessment practices, informing assessment decisions, and tailoring interventions (Meltzer, Pollica, & Barzillai, 2007). Described by Garner (2009, p. 406) as a set of “goal-directed neurocognitive processes that allow for the control and coordination of cognition and behavior,” the impact and involvement of executive functions in regulating behavior is extensive. Executive functions are described as having particular influence in setting goals, executing well-planned, organized behavior, maintaining cognitive flexibility, and inhibiting responses that are inappropriate or maladaptive (Garner, 2009; McAuley, Chen, Goos, Schachar, & Crosbie, 2010), all of which have significant impact on success in school settings.

There are two dominant ways to assess executive functions in schools today – direct measures and indirect measures. Two of the more widely used measures of executive functions are the Delis-Kaplan Executive Function System (D-KEFS; Delis, Kaplan, & Kramer, 2001a) and the Behavior Rating Inventory of Executive Function (BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2004). The D-KEFS reflects an individualized task-based or direct assessment of executive functions, whereas the BRIEF-SR reflects a rating scale or indirect assessment of self-reported executive functioning. In school-based assessments, where adherence to regulations with regard to timely assessment is crucial, indirect methods, such as rating scales, are often used as a means of determining relative dysfunction especially with regard to behavioral, emotional, and executive functioning. In some instances, where specific information with regard to cognitive or attentional functioning is desired, the practitioner makes the decision to use a more time intensive, direct measure, such as the D-KEFS.

There are competing viewpoints on the appropriateness of direct and indirect assessment methods of executive functions. Barkley (2012), for instance, recommends the use of indirect formats, comprising rating scales of executive functioning, noting that they can be widely used and are able to more accurately predict executive dysfunction or impairment. Despite research examining assessments of executive functions from either a direct or indirect format, little research exists in the extant literature evaluating the correspondence between these types of assessments. Such research could provide insight into the types of information each assessment provides. The current study examined the correspondence between a direct and an indirect measure of executive functions (Anderson, V., Anderson, P., Northam, Jacobs, & Mikiewicz, 2002; Vriezen & Pigott, 2002).

Method

Participants

The sample of convenience consisted of 30 participants from two liberal arts colleges and one high school in the North-central Pennsylvania area with a chronological age ranging from 18 years, 0 months to 18 years, 11 months. Of the sample obtained, eight participants were male and 22 were female. With regard to ethnicity classification, the sample was composed primarily of White/Caucasian individuals ( n = 25), with three individuals being classified as African-American and one participant being classified as Asian. Institutional Review Board (IRB) approval was obtained from each institution in which data were collected.

Instruments

Delis-Kaplan Executive Function System (D-KEFS; Delis, Kaplan, & Kramer, 2001a). The Delis-Kaplan Executive Function System (D-KEFS) was utilized as a direct measure of executive functions. Consisting of nine stand-alone tests, the D-KEFS reflects the first comprehensive assessment of executive functions constructed in a task-based or direct measurement format. The D-KEFS utilizes scaled scores. With regard to score interpretation, the higher the scaled score obtained for the selected tests or conditions administered, the better the performance on the specific executive function task measured (Delis et al., 2001a). The following D-KEFS tests were administered to all participants: the Verbal Fluency Test, the Design Fluency Test, the Color-Word Interference Test, and the Tower Test. Evidence supporting the validity of the D-KEFS has been noted in the technical manual as well as in other studies (Baldo, Shimamura, Delis, Kramer, & Kaplan, 2001; Baron, 2004; Homack, Lee, & Riccio, 2005; as cited in Delis et al., 2001b).

Table 1

Executive Functions Measured By the D-KEFS and BRIEF-SR

Executive Functions

D-KEFS Measures

BRIEF-SR Measures

Inhibition: ability to stop, modulate behavior, and demonstrate control of one's impulses (Best & Miller, 2010; Guy et al., 2004a)

Color-Word Interference Test; Verbal Fluency Test; Design Fluency Test; Tower Test

Inhibit Clinical Scale; Behavioral Regulation Index (BRI)

Cognitive Flexibility: ability to shift freely/switch between mental sets, tasks, or rules (Best & Miller, 2010; Guy et al., 2004a)

Color-Word Interference Test; Verbal Fluency Test; Design Fluency Test; Tower Test

Shift Clinical Scale; Cognitive Shift Clinical Subscale; Behavioral Regulation Index (BRI)

Planning Ability: ability to utilize goals, instruction, and feedback to regulate behavior; effectively begin and manage future-oriented tasks (Best & Miller, 2010; Guy et al., 2004a)

Tower Test

Plan/Organize Clinical Scale; Metacognition Index (MI)

Monitor: ability to monitor and assess performance (Best & Miller, 2010; Guy et al., 2004a)

Tower Test; Color-Word Interference Test

Monitor Clinical Scale; Metacognition Index (MI)

 

Behavior Rating Inventory of Executive Function (BRIEF); Gioia, Isquith, Guy, & Kenworthy, 2004; Guy, Isquith, & Gioia, 2004) . The Behavior Rating Inventory of Executive Function – Self-Report Version (BRIEF-SR) was administered as an indirect measure of executive functions. The BRIEF-SR consists of 80 items constructed on a 3-point scale that assess executive functioning as rated by the participant. With regard to score interpretation, higher T -scores indicate greater degrees of executive dysfunction, while lower T -scores indicate acceptable executive functioning. The BRIEF-SR clinical scales, subscales and indices utilized for all participants included: the Inhibit Clinical Scale, the Shift Clinical Scale, the Plan/Organize Clinical Scale, the Monitor Clinical Scale, the Cognitive Shift Clinical Subscale, the Behavioral Regulation Index (BRI), the Metacognition Index (MI), and the Global Executive Composite (GEC). Evidence supporting the validity of the BRIEF-SR has been noted in the technical manual as well as in other studies (Gioia et al., 2004; Guy et al., 2004; McAuley et al., 2010).

Procedure

Scales, subscales, and indices on the BRIEF-SR purporting to measure inhibition, cognitive flexibility, planning/organizing ability, and monitoring ability as well as summary measures of executive functions were selected along with direct measures on the D-KEFS purporting to measure the same executive functions (Baldo, Shimamura, Delis, Kramer, & Kaplan, 2001; Baron, 2004; Delis et al., 2001a; Gioia et al., 2004; Homack, Lee, & Riccio, 2005).

For example, tests included from the D-KEFS purporting to measure inhibition and inhibitory functioning (e.g., Color-Word Interference Test; Design Fluency Test) were correlated with the scale from the BRIEF-SR also purporting to measure inhibition and the index including inhibition (e.g., Inhibit; Behavioral Regulation Index). As another example, the test included from the D-KEFS purporting to measure planning ability (i.e., Tower Test) was correlated with the scale from the BRIEF-SR also purporting to measure planning ability as well as the index including planning ability (e.g., Plan/Organize; Metacognition Index). Summary measures of executive functions, comprising a measure of behavioral regulation, metacognition, and global executive functioning, were included in the analyses based on the measures including the selected executive functions within each index.

Participants were administered both assessments of executive functioning utilizing a counterbalanced administration procedure to control for potential priming effects of either instrument. Participants were placed into two groups in an alternating fashion based upon the coding of assessment forms. In the first group (Group A), participants were administered the D-KEFS first and the BRIEF-SR second; in the second group (Group B), participants were administered the BRIEF-SR first and the D-KEFS second.

Analyses

Pearson product-moment correlation coefficients were computed between scaled scores obtained from the selected D-KEFS (Delis et al., 2001a) tests and the scale and index T- scores obtained from the BRIEF-SR (Guy et al., 2004). Negative correlation coefficients were expected between the selected tests, scales, and indices of the two measures based upon differences in the scoring scales between the measures.

Results

Separate analyses taking into account order of administration were conducted. No significant differences were obtained in the Pearson correlation coefficients obtained between administration groups, indicating no significant effects due to order of administration.

It was expected that significant negative correlations would be obtained between the D-KEFS Design Fluency Test and the BRIEF-SR Inhibit Scale, Shift Scale, Cognitive Shift Subscale, Behavioral Regulation Index (BRI), and Global Executive Composite (GEC). The BRIEF-SR Shift Scale ( r = -.37; p < .05; r ² = .14), Cognitive Shift Subscale ( r = -.35; p < .05; r ² = .12), BRI ( r = -.38; p < .05; r ² = .14), and GEC ( r = -.32; p < .05; r ² = .10) were significantly correlated with the D-KEFS Design Fluency Test: Filled Dots Condition. The BRIEF-SR Shift Scale ( r = -.32; p < .05; r ² = .10) and BRI ( r = -.35; p < .05; r ² = .12) were significantly correlated with the D-KEFS Design Fluency Test: Total Correct Condition.

It was expected that significant negative correlations would be obtained between the D-KEFS Color-Word Interference Test and the BRIEF-SR Inhibit Scale, Shift Scale, Monitor Scale, Cognitive Shift Subscale, Behavioral Regulation Index (BRI), Metacognition Index (MI) and the Global Executive Composite (GEC). The BRIEF-SR MI ( r = -.33; p < .05; r ² = .11) and GEC ( r = -.31; p < .05; r ² = .10) were significantly correlated with the D-KEFS Color-Word Interference Test: Color Naming Condition. The BRIEF-SR Monitor Scale ( r = -.39; p < .05; r ² = .15), Cognitive Shift Subscale ( r = -.33; p < .05; r ² = .11), MI ( r = -.41; p < .05; r ² = .17), and GEC ( r = -.39; p < .05; r ² = .15) were significantly correlated with the D-KEFS Color-Word Interference Test: Word Reading Condition. The BRIEF-SR Inhibit Scale ( r = -.37, p < .05; r ² = .14), Monitor Scale ( r = -.39; p < .05; r ² = .15), BRI ( r = -.39, p < .05; r ² = .15), MI ( r = -.35; p < .05; r ² = .13) and GEC ( r = -.42, p < .01; r ² = .18) were significantly correlated with the D-KEFS Color-Word Interference Test: Inhibition/Switching Condition.

It was also expected that significant negative correlations would be obtained between the D-KEFS Tower Test and the BRIEF-SR Inhibit Scale, Shift Scale, Plan/Organize Scale, Monitor Scale, Cognitive Shift Subscale, Metacognition Index (MI), and Global Executive Composite (GEC). The BRIEF-SR Shift Scale ( r = -.34; p < .05; r ² = .12) and Cognitive Shift Subscale ( r = -.32; p < .05; r ² = .10) were significantly correlated with the D-KEFS Tower Test: Move Accuracy Ratio.

Finally, significant negative correlations were expected between the D-KEFS Verbal Fluency Test and the BRIEF-SR Inhibit Scale, Shift Scale, Monitor Scale, Cognitive Shift Subscale, Behavioral Regulation Index (BRI), and Global Executive Composite (GEC). No significant correlations were found.

Discussion

Overall, the results indicated low correlations between the D-KEFS and the BRIEF-SR. Significant negative correlations were obtained between several D-KEFS tests and BRIEF-SR scales and indices, providing some evidence of similar measurement of executive functions (e.g., cognitive flexibility) between the BRIEF-SR and the D-KEFS. However, there were also considerable non-significant findings. An analysis of shared variance between the correlation coefficients obtained reveals little overlap between the measures, with shared variances ranging from 10 percent to 18 percent among significant correlations.

The findings suggest that direct and indirect measures of executive functions may provide unique information based upon the specific type of measure utilized. Perhaps there is a difference in the way executive dysfunction manifests itself in the everyday work of the classroom – whose description might be most accessible by those who work with the child with the dysfunction – from the way dysfunction might manifest itself in the formal tasks that tap the underlying processes of executive functions. For instance, a lack of agreement among the instruments in measuring inhibition may stem from differences in the manifestation of the ability to inhibit behavioral impulses that contribute to behavioral regulation, compared with the ability to inhibit cognitive processes that contribute to efficient cognitive processing (Delis et al., 2001a; Guy et al., 2004).

Because direct and indirect measures of executive functions appear to provide different estimates in several areas, as shown by this study and others focused on clinical samples (Anderson et al., 2002; Vriezen & Pigott, 2002), and because there are some who feel that rating scales more accurately predict executive dysfunction (Barkley, 2012), practitioners might be well-advised to consider exactly what information is provided by the two measures. The indirect measure might help to provide insight into how disruptive a child's inability to engage and utilize a given executive function is in the environment; the direct measure might provide insight into what cognitive processes are particularly affected so that interventions can be developed that more accurately address the child's needs. Thus, the important question might not be one of congruence; instead, it might be one of complementarity.

Several limitations with implications for generalizability are noted. Data were obtained from thirty participants, resulting in a limited sample size from which to derive correlational data and conclusions. Further, restricting the sample to those individuals eighteen years of age introduces the possibility that the sample obtained reflected one that is truncated and more homogenous. A larger and more age-heterogeneous sample size might have yielded more support for generalizability of findings. The difference in measurement format between the assessments is also an important consideration in evaluating the complementarity of the information provided by the measures.

These results highlight both the complexity and the dimensionality of measuring executive functions as well as the need to consider the specific information obtained from each type of instrument. It may be that the information provided by rating scales emphasizes a pragmatic assessment of executive functions, whereas the information provided by direct measures emphasizes cognitive processing and related neuropsychological information. The issue then might be to consider that the two types of data provide unique information that helps with educational programming and intervention monitoring. The practitioner's awareness and knowledge of the specific types of information an instrument provides, as well as a corresponding understanding of the educational implications that stem from such information, is instrumental in tailoring instructional and intervention practices.

References

Anderson, V., Anderson, P. J., Northam, E., Jacobs, R., & Mikiewicz, O. (2002). Relationships between cognitive and behavioral measures of executive function in children with brain disease. Child Neuropsychology, 8 (4), 231-240.

Baldo, J. V., Shimamura, A. P., Delis, D. C., Kramer, J., & Kaplan, E. (2001). Verbal and design fluency in patients with frontal lobe lesions. Journal of the International Neuropsychological Society, 7 (5), 586-596.

Barkley, R. A. (2012). Barkley Deficits in Executive Functioning Scale – Children and Adolescents. New York: NY: The Guilford Press.

Baron, I. S. (2004). Test review: Delis-Kaplan executive function system. Child Neuropsychology, 10 (2), 147-152.

Delis, D. C., Kaplan, E., & Kramer, J. H. (2001a). The Delis-Kaplan Executive Function System: Examiner's Manual. San Antonio: The Psychological Corporation.

Delis, D. C., Kaplan, E., & Kramer, J. H. (2001b). The Delis-Kaplan Executive Function System: Technical Manual. San Antonio: The Psychological Corporation.

Garner, J. K. (2009). Conceptualizing the relations between executive functions and self- regulated learning. The Journal of Psychology: Interdisciplinary and Applied, 143 (4), 405-426.

Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2004). The Behavior Rating Inventory of Executive Function. Lutz, FL: Psychological Association Resources.

Guy, S. C., Isquith, P. K., & Gioia, G. A. (2004). The Behavior Rating Inventory of Executive Function – Self-Report Version: Professional Manual. Lutz, FL: Psychological Association Resources.

Homack, S., Lee, D., & Riccio, C. A. (2005). Test review: Delis-Kaplan executive function system. Journal of Clinical and Experimental Neuropsychology, 27 (1), 599-609.

McAuley, T., Chen, S., Goos, L., Schachar, R., & Crosbie, J. (2010). Is the behavior rating inventory of executive function more strongly associated with measures of impairment or executive function? Journal of the International Neuropsychological Society, 16 (1), 495-505.

Meltzer, L., Pollica, L. S., & Barzillai, M. (2007). Executive function in the classroom: Embedding strategy instruction into daily teaching practices. In L. Meltzer (Ed.), Executive Function in Education (5-18). New York, NY: The Guilford Press.