Harvard statistics professor Joseph Blitzstein, commenting on the exquisite simplicity of Bayes’ Theorem, describes it as having “incredibly deep consequences and to be applicable to just about every field of inquiry.” (Roberts, 2020) In spite of this vaunted profundity, simplicity, and general applicability, Bayes’ Theorem in its most basic form has not found favor in the construction of instruments for psychometric assessment. Although a good deal has been written about Bayesian psychometric modeling in comparison to classical test theory and item response theory approaches (Levy & Mislevy, 2016), I am not aware of any published test that explicitly incorporates Bayesian techniques in its development.
Empirical keying of clinical test items was at one time a favored strategy for test development. It dates back to the 1930s with the item development phase of the original MMPI (Hathaway & McKinley, 1942). In their development of scales for this comprehensive clinical personality measure, Hathaway and McKinley followed the brilliant strategy of requiring a significant correlation between a test item and membership in the group of clinical subjects corresponding the scale to which that item would be assigned. While this doubtless produced powerful sets of items that individually predicted the construct represented by their particular scale, empirical keying of items created a dilemma. Although the individual items within scales met the requirements of significant correlation with membership in the clinical group, these items did not correlate very well with each other within scales, resulting in deficits in internal consistency.
The hallmarks of a clinical measure to predict diagnoses or behavioral characteristics are empirical validity and absence of subjectivity in determining scores. Perhaps the most notable example of an instrument that runs counter to these ideals is the House-Tree-Person (H-T-P) projective technique. Interpretation of this technique requires reference to elements of the case history and clinical theory, usually psychoanalytic theory, as well as subjective interpretation of the relationship of each indication from the subject’s drawings and associations. This “rational/clinical” interpretive model for drawings methods is required study at about 60% of psychology graduate programs geared toward training practitioners.
The subjective approach to drawings methods has had a rather unfortunate history. Perhaps the most dramatic illustration of the failure of empirical research to validate projective use of drawings is a series of investigations of human figure drawing interpretive hypotheses developed by psychologist Karen Machover (1949). These clinically derived hypotheses were empirically tested against relevant criteria. Out of the 30 interpretive hypotheses tested, only two were statistically significant (Kahill, 1984). Thus, while the clinical theory may be valid with respect to general therapeutic and diagnostic questions, purely theoretical interpretations of diagnostic drawing signs proved useless.
I have developed an adaptation of the H-T-P that requires “blind” interpretation, i.e., without reference to the case details or theory, employs empirically keyed items generated through meticulous criterion development, and includes Bayesian weights for individual items as well as a Bayesian procedure for aggregating individual items that relate to the same criteria. The procedure possesses adequate interscorer reliability and is supported by empirical validity studies as well as a parallel Bayesian validity measure through the empirically assigned weights.
It should be noted that, unlike self-report clinical personality inventories, the H-T-P does not entail the presentation of a standard set of stimuli to all subjects. Even in computerized adaptive testing under an item response theory model, subjects are exposed to a subset of standard stimuli depending on their performance on prior items. In the H-T-P, the content is entirely left to the subject. While the test items of the present procedure reflect whether specific test indicators are present or absent from the protocol, subjects are not responding to a set or subset of standard stimuli, but are left entirely to their own devices. This characteristic of the procedure requires a probabilistic psychometric model as a basis for valid interpretation.
The development phase began with a sample of 250 forensic psychological evaluation cases in which the test subject’s files were reviewed for evidence of criterion conditions such as Antisocial Personality Disorder, Substance Abuse Disorder, and Depressive Disorder, as well as behavioral characteristics such as history of assaultive behavior, psychiatric hospitalization, more than three arrests, and having served a sentence in county jails or state prisons. Potential test items were culled from books and manuals on the H-T-P and notes from assessment presentations by recognized experts in the technique.
A correlation matrix comprising approximately 200 trial items and 16 criterion categories, based on the 250 subjects’ file reviews, yielded a number of significant relationships between individual predictor items and clinical conditions or behavioral characteristics. This was followed by a cross-validation study conducted on a sample of 150 new forensic psychological evaluation cases that demonstrated convergent and discriminant validity for unweighted individual test items. In this cross-validation study, interscorer agreement assessed by Fleiss’ Kappa using three experienced clinical and forensic psychologists as raters was adequate (Dyer, 2018).
The next phase of development involved the computation of Bayesian posterior probabilities for each test item against each of the criteria with which it had a significant relationship. This analysis was conducted on a combined sample using some new cases and the original item development and cross validation samples. The percentage difference between the prior probability and Bayesian posterior probability for each of the item-criterion combinations provided the basis for the development of item weights.
For each individual item-criterion combination, the percentage difference between the prior probability and posterior probability was converted to a scale in which each seven-point increase in the likelihood of the posterior probability over the prior probability was assigned one scale point. This blunting was for the purpose of avoiding overly precise interpretations of the results, in other words not attempting to interpret differences of one or two percentage points in drawing conclusions about subjects.
Being that each Bayesian posterior probability may be used as the prior probability for assessment of the impact of another variable on the presence of the criterion condition, these seven-percentage point scores are simply added across items for each item-criterion combination to yield a total score for that criterion.
These score intervals are interpreted in the following manner:
Not to be interpreted individually, but can be added to other scores for the same variable
Mildly suggestive of the condition
Suggestive of the condition
Strongly suggestive of the condition
Indicative of the condition
Strongly indicative of the presence of the condition
Very strongly indicative of the presence of condition
This psychometric model is exactly the opposite requirement of classical test theory for the composition of a scale. Whereas extremely low item intercorrelations are a negative characteristic for classical test theory scales, a Bayesian format actually requires predictors of the same criterion to be at most weakly correlated with one another. This is because the use of highly correlated predictors for the same variable is redundant and produces artificially inflated relationships.
The scoring system is not intended to yield a precise estimate of the posterior probability of the condition. These scores provide an estimate of the increase in the likelihood of the presence of the criterion given the positive items for the criterion in the test record. The recommended interpretive language for each score level is intended to be appropriate for any base rate of the target condition in the sample of cases assessed by the test user.
A logistic regression study of weighted and unweighted scores using a new sample of 179 subjects found that weighted scores significantly predicted the majority of criteria studied. Weighted scores were found to be somewhat superior to unweighted scores as predictors.
The development technique presented above employed the tasks of a theory-dependent, purely subjectively interpreted projective technique to develop an empirically valid and reliable psychometric instrument using empirical item keying with an underlying Bayesian probabilistic model.
References
Dyer, F. J. (2018). Prediction of Psychiatric Hospitalization, Diagnoses, Arrests, and Violent Behavior Through Scored Drawings and Associations. Psychological Reports, 121(1), 4–25.
Hathaway, S. R., & McKinley, J. C. (1942). Minnesota Multiphasic Personality Inventory Manual. University of Minnesota Press.
Kahill, S. (1984). Human figure drawing in adults: An update of the empirical evidence, 1967–1982. Canadian Psychology, 25, 269–292.
Levy, R. & Mislevy, R. (2016). Bayesian psychometric modeling. Boca Raton London New York: CEC Press.
Machover, K. (1949). Personality projection in the drawing of the human figure. Springfield, IL.: Charles C. Thomas.
Roberts, S. (2020). Thinking like an epidemiologist. The New York Times p. D8, Aug. 4, 2020.
Contact
Frank J. Dyer, PhD, ABPP
Psychologist in Independent Practice
Email