In my dissertation, I examined novel methods for leveraging artificial intelligence (AI), natural language processing (NLP), machine learning (ML), and automation to systematically glean personality-related information from textual data (which offers rich information and reflects various aspects of personality but has been severely underutilized). In two studies, I connected the five-factor (or Big Five) model with NLP from two angles: 1) a construct validity perspective (i.e., the degree to which information extracted from textual data reflects personality constructs), and 2) an applicability perspective (i.e., the ability to elicit personality-relevant information from text in line with psychometric principles).
Study 1: A meta-analytic review on the psychometric validity evidence
Study 1 of my dissertation is a meta-analytical examination of the construct validity of language-based ML personality assessment for predicting the Big Five trait scores. I focused on the ML approaches to language-based personality assessment, where computers analyze texts about or from people and derive estimates of their personality traits; therefore, studies where humans performed coding/judgment on language data to assess personality were not considered.
To locate primary studies, I conducted a comprehensive literature search on various psychology and broader discipline databases by pairing two groups of keywords: a group relating to different analytic approaches and another relating to personality. To be included in the meta-analysis, a study must: (1) utilize text (e.g., excluded if only using log data), (2) use computer-automated data processing, (3) link language/text data to Big Five personality traits, (4) assess at least one of the Big Five, (5) include information about the accuracy of predicting the personality trait(s), (6) include necessary information to compute an effect size (convertible to Pearson’s r), (7) predict (target) one or more Big Five personality traits assessed using traditional established approaches, and (8) not rely on non-independent data (e.g., excluded if using overlapping samples). As a result, 26 primary studies were identified, with 40 independent sub-studies. Study 1 results showed that the convergent validity evidence for the Big Five in the form of meta-analytic Pearson’s r was .34 on average (.29 for agreeableness, .36 for conscientiousness, .34 for extroversion, .30 for neuroticism, and .40 for openness), indicating a small to moderate convergent relationship between traditionally-assessed and language-based ML-assessed Big Five.
I also examined the moderation effects of text data sources, traditional personality score sources, whether only text information was involved, and whether a closed- or open-vocabulary NLP approach was adopted, respectively, on the meta-analytic convergent validity evidence. Results showed that 1) text data source did not contribute to the convergence differences, 2) traditional personality score source contributed significantly to the convergence differences for the agreeableness, conscientiousness, and extroversion dimensions, 3) whether only text data was involved contributed significantly to the convergence differences for the openness dimension only, and 4) whether an open- or closed-vocabulary NLP approach was adopted contributed significantly to the convergence differences for all dimensions but agreeableness.
I also reviewed the primary studies and examined evidence for reliability, discriminant validity, factorial structure, and criterion-related validity: out of the 40 studies, 1) only three addressed measurement reliability and yielded diverging findings; 2) only three presented discriminant validity information, and existing language-based ML personality assessments tended to have relatively poor discriminant validity evidence; 3) virtually all treated the Big Five personality prediction as single-objective prediction tasks; and 4) only four reported criterion-related validity information and the predictive relationships were significant.
Considering that the established meta-analytic correlation between self- and other-rated Big Five scores is .36, I conclude that there is some convergent validity evidence for language-based ML personality assessment if the machine is considered as an observer. This study also revealed a great lack of consideration of the broader construct validity of the language-based ML personality assessments. Additionally, considerable disconnections were identified in literature from the broader fields focusing on technology and the narrower fields involving psychology.
Study 2: Development and validation of an AI personality assessment chatbot
In Study 2, I empirically examined the effectiveness of a newly designed AI chatbot aimed to improve the validity of language-based ML personality assessment. The AI chatbot in this study was designed to conduct interactive text-based conversations in a human-like and relational manner. The conversations are interview-like, following a workflow to elicit information on personality-relevant topics; Additionally, the prompting procedure resembles a rudimentary adaptive test (i.e., a step-by-step word-count-based evaluation of what next questions to ask for eliciting as much target trait-relevant information as possible).
To maintain conversations and assess personality, the chatbot is powered by two core engines: a conversation engine and a personality engine. The conversation engine engages user in a mixed-initiative conversation: Similar to a human interviewer, the chatbot guides the conversation flow and elicits information from users, while also allowing the user to pose questions (e.g., for clarification or process inquiries). The personality engine analyzes users’ input and automatically infers their personality characteristics.
From the conversation, the AI chatbot transforms participants’ inputs into personality scores, and this automatic process can be conceptually understood as three steps. Step 1 is to map the trait-relevant text. Upon receiving participants’ input, the chatbot processes the text and matches the input into pre-determined feature categories and dimensions using pre-trained models. These pre-trained models include state-of-the-art NLP algorithms (i.e., universal sentence encoders, USE) and their subsequently Siamese Network fine-tuned variants (to specifically serve this study). Step 1 results in one 512-dimensional vector for each user’s input. Step 2 is to mine the text-trait relationships. An archival sample (N = 1,037 interacted freely with a generic chatbot and completed Big Five inventories) was used to train the personality prediction algorithms (from the 512-dimensional representations), where the elastic net was chosen as the predictive model. Hyperparameter tuning was performed, and optimal combinations were selected via 5-fold cross validation (separately for each personality dimension). Personality scores predicted from active (i.e., retained from elastic-net-based variable selection) USE-encoded features can be considered psychometric items for the Big Five, as they contribute to the personality dimension scorings. Therefore, a psychometric model can be applied. Continuous multidimensional item response theory (IRT) models were then adopted for item and person parameter estimations. Step 3 was the evidence-based inference of trait scores. Following the IRT models, users’ Big Five latent trait scores are computed. For this validation, participants’ scores were compared with scores calculated from the traditional personality scales to examine the construct validity and criterion-related validity evidence.
To examine the validity of the AI chatbot system for personality assessment, the current study had four components: 1) conversation with the chatbot, 2) traditional personality scales (self- and observer-ratings), 3) external and criterion variable measures, and 4) feedback. Construct validity evidence was analyzed following a multi-trait multi-method (MTMM) analytical approach. Criterion-related validity evidence using the personality scores (calculated from the self-reports, observer ratings, and the chatbot) to predict outcomes was also examined. All convergent correlations among the self-, observer-, and machine-scored personality were positive and significant, averaging .32 across the five traits for the self-machine convergence, .36 for self-observer, and .37 for machine-observer. The discriminant correlations among the five dimensions were on average .25 for the self-report, .18 for observer-report, and .29 for machine-inference. Additionally, the patterns of predicting outcomes from machine-inferred personality scores were more similar to that from observer-ratings than from self-reports. Good reliability evidence (i.e., alphas, inter-rater, marginal) for self-, observer-, and machine-reports was also observed. Overall, results suggested that the chatbot system shows adequate construct and criterion-related validities, on par with past findings.