From Lexicons to Large Language Models: A Holistic Evaluation of Psychometric Text Analysis in Social Science Research

1. Introduction

Online platforms have greatly expanded the volume and variety of text data available to social science researchers, offering a direct and scalable view into people’s attitudes, perceptions, and traits (Ahmad et al. 2020). An important approach for distilling such psychological attributes from text is psychometric natural language processing (NLP), an umbrella term for the measurement of such latent constructs (Abbasi et al. 2018, Ahmad et al. 2020, Mousavi and Gu 2024). These techniques have proven indispensable across social science domains, from modeling how personality traits manifest in social media trace data (Yang et al. 2023), to using word-embedding trajectories in news to reveal evolving weight stigma (Arseniev-Koehler and Foster 2022), to linking affective language in Supreme Court exchanges to case outcomes (Black et al. 2011).

Modern psychometric NLP has evolved to enable understanding of nuanced, context-dependent aspects of language that earlier lexicon-based approaches may miss. However, despite significant progress fueled by machine learning (ML) and artificial intelligence (AI) over the past two decades, social science researchers still face three core challenges when measuring psychological constructs from text. Domain expertise gaps can limit the creation of labeled data sets tailored to highly specific topics. NLP expertise gaps may hamper the deployment of advanced or fine-tuned deep learning techniques capable of nuanced analysis. Identification of constructs for study may suffer from inaccuracies inherent in simpler methods or a lack in proficiencies required to effectively implement more robust NLP techniques. For social science researchers, overcoming these barriers is critical to realizing the promise of large-scale text data for understanding human psychology and behavior.

More recently, large language models (LLMs) show promise for resolving these barriers by offering high predictive performance alongside reduced demands for labeled data or specialized AI skills. Recognizing the importance of NLP methods in social science research, we conduct a holistic evaluation of LLMs against existing psychometric NLP paradigms: (1) lexicon-based methods, (2) custom-built models, and (3) fine-tuned masked language models (FLMs). Our holistic evaluation across four distinct data sets reveals that LLMs can meet or surpass other paradigms, including FLMs, in classifying psychological constructs along multiple dimensions, while requiring fewer resources and less technological expertise or domain-specific knowledge. This represents a significant contribution to the NLP literature by demonstrating a practical path to democratize access to state-of-the-art text analysis. More broadly, it signals a potential paradigm shift for social science research, enabling more ambitious and nuanced textual inquiry at scale without prohibitive technical overhead.

These advancements are especially relevant for coding constructs such as resilience, empathy, and sentiment polarity (hereafter, polarity), each of which demands distinct levels of cognitive and emotional engagement (Pennebaker and Francis 1996). Leveraging dual-process theory (Kahneman and Frederick 2002) to deepen our holistic evaluation, we demonstrate that cognitive and affective capacities of nonexperts substantially influence their ability to code these constructs in text, as well as performance of downstream models trained on these annotations. Our theoretically informed findings illustrate risks inherent in existing paradigms reliant on human annotation and potential avenues for improving the performance of LLMs by emulating enhanced cognitive and affective abilities through “cognitive-affective prompting.” This approach addresses the inconsistencies observed in the emerging research on persona-based prompting (see Tseng et al. 2024) and extends this body of work by introducing a generalizable practice for improving LLMs’ performance.

This study contributes to the literature on psychometric NLP by developing a theory-guided framework for holistic evaluation and applying it to demonstrate that LLMs exhibit comparable or superior predictive performance, consistency, and fairness relative to established approaches, while reducing demands for expertise and resources. We examine the cognitive and affective processes underlying the identification of psychological constructs and their effects on downstream models, as well as demonstrate how cognitive-affective prompting enables LLMs to emulate traits aligned with psychometric NLP tasks to improve performance. Salient design insights from our novel framework provide practical guidance for method selection and deployment, as well as support for emerging computationally intensive theory construction research (Miranda et al. 2022) and design of psychometric NLP-based artifacts (Abbasi et al. 2024), underscoring the promise of LLMs to enhance measurement reliability and uncover deeper psychological dynamics from digital text.

2. Background—NLP Methods for Identifying Psychological Constructs

2.1. Psychometric NLP Methods

Text classification is a foundational NLP task in which text is categorized into predetermined classes (e.g., “Empathetic” or “Nonempathetic”), often with assigned confidence scores, which may be dichotomized into labels or be used directly for subsequent analyses. These methods may be deployed for a variety of applications, although a prevalent use case, and the focus of our study, is the use of NLP to identify psychological constructs. These methods fall into four primary categories, each with distinct strengths and limitations.

Table 1 provides an overview of the landscape. Lexicon-based methods are the simplest and least resource-intensive to apply, requiring little domain or NLP expertise. Using predefined lists of words or phrases, researchers can aggregate scores based on term frequency to infer psychological constructs. Tools like Linguistic Inquiry and Word Count (LIWC) (Tausczik and Pennebaker 2010) and WordNet (Miller 1995) are widely used but criticized for their inability to incorporate contextual nuances. Domain-specific lexicons (e.g., Loughran and McDonald 2011) address some limitations but remain fundamentally constrained. Supervised Machine Learning models use labeled data sets to train algorithms for identifying labels in larger, unlabeled data sets. Innovations such as Recurrent Neural Networks (RNNs) (Rumelhart et al. 1986) and their derivatives such as Long Short-Term Memory networks (LSTMs) (Hochreiter and Schmidhuber 1997) enhanced the ability to handle sequential data. However, these architectures struggle with long-term dependencies and are computationally demanding. These custom-built models (CBMs) require substantial domain expertise to create a labeled training set and substantial NLP expertise to deploy.

Table 1. Psychometric NLP Paradigms for Text Classification

Transformers revolutionized NLP by introducing self-attention mechanisms that process entire text sequences simultaneously. Transformer-based models like Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. 2019) leverage Masked Language Modeling to learn bidirectional representations of text. Fine-tuned models such as FinBERT (Araci 2019) demonstrate the adaptability of this approach for domain-specific tasks. Like CBMs, FLMs require high domain expertise for the creation of labeled data sets for fine tuning and high NLP expertise to deploy.

LLMs, the latest advancement, build on transformer architecture but employ autoregressive principles. LLMs predict each token based solely on preceding tokens, generating coherent and contextually relevant text incrementally. Examples include Generative Pre-Trained Transformer (GPT) (Brown et al. 2020), the Pathways Language Model (Chowdhery et al. 2022), Llama 3 (Grattafiori et al. 2024), and Mistral (Jiang et al. 2023). LLMs, exemplified by GPT’s 175-billion-parameter model, achieve superior performance even without fine-tuning, requiring minimal or no training data. Because of their natural language interface and low-cost commercial availability, LLMs may be deployed with very little NLP expertise. Moderate levels of domain expertise may be required to effectively communicate tasks to the LLM.

The performance of LLMs may be further amplified by prompt engineering techniques, which play a crucial role in optimizing their performance across various tasks. Few-shot prompting serves as a foundational method that provides limited input-output examples within prompts to facilitate in-context learning, enabling models to generalize from these examples effectively. Chain-of-thought (CoT) prompting enhances reasoning abilities by breaking down complex problems into simpler substeps, guiding models through a logical progression (Wei et al. 2023). Contrastive CoT introduces both valid and invalid reasoning demonstrations, allowing models to learn discernment between correct and flawed reasoning patterns (Chia et al. 2023). The Tree of Thoughts extends this framework by exploring multiple reasoning paths simultaneously, akin to a tree structure, thereby enabling more effective navigation through complex problem spaces (Yao et al. 2023). Finally, Self-Refine allows models to iteratively improve their responses by self-critiquing mechanisms, enhancing output quality by refining initial responses over successive iterations (Madaan et al. 2023).

Further advances, such as soft prompting and prompt tuning, have emerged as effective strategies for enhancing model performance. This involves introducing learnable, continuous input vectors—known as soft prompts—into the model’s input to guide its behavior for specific tasks without altering the model’s core parameters (Lester et al. 2021). In addition, although fine-tuning LLMs can be a computationally expensive task, Low-Rank Adaptation, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each Transformer layer, provides an efficient way to fine-tune LLMs for downstream tasks with minimal computational overhead (Hu et al. 2021). These innovations highlight the ongoing evolution in how LLMs can be effectively utilized across domains, including psychometric NLP.

2.2. Psychometric NLP in Social Science Research

Psychometric NLP, which we define as the use of NLP methods to identify psychological constructs, has become a pivotal tool in research across diverse disciplines. Its applications span information systems (IS), business research, and social science more broadly. A comprehensive review of such work is infeasible, but to demonstrate the breadth of adoption and variety in use cases, we utilize Web of Science to systematically identify over 5,000 studies across 13 social science disciplines citing selected seminal NLP studies. Reviewing top-cited studies in each field, we note that NLP is used for a variety of purposes, but that identification of psychometric constructs is consistently a primary focus. See Online Appendix K for detailed analysis.

In IS research, psychometric NLP has been employed to examine technologies influencing communication and societal issues. For example, Zhang et al. (2024) used lexicons and transformers to identify suicidal ideation in social media; Adamopoulos et al. (2018) use psychometric NLP to show how personality traits influence the effectiveness of word-of-mouth; and Lee and Ram (2024) developed ML models to identify false information and its impact. In business research more broadly, marketing researchers utilize psychometric NLP for brand analytics, user engagement, and sales prediction (Rust et al. 2021, Xu et al. 2021). Finance researchers have largely focused on measuring polarity of news and other content related to financial markets, largely with lexicon-based methods, but have recently adopted more advanced tools, such as FinBERT (Loughran and McDonald 2011, Araci 2019, Huang et al. 2023). Management studies frequently apply psychometric NLP to analyze management communications (Gamache and McNamara 2019, Hyde et al. 2024).

In social science disciplines more broadly, psychometric NLP is utilized for a wide variety of goals. Archaeologists identify semantic characteristics of operas (Fan et al. 2023). Communication researchers analyze online dating profiles for psychometric traits related to deception (Toma and Hancock 2012). Criminologists analyze prisoners’ descriptions of depression symptoms (Willemsen et al. 2011). Education scholars identify cues of emotional and cognitive abilities, relating these with learning outcomes (Liu et al. 2022). Sociologists examine aspects of stigma and cultural views related to body weight (Arseniev-Koehler and Foster 2022).

These broad applications utilize a variety of methods, ontologies, and theoretical framings, yet share a common thread of utilizing NLP techniques to identify psychological constructs in order to understand the phenomena they research. Over the past few decades, the methods for performing psychometric NLP have evolved from lexicon-based techniques to FLMs, with researchers in various fields adopting newer methods at different rates, with some researchers increasingly favoring FLMs for their higher predictive performance, whereas others continue to consistently utilize lexicons for higher interpretability and ease of implementation.

3. Theory-Guided Framework for Holistic Evaluation of Psychometric NLP

In this section, we synthesize existing theory and research to develop a framework for holistically evaluating NLP paradigms for extracting psychological constructs from text. To guide our discussion, Figure 1 outlines this framework in three stages. First, we broadly compare and evaluate leading methods from primary NLP paradigms (lexicons, CBMs, FLMs, and LLMs) across a range of representative text classification tasks with a focus on outcomes critical for social science researchers, including predictive performance, consistency, and fairness. We then take a deeper dive to understand how cognitive and affective processes impact the inputs to NLP tools through the lens of dual-process theory. Finally, we apply this understanding to demonstrate how consideration of cognitive and affective perspectives can influence model performance.

3.1. Performance of NLP Paradigms Across Critical Dimensions

Recently, the superior performance of LLMs in few-shot learning has drawn attention in psychometric NLP tasks. For instance, Rathje et al. (2023) found that GPTs’ accuracy excels in evaluating sentiment, discrete emotions, offensiveness, and moral foundations. Similarly, Peters and Matz (2023) studied GPT accuracy in inferring Big Five personality traits from Facebook status updates using zero-shot learning. Tan et al. (2024) further studied LLM-based annotation generation, LLM-generated annotations assessment, and LLM-generated annotations utilization tasks. Although initial studies show promise in LLMs for psychometric NLP compared with existing methods, they lack a comprehensive and systematic evaluation of psychometric NLP paradigms (including lexicons, CBMs, and FLMs) across tasks representative of a spectrum of psychological constructs (which we identify below). Expanding on this body of work, we propose:

Although previous studies have focused on the raw predictive performance of LLMs, consistency in performance remains underexamined. Consistent performance across data segments is crucial to avoid bias in downstream analyses such as econometrics. Prior research has highlighted how significant downstream biased estimation may be caused by measurement error in classification tasks performed by CBMs and FLMs (Yang et al. 2018, Qiao and Huang 2021). To our knowledge, no studies have examined the performance consistency of psychometric NLP methods across data segments. However, we hypothesize that LLMs may deliver more uniform performance across various data segments for two reasons: First, with their extensive parameters and training data, they often capture a broader range of linguistic nuances (Liu et al. 2024). This extensive pretraining enables them to generalize more effectively across diverse text segments, leading to more consistent performance. These leaps in performance at extreme architectural scale have been characterized as emergent abilities (Wei et al. 2022). Second, training or fine-tuning models on specific tasks (in CBMs and FLMs) can introduce variability, especially if training data are limited or not fully representative of all possible text segments, potentially leading to inconsistent performance across segments (Althammer et al. 2023). Hence, we propose:

Another critical, yet understudied, aspect of psychometric NLP is fairness, particularly how models handle texts from diverse demographic groups (Mehrabi et al. 2022). Systematic biases could lead to skewed conclusions that disproportionately affect certain demographics. Prior research shows that biases in lexicon-based methods might originate from initial seed word selections (Antoniak and Mimno 2021), whereas FLMs often inherit biases from the labeled data sets used for training (Rozado 2020, Bhardwaj et al. 2021). The broad, diverse data sets used to train LLMs suggest they may demonstrate lower demographic biases (e.g., BERT models: 3.3 billion tokens/30,000 (30K) vocabulary (Devlin et al. 2019) versus Llama 3: 15 trillion tokens/128K vocabulary (Grattafiori et al. 2024)). Significant efforts have also been made to enhance the alignment of foundation LLMs with fairness objectives (Eloundou et al. 2024, Ji et al. 2024), which should reduce biases in NLP applications of LLMs. We propose:

3.2. Dual-Process Theory

For the past several decades, text corpora annotated with psychological constructs have been central to the field of psychometric NLP. Ground-truth labels are typically supplied by human annotators, yet their judgments, as shaped by their individual cognitive and affective abilities (Pennebaker and Francis 1996, Panksepp 2003), vary in reliability. Dual‐Process Theory (DPT) offers a parsimonious lens for understanding this variability, relating discrepancies in annotation quality to the shifting balance between autonomous, low‐effort, affectively tinged Type 1 processes and working-memory-dependent, rule-based Type 2 processes that enable cognitive decoupling (Evans and Stanovich 2013, De Neys 2018). Because Type 2 processes require cognitive decoupling, their effectiveness covaries with cognitive ability (Evans 2003), whereas the efficiency of Type 1’s autonomous appraisal tracks emotional intelligence (Fiori 2009, Suslow et al. 2022). These dual routes generate clear, testable predictions for annotation performance: tasks that hinge more on reasoning and cognitive efforts (e.g., discerning nuanced resilience narratives) should benefit disproportionately from cognitive-driven Type 2 processing, whereas tasks dominated by affective resonance (e.g., labeling polarity) should benefit more from emotional intelligence-driven Type 1 processing. Critically, although excelling in both domains might seem ideal, empirical studies show that cognitive ability and emotional intelligence are distinct constructs that rarely co-occur at exceptionally high levels (Barbey et al. 2014). This distinction arises from different neurological underpinnings and developmental pathways, and empirical studies consistently show only a weak correlation between them (Joseph and Newman 2010, Khemlani et al. 2014). This dissociation is key because it implies that excelling in one domain does not predict excellence in the other, allowing cognitive ability and emotional intelligence to serve as differential predictors for tasks that lean on distinct cognitive or affective processes. For more detailed development of these concepts, please see Online Appendix M.

3.3. Cognitive-Affective Spectrum and Annotator Performance

We apply this dual-process framework by conceptualizing psychological annotation tasks along a cognitive-affective spectrum, reflecting the different blends of processing they demand (Pennebaker and Francis 1996, Panksepp 2003). To represent this spectrum, we examine constructs of resilience, empathy, and polarity, which are commonly studied in social science literature (see Online Appendix L for detail), and exemplify a range of cognitive and affective demands. At the cognitive pole are tasks demanding deliberative, analytical thought characteristic of Type 2 processing. The annotation of resilience exemplifies this because it is an act of demanding cognitive appraisal. An annotator must deconstruct a narrative, analyze coping strategies, and synthesize this information into a holistic judgment—a process reliant on working memory and “cognitive decoupling,” the ability to simulate another’s mental state (Evans and Stanovich 2013). At the affective pole are tasks dominated by the rapid, intuitive judgments of Type 1 processing. Annotation of sentiment polarity is representative of this end, guided primarily by the “affect heuristic,” where an annotator’s immediate “gut feeling” serves as the basis for evaluation (Slovic et al. 2007). This is a fast, automatic process of registering an internal affective state, rather than analyzing an external situation (Clore and Huntsinger 2007). Positioned between these poles, empathy annotation represents a hybrid task. As a multidimensional construct, its successful annotation requires both cognitive empathy (a controlled, Type 2 effort to understand another’s perspective) and affective empathy (an automatic, Type 1 resonance with another’s emotional state) (Decety and Jackson 2004, Singer and Lamm 2009). A successful annotator must synthesize both logical deduction and emotional reaction.

This framework (further detailed in Online Appendix M) provides a clear basis for predicting how annotation accuracy varies with an individual’s cognitive and affective abilities (Salovey and Mayer 1990, Stanovich and West 1998, Joseph and Newman 2010). Accordingly, we propose the following hypotheses:

3.4. Cognitive-Affective Spectrum and Model Performance

3.4.1. Effects of Annotator Cognitive-Affective Traits on Downstream Model Performance.

Building upon the premise that human annotation accuracy is influenced by the alignment between individual traits and task demands, we posit that fine-tuning NLP models with data labeled by annotators whose cognitive abilities or emotional intelligence align with specific task requirements will enhance model performance. Supporting this, research indicates that high-quality data sets are essential for robust model training, because models trained on superior data show improved generalization (Mishra et al. 2020), and that the quality of training data critically affects model outcomes, especially in specialized domains (Bras et al. 2020). Aligning annotator traits with task demands is expected to yield higher-quality annotations, thereby enhancing fine-tuning and overall model performance. Thus, we hypothesize:

3.4.2. Cognitive-Affective Prompting for LLMs.

Given that individuals’ cognitive ability and emotional intelligence affect their performance in text annotation tasks, it is pertinent to investigate whether focusing on cognitive-affective processes may likewise be used to influence the performance of LLMs. A number of studies provide theoretical rationales for why LLMs calibrated to incorporate both cognitive and affective signals may outperform more conventional prompting approaches. For example, Kong et al. (2024) observe that LLMs exhibit a strong capacity for role-playing: their experiments demonstrate that role-playing prompts (e.g., “You are an excellent math teacher”) outperform standard zero-shot prompting across multiple data sets, corroborating the earlier findings of Wu et al. (2023) on the positive impact of role-playing on LLMs’ performance. This line of research suggests that, by incorporating personas, LLMs can generate more contextually appropriate responses and thereby enhance their utility and effectiveness for specific tasks (Chen et al. 2023, 2024; Tseng et al. 2024).

Shen et al. (2024) extend this stream of research by conducting extensive experiments that reveal stable differences in four aspects of decision-making abilities across distinct role emulations, indicating a robust correlation between these roles and the LLMs’ decision-making abilities. Despite this progress, the performance of LLMs when adopting particular personas depends on how well those personas align with the task at hand, as illustrated by Zheng et al. (2024) in extensive experiments using 162 personas. Their findings show that although choosing the best persona for each question significantly increases prediction accuracy, identifying this persona automatically remains difficult, often yielding performance no better than random selection.

This difficulty in identifying the most suitable persona for a downstream task continues to be a focus of current NLP research (Fröhling et al. 2024). In response, we seek to extend this body of work by integrating the cognitive-affective framework. Because humans’ ability to annotate text depends on their cognitive-affective capacities and alignment with the task, we propose that LLMs could likewise be prompted based on the interplay between a task’s cognitive and affective requirements. Specifically, LLMs could be prompted to assume superior cognitive abilities when working on tasks that involve cognitive processes and superior emotional intelligence when working on tasks that involve affective processes. Given that most tasks can be situated along the cognitive-affective spectrum, researchers can then determine whether emphasizing cognitive abilities, emotional intelligence, or both would yield the best performance for a given task.

Based on dual-process theory and the distinctions between cognitive and affective processes, we propose the following hypothesis regarding how individual differences in cognitive ability and emotional intelligence affect annotation accuracy across resilience, empathy, and polarity tasks:

4. Analysis and Results

4.1. Comparing LLMs with Other Psychometric NLP Methods

Guided by our framework, we begin our analysis by evaluating performance and practicality of various approaches to text classification for three representative case studies along the cognitive-affective spectrum, referring to psychological constructs of resilience (case 1), empathy (case 2), and polarity (case 3). We selected these three constructs because, although all three draw on both cognitive and affective processes, they vary in extent, allowing us to examine the impact of individual abilities. We first report our findings regarding the comparison of LLMs with the other psychometric NLP methods: (1) lexicons, (2) custom-built machine learning models, and (3) fine-tuned language models (details are in Online Appendices A, B, and C). We then focus on cognitive and affective processes and test whether LLMs’ performance can be enhanced using this theoretical perspective.¹

• For case 1, we use the data set from Mousavi and Gu’s (2024) work on measuring resilience content, a concept defined as the ability to adapt and recover from disruptions. We predict whether tweets are labeled as “resilience” or “nonresilience” content by trained annotators.

• For case 2, we utilize a data set developed from corporate earnings calls in which we measured empathy content. Sentences extracted from these calls were manually labeled by trained annotators, as described in Online Appendix B. We predict whether the sentences contain or do not contain empathetic language.

• For case 3, we use a data set which was used to create the FinBERT model (Araci 2019), in which excerpts from financial news and press releases were categorized for polarity by trained annotators. We predict the polarity label for each phrase.²

4.1.1. Results: Predictive Performance.

Table 2 reports the area under the receiver operating characteristic curve (AUC) scores for each method across cases, tested on held-out samples. As noted, we focused on text classification, which measures the (degree of) presence or absence of a psychological construct in text. The main objective in classification is to separate samples that contain the psychological construct (i.e., positive samples) from others (i.e., negative samples). We focus on AUC because it evaluates a model’s ability to separate classes irrespective of class imbalance.

Table 2. Evaluation of Predictive Performance Across Psychometric NLP Paradigms

To provide a comprehensive assessment of LLMs, we include OpenAI’s proprietary GPT-4o alongside several open-weight models (Llama 3.1, Mistral) of varying sizes. We test these LLMs in a standard few-shot inference setting, as well as with two common adaptation techniques for Llama and Mistral: soft prompting (adapting the model via learnable input vectors (Xu et al. 2023)) and fine-tuning (further training the model on our task-specific labeled data (Hu et al. 2021)).

As shown in Table 2, the fine-tuned BERT (FLM) sets a strong benchmark, achieving the top score on the resilience task (AUC = 98.59). However, GPT-4o demonstrates a small improvement over BERT on the more affective empathy and polarity tasks. This demonstrates that standard LLMs can perform nearly as well or better than FLMs (partially supporting Hypothesis 1(a)). A closer look at the various LLMs tested reveals several consistent patterns: First, model size correlates with performance, as the Llama 3.1 70B model consistently outperformed its 8B version. Second, fine-tuning proves to be a reliable enhancement strategy, boosting performance for nearly every model. In contrast, soft-prompting produced mixed results—improving some models while degrading others. Finally, model architecture is a significant factor, with Mistral’s Mixture-of-Experts (MoE) design consistently outperforming the similarly sized dense transformer of Llama 3.1 8B. This finding is aligned with established research showing that MoE architectures achieve greater computational efficiency, allowing them to leverage a vast number of parameters with the computational cost of a much smaller dense model, thereby providing a more effective path to high performance (Shazeer et al. 2017, Fedus et al. 2022).

4.1.2. Results: Consistency in Predictive Performances.

Social science researchers often use the psychological constructs extracted from text in a subsequent downstream task. For instance, Mousavi and Gu (2024) used psychometric NLP to measure resilience content in a large collection of text samples and then used this as a variable in an econometric model to show that leaders’ resilience communication improves community compliance. Natural variance within such corpora means that text samples can range widely, from those that are highly typical or representative of the overall corpus to those that are highly distinct or unique. Because these features are frequently used in subsequent analytical processes, such as in econometric and regression models, it is imperative that the extraction method for these psychological features maintains a consistent level of performance across different samples in the data. For example, financial headlines vary greatly; some follow conventional patterns and language typical of financial reporting, whereas others might use unique or unconventional expressions to grab attention or convey nuanced perspectives. If the chosen method for measuring polarity is more adept at accurately identifying sentiment in conventional headlines but struggles with those that are less typical (but more engaging), this discrepancy can lead to skewed polarity scores, causing biased estimates in subsequent econometric models. Such systematic measurement error that varies with document characteristics is especially problematic because it is difficult to accommodate using econometric techniques (Yang et al. 2018).

Borrowing from the parallel coordinates technique (Inselberg 2009), we implemented a novel method to assess performance consistency across different segments of data. First, we employed pretrained language model “all-mpnet-base-v2”³ from the Python package “sentence-transformers” for converting each text sample (e.g., sentence, tweet, news headline) to a vector representation with 768 dimensions. We computed cosine similarity to gauge the degree of resemblance between each sample vector and a composite mean vector computed across all samples. The data set was then stratified into quartiles based on these cosine similarity scores, with the most distinct samples in the lowest quartile (Q1) and the most typical samples in the highest quartile (Q4). We computed AUC scores for each quartile in each case along with standard deviations (SDs) and coefficients of variation (CVs; CV = SD/mean). Results are presented in Table 3, comparing GPT-4o (the best-performing LLM) with established methods. Across the three cases, GPT-4o exhibited the lowest CV by a sizable margin, showing the highest level of consistency in measurement across different segments of data (supporting Hypothesis 1(b)).

Table 3. Evaluation of Consistency Across Psychometric NLP Paradigms

4.1.3. Fairness in Psychometric NLP.

Beyond variation in performance across individual samples within corpora, there may be more systematic variation in the ability of models to consistently perform across distinct subpopulations. In addition to issues this may cause for downstream models, it may also result in bias or lack of fairness. In order to evaluate fairness across subpopulations, we required a context where significant disparities could be present and impactful, as well as a data set that captured demographic characteristics of content authors. Unfortunately, the three cases examined thus far do not meet these criteria, and therefore, we turn to a data set explicitly created for evaluating fairness in NLP models. This distinctive data set captures user-generated content regarding the highly sensitive context of healthcare, along with a variety of related psychological constructs (trust, anxiety, numeracy, and literacy) and demographic details, “affording opportunities for measuring bias and benchmarking fairness of text classification methods” (Abbasi et al. 2021, p. 3748). Demographic factors of race and sex facilitate our examination of potential biases and the evaluation of fairness across NLP methods.

To assess and compare the fairness of various methods, we applied each one to the four psychological constructs—trust, anxiety, numeracy, and literacy. We then measured each method’s fairness using the Disparate Impact (DI) metric, as described in Abbasi et al. (2021, equation 1). DI compares the proportion of positive predictions (e.g., high trust, high anxiety, etc.) for a protected group (often referred to as the “nonprivileged” group) to the proportion of positive predictions for a reference group (“privileged” group):

A DI value of one indicates perfect equity, meaning that both groups are receiving positive predictions at the same rate. This is seen as the ideal scenario under the concept of fairness known as “demographic parity,” which emphasizes equal predictions across groups. A DI value below one indicates that the nonprivileged group is less likely to receive positive predictions compared with the privileged group, suggesting potential bias against the nonprivileged group. Conversely, a DI value above one would indicate bias in favor of the nonprivileged group, with this group receiving a disproportionately higher rate of positive predictions.

4.1.4. Results: Comparing Fairness.

We used DI to measure fairness with respect to sex and race. Consistent with prior work (Abbasi et al. 2021), we defined groups of sex (“male” versus “female”) and race (simplified to “white” versus “nonwhite,” following Friedler et al. 2018). Table 4 reports the AUC and DI values (additional details in Online Appendix G), which are aligned with those reported in prior research (Abbasi et al. 2021), as well as the trends noted in our performance analysis. In comparing DI outcomes, we find that across all four cases, the LIWC lexicon is the most biased and LLM is the most fair method by a significant margin with respect to both sex (with mean deviations 40%–60% lower than all other methods) and race (20%–40% lower), supporting Hypothesis 1(b).

Table 4. Evaluation of Fairness Across Psychometric NLP Paradigms

4.2. Role of Cognitive-Affective Abilities in Text Annotation

Our preceding analysis established the superior performance, consistency, and fairness of LLMs compared with established paradigms. Yet, many of these methods, particularly custom-built models and fine-tuned language models, are fundamentally dependent on human-labeled data to establish a “ground truth” for training and evaluation. This reliance raises a critical question about the very foundation of these benchmarks: do individuals’ traits (i.e., cognitive abilities and emotional intelligence) affect their ability to annotate psychological constructs. To answer this question, we recruited 600 participants through CloudResearch Connect each to annotate 12 samples—4 samples with respect to resilience, 4 for empathy, and 4 for polarity. This study was preregistered (https://aspredicted.org/ggwp-jn8t.pdf) and received Institutional Review Board (IRB) approval from one of the author’s institutions (IRB-SBS 6999). A total of 240 samples per construct were drawn from our original testbed.

After consenting to take the survey, the participants first answered a commitment question (“Do you commit to providing thoughtful answers to the questions in this survey?”) and then answered socio-demographic questions related to age, gender, race, education, and household size and income. We then provided instructions (including definitions) for annotating the data with respect to the construct of interest (i.e., resilience, empathy, and polarity) and then asked them to annotate the data. We used the same annotation instruments that were used in previous studies to create the ground-truth data (Strapparava and Mihalcea 2007, Mousavi and Gu 2024).⁴ In addition to the annotations, we asked the participants to highlight “the key words or phrases that influenced” their annotation decisions. This data set allows us to identify more nuanced patterns. The order of constructs was also randomized. After the annotations, we asked the participants to take questions from the Quality of Life in Neurological Disorders Short Form (Cella et al. 2012, Gershon et al. 2012) to measure cognitive abilities; Wong and Law’s Emotional Intelligence Scale (Wong and Law 2002) to measure their emotional intelligence; and the Positive and Negative Affect Schedule (Crawford and Henry 2004) to measure current positive and negative mood (see Online Appendix H for instrument validity). The last two constructs (positive and negative mood) were used as control variables, consistent with the recommendations of Podsakoff et al. (2003).

4.2.1. Results: Cognitive-Affective Abilities in Human Data Annotations.

After the data collection, we removed anyone who did not pass our attention checks, bot detection check (reCAPTCHA score below 51%), and finished the survey faster than the first percentile in terms of duration of completion, per preregistered conditions. This resulted in 571 valid participants who coded 4 sample items per task, for a total of 2,284 user-task-level observations. Table 5 reports summary statistics for key variables.

Table 5. Summary Statistics of the Key Variables in the Data Annotation Survey

To examine the impacts of cognitive abilities and emotional intelligence on the quality of annotation by participants, we ran logistic regressions using the following equation:

Here, i denotes the sample to be annotated; h represents the participant; $HE\_\mathit{\text{Match}}$indicates alignment between participant (nonexpert) annotations and those of experts (trained annotators who generated the original ground truth data) with a value of one indicating a match and zero indicating no match; controls include education, age, gender, and race-related factors; ${\sigma }_i$ represents sample fixed effects; and ${\epsilon }_{ih}$ is the error term. It is worth noting that in constructing $HE\_\mathit{\text{Match}}$, participants’ responses were binarized by averaging the scores they assigned to each item (e.g., “the text focuses on the tendency to perceive others as in need,” “the text focuses on adopting others’ perspectives,” and “the text focuses on valuing others’ welfare” for annotating Empathy samples). A score of one was assigned if the mean was equal to or exceeded 5 (representing “somewhat agree” on our 7-point Likert scale; 4 was “neutral”), and zero otherwise. We employed two specifications: a conditional logistic regression with ${\sigma }_i$ as the grouping variable and robust standard errors; and a logistic regression where ${\sigma }_i$ was included as dummy variables, with robust errors clustered at the annotator level. The primary distinction between the two approaches is that the first uses robust standard errors, whereas the second applies clustered robust standard errors. Table 6 reports the results of these regression models.

Table 6. Regression Results for Comparing Experts and Nonexperts

The results demonstrate a positive and significant coefficient for cognitive ability in the resilience annotation, indicating that participants with higher cognitive ability scores exhibited greater accuracy in annotating resilience-related samples. In contrast, emotional intelligence does not exhibit a significant relationship with accuracy in resilience annotation, suggesting that participants’ emotional intelligence levels do not influence their ability to correctly label these samples. These findings align with our theoretical expectation that cognitive processes, rather than affective ones, are critical for annotating resilience.

In Table 6, the third and fourth columns provide the results for empathy annotation. Here, the coefficients for cognitive ability remain positive and significant, reaffirming the association between higher cognitive abilities and greater accuracy in this domain. Interestingly, emotional intelligence also emerges as a significant predictor, albeit with a negative coefficient. This counterintuitive finding suggests that participants with higher levels of emotional intelligence were less accurate in annotating empathy-related samples.

A robustness check confirming that our findings are not contingent on task complexity or difficulty is detailed in Online Appendix N.

A potential explanation for this result lies in the nature of emotional intelligence. Individuals with heightened emotional intelligence may be more susceptible to the emotional tone of the text, potentially misinterpreting authorial intent and focusing less on perspective-taking or the welfare of others. This pattern highlights a significant challenge of overattribution in annotations driven by emotional intelligence. We posit that for empathy annotation, which requires a delicate balance between cognitive and affective processing, a high degree of emotional intelligence can paradoxically degrade performance through affective dominance, where potent emotional responses overshadow the requisite cognitive appraisal. This aligns with dual-process theory’s predictions about Type 1 processing dominance: fast, intuitive, emotion-laden responses can generate powerful initial judgments that slower, more deliberative Type 2 processing fails to correct or override (Kahneman 2011). To further investigate this finding, we conducted a careful qualitative examination of tokens participants highlighted as key words or phrases that influenced their decisions during annotation (please see Online Appendix H). Our analysis reveals that participants with higher emotional intelligence seem to have systematically misinterpreted superficial affective language—such as apologies, expressions of gratitude, or mentions of concern—as genuine empathy, while overlooking the deeper empathic markers that experts used to assess perspective-taking and altruistic intent. This finding underscores the need for future research on hybrid cognitive-affective constructs, where processing system interactions create complex dynamics beyond simple additive models.

In Table 6, the final two columns are for polarity. Unlike the previous tasks, cognitive ability does not exhibit a significant relationship with accuracy in this domain. However, emotional intelligence is positively and significantly associated with performance, suggesting that individuals with higher emotional intelligence are better suited for tasks requiring sensitivity to polarity, which likely involves affective discernment.

Overall, these results support our hypothesis regarding the cognitive-affective spectrum of annotation tasks. Specifically, tasks that rely heavily on cognitive processes (e.g., resilience annotation) are shaped predominantly by participants’ cognitive abilities, whereas those requiring affective judgments (e.g., empathy and polarity annotation) are strongly impacted by emotional intelligence. Notably, the unexpected negative relationship between emotional intelligence and empathy annotation accuracy underscores a more intricate interplay between affective processes and task demands, meriting additional inquiry (see Online Appendix H).

A key finding from our analysis is that individual annotators’ characteristics can lead to variability in the labels they assign. Such variability has significant implications for psychometric research reliant on human-labeled data because inconsistent annotations can affect downstream measurements and models.

Our conclusions are based on responses from nonexpert annotators who received minimal training, an unavoidable limitation given the challenges that preclude recruiting a large cohort of experts for analysis. To further clarify this, we conducted an additional analysis contrasting data from trained experts with data from nonexperts who varied in cognitive ability and emotional intelligence (see Online Appendix I for details regarding the data preparation and fine-tuning processes). The main results of this analysis are presented in Table 7. Our results show that whereas nonexpert participants with above-median levels of relevant traits for each task (cognitive ability for resilience, emotional intelligence for polarity, and a combination of the two for empathy) performed at levels comparable to experts, those with lower levels performed significantly worse. Although further research is needed to determine whether experts can better compartmentalize or mitigate biases tied to their underlying traits and skills, our findings suggest that underlying traits can impact labeling tasks, supporting Hypothesis 3(a).

Table 7. Using Nonexpert Data in FLMs

4.3. Cognitive-Affective Prompting

Building on our findings that human annotators’ performance varies according to task fit with their cognitive abilities and emotional intelligence, we examined how LLMs might emulate this effect. Specifically, we asked LLMs to assume the role of a human annotator with higher levels of cognitive abilities and/or emotional intelligence while labeling text samples for resilience, empathy, and polarity.

We tested five prompting strategies (Online Appendix F) and identified the one that consistently yielded the best classification performance (i.e., highest AUC scores). We then refined the best-performing strategy by embedding a “cognitive-affective” component in each prompt. In each prompt, we asked the model to “assume the role of an expert human coder who possesses [X or Y or X and Y],” where X is “exceptional emotional intelligence” (Table 8, “High Emo”) and Y is “superior cognitive abilities” (Table 8, “High Cog”). We then annotated the data using each prompt.

Table 8. Predictive Performance of GPT-4o Across Various Prompting Strategies

Together, these strategies address challenges in eliciting complex reasoning from LLMs. By systematically comparing them, we identified the optimal approach for combining cognitive-affective cues within prompts.

4.3.1. Results: Cognitive-Affective Prompting.

Table 8 reports the results of this analysis. We used “gpt-4o-2024-11-20” with identical settings (temperature = 0) in all cases. We also used a custom Python function to identify the threshold that results in the highest F1 score and used that threshold to report F1, accuracy, positive and negative recall, and precision.

As outlined in the results, GPT-4o achieved its highest performance using the contrastive chain-of-thought prompting strategy for resilience and empathy annotations, whereas the tree-of-thoughts prompting strategy proved most effective for annotating polarity cases. More significantly, our results reveal that tailoring the prompting strategy based on a cognitive-affective framework leads to even greater performance improvements. For resilience annotations, which predominantly require cognitive reasoning, appending the phrase “assume the role of an expert human coder who possesses superior cognitive abilities” to the prompt significantly enhanced performance. For empathy annotations, where both affective and cognitive dimensions are crucial, prompts that included both “exceptional emotional intelligence” and “superior cognitive abilities” yielded the best results. Polarity annotations, which rely more on affective judgment, benefited marginally from prompts emphasizing “exceptional emotional intelligence,” due to almost perfect baseline performance in this case. Overall, if we consider the gap between the baseline performance for both resilience (i.e., 0.9492) and empathy (i.e., 0.9435) and perfect ranking (i.e., AUC = 1), the proposed approach closes more than 10% of that gap. These meaningful performance improvements can be achieved when including such cognitive-affective prompting aligned with task demands, as proposed in Hypothesis 3(b).⁵ It is notable that simply prompting a model to exhibit both high emotional intelligence and cognitive ability is not universally beneficial (e.g., “high cog” is better than “high cog and emo” for the resilience task). From a human psychological standpoint, this might be attributed to hypersensitivity of emotion hindering the application of cognitive skills to a cognitively aligned task (e.g., Truninger et al. 2018, Fiori and Ortony 2021) or possibly to challenges introduced by multigoal pursuit (Neal et al. 2017) or role conflict (Biddle and Thomas 1966, Anglin et al. 2022). Further research is required to determine if such phenomena translate to LLM performance as well.

4.3.2. Analysis of Attention Weights.

To deepen our understanding of how LLMs leverage concepts such as “cognitive abilities” and “emotional intelligence” during annotation tasks, we conducted a comprehensive analysis of attention weights (see Online Appendix J for detailed methodology). Using Meta’s Llama 3.1 70B Instruct model, we tracked the attention paid to the phrases “cognitive abilities” and “emotional intelligence” as the model generated annotations for resilience, empathy, and polarity samples in our validation sets. By analyzing how attention to these key phrases varied during the generation process, we were able to identify patterns that correlate with annotation accuracy.

As reported in Table 9, a discernible pattern emerges regarding the differential impact of attentional focus on annotation accuracy.⁶ For resilience and empathy samples, enhanced performance correlates with increased attention directed toward the phrase “cognitive abilities” relative to “emotional intelligence.” Conversely, annotation accuracy for polarity samples demonstrates a positive relationship with heightened attention to “emotional intelligence” compared with “cognitive abilities.” These observations corroborate our central hypothesis: the inclusion of these conceptual anchors within the prompt modulates LLM performance, with heightened accuracy contingent upon attentional alignment with the contextually appropriate concept, as prescribed by our cognitive-affective prompting strategy. Furthermore, consistent with our a priori expectations, increased instance hardness—a measure of inherent sample annotation difficulty—is associated with diminished model performance. It is noteworthy that the coefficient for instance hardness did not attain statistical significance in the polarity sample analysis. This can be attributed to the exceptionally high AUC achieved by the model for this specific data set, indicating near-perfect performance and thus limiting the observable impact of instance hardness on the already robustly accurate annotations.

Table 9. Attention Weights Analysis

5. Discussion and Conclusion

Empirical research in social science has been markedly influenced by the adoption of NLP techniques to measure psychological constructs in text and the evolution (and revolutions) in these methods over time. Despite advancements, social science researchers still grapple with challenges in data collection, ground-truth labeling of training sets, and mastering NLP techniques to effectively utilize advanced methods. The dilemma often lies in choosing between simple, but less accurate, lexicon-based methods and the demanding process of labeling data and deploying FLMs. In contrast, LLMs offer a revolutionary approach. They eliminate the need for manual labeling or technical expertise⁷ required to develop NLP models, while still delivering results comparable (or superior) to FLMs. Stage 1 of our holistic evaluation demonstrates LLMs have predictive performance on par with FLMs (Hypothesis 1(a)), make more consistent predictions over different segments of data (Hypothesis 1(b)), and provide more fair predictions (Hypothesis 1(c)), despite requiring less domain or NLP expertise (see Table 10). These benefits also come at extremely low cost. Each application programming interface (API) request to GPT-4o cost approximately $0.001, yielding a total expenditure of $3.35 for processing a data set of 3,000 items. At the time of writing, the published rates for more advanced models such as GPT-5 and Gemini 2.5 Pro are marginally lower than the costs we incurred. Comparable pricing is also available through third-party platforms, including replicate.com and fireworks.ai, which host a range of open- and closed-source models. By contrast, human annotation of the same 3,000-item data set required an outlay of roughly $300 to produce ground-truth labels for the empathy training set, after which CBMs or FLMs remain necessary to generalize beyond the labeled subset.

Table 10. Holistic Evaluation of Psychometric NLP Paradigms

In stages 2 and 3 of our holistic evaluation, we further demonstrate that human annotators’ abilities in identifying psychological constructs depend on their cognitive and affective abilities (Hypotheses 2(a), (b), and (c)) and task-trait fit along these dimensions has a significant influence on the predictive performance of downstream FLMs (Hypothesis 3(a)). Inspired by these findings, we introduce a cognitive-affective prompting strategy that improves LLM performance (Hypothesis 3(b)). These findings suggest that when utilizing LLMs for psychometric NLP tasks, considering cognitive-affective demands and crafting prompts that encourage alignment with those demands can improve performance. This contributes to the research on persona-based prompting by offering a strategy that addresses discrepancies in prior studies and enhances LLM performance across psychometric tasks. Practically, it gives social scientists a direct lever to enhance the validity of automated coding, moving beyond generic prompts to ones tailored for psychological measurement.

Positioned toward the “data science and analytics” end of the abstraction spectrum defined in Abbasi et al. (2024) and inspired by studies they identify (Liu et al. 2020, Lee et al. 2025), our theory-guided framework for holistic evaluation of paradigms for psychometric NLP affords three notable salient design insights. First, we elucidate five key dimensions for evaluating the different paradigms. Using these dimensions, we note that LLMs may exhibit comparable performance, and better fairness and consistency, with lower requirements for domain and NLP expertise (Hypothesis 1). Second, our framework and evaluation underscore the importance of the interplay between annotators’ cognitive-affective abilities and task-trait fit on labeling performance as well as downstream performance for FLMs (Hypothesis 2 and Hypothesis 3(a)). Third, we empirically demonstrate how cognitive-affective considerations may positively inform LLM-based scoring (Hypothesis 3(b)).

These design insights have important implications for future social science research leveraging psychometric NLP. As specific examples, design insights from our framework can inform computationally intensive theory construction research in deriving psychometric information from text (Miranda et al. 2022), as well as design of novel psychometric NLP artifacts for social science research extracting information from technology-enabled communication (e.g., Liu et al. 2020, Yang et al. 2023, Kitchens et al. 2024). Our work highlights how LLMs can be incorporated into future artifacts and also provides a high-performing, easy to implement, objective, and consistent benchmark for comparison in evaluating proposed artifacts. Considering required domain and NLP expertise, predictive performance, consistency, fairness, and interpretability, we advocate for the acceptance of LLMs as a standard for psychometric NLP in social science research.

6. Limitations and Future Research

Our study has inherent limitations that, in turn, suggest promising avenues for future research. To begin, beyond those we investigate in this study, there are a variety of other dimensions in which LLMs may (or may not) exhibit superior performance over established psychometric NLP approaches. For instance, the ability of LLMs to provide rationale for their outputs has potential impacts for interpretability. The practical relevance of identified constructs may also be improved by the ability to customize prompts and focus LLMs on the precise definition of constructs that researchers wish to identify. Similarly, although we find that GPT-4o has desirable consistency and fairness benefits over other paradigms, we did not examine the impact of cognitive-affective prompting on these dimensions. This also seems to be a rich area for future inquiry. Our findings regarding the benefits of cognitive-affective prompting strategy for LLM annotations suggests additional future research potential: this type of benefit may also accrue to human annotators through behavioral nudging to emphasize cognitive or affective abilities, which may be examined in future studies. Regarding our findings for annotation and personal traits, as with any study surveying human subjects, we had to make choices regarding ordering and length of the survey instrument. Although we followed best practices, future replications with alternative formats could improve the evidence relative to our findings. Although we have proposed and preliminarily examined possible mechanisms, the counterintuitive result that higher emotional intelligence negatively influences empathy annotation merits further study. We suggest that further investigating psychological constructs demanding high levels of both cognitive and affective processes is a promising future direction. It may also be interesting to explore targeted training interventions designed to help compensate for annotator trait-task misalignment.