Reference Aware Delexicalization (RAD) Framework: Theory Driven Artificial Intelligence Modeling for Domain Generalization

The ability to generalize is critical for machine learning and natural language processing (NLP) models to perform effectively across a wide range of domains. However, state-of-the-art neural models often struggle to maintain performance when tested out-of-domain (OOD), emphasizing their inability to generalize beyond the data distribution used for training. This challenge of oversensitivity to spurious biases in the training data remains an open research problem across various NLP task areas. Although prior work has introduced techniques to improve domain generalization capabilities of neural networks, existing methods are constrained in their ability to identify and mitigate biases that are both subtle and multidimensional within the training data. These biases can include unintended correlations between features or domain-specific terminology that may not transfer well to other contexts. In response to these limitations, we propose the novel reference aware delexicalization (RAD) data augmentation framework designed to improve generalization for inference-based NLP tasks. The RAD framework uses attention weighting to detect biases and extract bias-prone lexical concepts. Grounded in the theory of reference, RAD’s delexicalization method utilizes the principles of reference fixing and borrowing to generate context-aware placeholder mappings to reduce data oversensitivity. We conducted rigorous benchmark evaluations using RAD-augmented data across various transformer architectures (e.g., BERT, RoBERTa) on several natural language inference (NLI), recognizing textual entailment (RTE), and fact verification data sets. Our findings demonstrate consistent improvements in OOD performance, indicating RAD’s ability to improve model generalization for key NLI and RTE tasks. Because advancing fundamental NLI and RTE capabilities remains crucial for many downstream NLP applications, this work highlights RAD’s potential for positive impact across areas where inference and OOD robustness are highly valued. We also conducted additional qualitative analysis on large language models using RAD across various case studies, demonstrating RAD’s potential as a complementary framework for improving systematic reasoning.

History: Gautam Pant, Senior Editor; Heng Xu, Associate Editor.

Supplemental Material: The online appendices are available at https://doi.org/10.1287/isre.2023.0457.