Investigating the Impact of Conceptual Metaphors on LLM-based NLI through Shapley Interactions

In everyday life, metaphorical language is frequently used in the flow of conversations.While metaphors can be observed in explicit cases such as “she was the light of my life”, the meaning manifestation of conventionalized metaphors such as “tax the rich” is more fundamentally grounded in language.  

Introduciton

According to Lakoff and Johnson , a metaphorical meaning construction is the product of a meaning mapping that connects one concept domain to another: A metaphor is taken from a source domain to explain a target domain. In the sentence “Gun addicts increasingly realize that society is rejecting them”, the source domain is addiction and the target domain is guns.

Existing approaches on interpreting metaphorical language follow a cognitive decoding of conceptual metaphors, integrating source and target domain information into downstream tasks.  Shutoval et al. used hierarchical clustering and conceptual metaphors for metaphor identification and interpretation,  whereas Stowe et al.  paraphrased metaphors into literal counterparts by extracting source and target domain information from FrameNet.  Recently, Sengupta et al.  proposed a multitask approach to jointly predicting source domains and highlighted aspects. 

Unlike previous work, we explore metaphor interpretation in implicit metaphor usage, that is, beyond their explicit identification. Chakrabarty et al.  introduced an NLI dataset, FLUTE, in which many hypotheses are metaphorical. They benchmarked transformer-based models, highlighting the challenge of implicitly figurative downstream tasks. For metaphors, we argue that it is crucial to examine their context and conceptual metaphor interactions. We address this gap in NLP by analyzing LLM performance using a Shapley-based analysis. 

Despite progress in computational metaphor interpretation,  research shows that large language models (LLMs) underperform on downstream tasks that require correct interpretation of figurative language such as metaphors.  Skrynnikova  highlights that LLMs imitate data rather than that reasoning analogically, a key requirement for successful metaphor comprehension. Similarly, Comşa et al.  attribute this struggle to dependency on contextual variables. We address this research gap, and inspired by advances of Shapley-based analyses in explainable AI,  we study LLM interpretability in metaphorical tasks with two research questions: 

(1) How well do LLMs handle implicit metaphorical language in downstream tasks such as natural language inference (NLI)? (2) Does providing source and target domains enhance NLI performance, and how does this information interact with metaphorical texts? 

We evaluate five LLMs, in an NLI setup for given pairs of premise and metaphorical hypothesis. To that end, the contribution of this work is two-fold: (1) We extend the metaphorical samples in the flute dataset by annotations of source and target domains. (2) We investigate the impact of the domains on LLMs in the task of NLI with an analysis of Shapley values and Shapley interactions. 

Takeaway 

Research in metaphor interpretation has explored various approaches. This paper has explored how conceptual metaphors (source and target domains) influence LLM performance in NLI on texts with metaphorical language that is implicitly embedded in the meaning manifestation. To that end, we have first extended the flute dataset with source and target domains. Subsequently, our ablation study using zero-shot and few-shot prompts for two LLMs has showed the best results when explanations are combined with conceptual metaphors. A Shapley-based analysis confirms their positive impact, consistently improving model performance. Our findings suggest that incorporating information about the domains in a mere inferential setup with zero-shot and few-shot learning contributes to improved performance for the LLMs in 70% of the experiments. Our findings lay the ground for future work: Advanced techniques that leverage metaphorical knowledge likely will improve the understanding of implicit metaphorical language by LLMs.