Towards Symbolic XAI – Explanation Through Human Understandable Logical Relationships Between Features

Machine learning (ML) models are increasingly used in science, industry, and for solving everyday problems. However, their predictions — particularly those of neural networks — are generally not traceable by the user. The field of explainable artificial intelligence (XAI) aims to address this issue by providing additional evidence for why a model made a particular prediction.  For most existing methodologies, such evidence is typically provided in the form of importance scores for individual input features,  which can be too rigid to capture domain-specific queries by the user, such as evidence for the complex interplay between features. Alternatively, surrogate models such as decision trees, trained to mimic the model’s predictions, have been proposed to provide a human-readable but detailed explanation.  However, they often lack faithfulness since they do not have access to the original model’s complex mechanisms and feature interactions, only its output.

We propose an explanation framework that attributes relevance to arbitrary logical feature dependencies and can be customized to the user’s needs.  We call this framework Symbolic XAI (SymbXAI). The relevance of logical formulas incorporates the information of all possible interactions between features contributing to the prediction, thus providing a sample-wise and faithful explanation. For cases where the appropriate logical formula is unknown, the frameworkautomatically searches for the formula that is most relevant or best summarizes the model’s prediction strategy. 

The conceptual basis is to decompose the model’s prediction into distinct contributions of all possible feature combinations—called multi-order decomposition. This can be specified using perturbation-based methods,  which correspond to the Harsanyi dividends from cooperative game theory,  or propagation-based methods,  for which walk relevance values from GNN-LRP  are aggregated. Logical queries, composed of conjunctions and negation of input features, specify whether the presence or absence of features in each feature set satisfies the logical expression. Aggregating the multi-order terms selected by the logical query yields the relevance of the logical expression. For more details on the methodology, we refer to Schnake et al., Section 3.

SymbXAI has broad applicability and is, in principle, not limited to a specific domain or model architecture. We discuss a few exemplary findings in the domains of NLP, computer vision, and chemistry, for which we used Transformer models  and graph neural networks.  For a more detailed discussion, we refer to Schnake et al., Section 4.  

In NLP, especially sentiment analysis,  SymbXAI detects grammatical constructions such as negation. It captures contextually dependent word relevance, which classical methods are unable to express. For complex grammatical structures like contrastive conjunctions, SymbXAI shows that models have learned these patterns without being explicitly trained on them. 

In computer vision, particularly facial emotion recognition (FER),  SymbXAI reveals the relevance of absent facial parts (e.g., masked mouth, sunglasses), which is usually assessed via computationally expensive data augmentation. In object detection, models rely on recurring logical combinations of sub-parts, such as identifying both tires and the frame to detect a mountain bike. 

In drug design, SymbXAI aligns with chemical intuition; for example, the presence of a toxicophore implies the presence of a detoxifying substructure for a molecule to be classified as safe.  Such logical implications can be extracted automatically. 

We demonstrate that prior knowledge about expected feature interactions enhances interpretability and trust. Human-aligned logical interplays increase the utility of explanations in both NLP and chemistry. Our experiments underline that user modeling is key. Classical methods lack the capacity for adaptation. SymbXAI enables logical abstractions over features—an essential capability in complex or high-stakes settings such as computer vision or medicinal chemistry. 

SymbXAI tailors explanations to the user’s context. It captures grammatical rules in NLP and substructure logic in chemistry. Its flexibility and interpretability allow users to adapt the explanation process to their specific needs. 

In summary, the Symbolic XAI framework provides a local understanding of a model’s decision-making process that is both flexible for user customization and human-readable through logical formulas.