SHAP (SHapley Additive exPlanations) is an XAI technique based on game theory that assigns each feature a contribution score for a specific prediction. It asks: how much did each feature increase or decrease the prediction compared to the average? SHAP values are consistent, theoretically grounded, and work with any model — making them the most widely used XAI technique in production.

Explainable AI (XAI) – UseCaseinAI

Q: What is Explainable AI in simple terms?

Explainable AI is a set of techniques that answer the question: why did the AI make this decision? When a model rejects a loan application, XAI can tell you: the most important factor was a debt-to-income ratio of 0.62, followed by two missed payments in the last 12 months. You can evaluate whether that reasoning is fair, accurate, and legally defensible.

Q: What is the difference between interpretable and explainable AI?

Interpretable models are inherently transparent — decision trees and linear regression show their reasoning directly. You do not need XAI techniques; the model itself is the explanation. Explainable AI techniques (SHAP, LIME) are applied post-hoc to black-box models that are not inherently interpretable — they approximate the model's reasoning rather than revealing its true internal logic.

Q: Is Explainable AI legally required?

In growing contexts, yes. The EU AI Act requires explanations for high-risk AI decisions in hiring, credit, healthcare, and law enforcement. The EU GDPR's right to explanation gives individuals the right to understand automated decisions that significantly affect them. In the US, financial regulators require that adverse credit action notices explain the primary reasons for denial.

⚡ Explainable AI (XAI) is the field of techniques that make AI decisions understandable to humans. When a black-box model rejects a loan, denies insurance, or flags a medical scan, XAI surfaces which factors drove the decision — enabling auditing, bias detection, regulatory compliance, and meaningful recourse for affected people. It is increasingly required by law.

Category: AI Safety & Ethics · Difficulty: Intermediate · Last updated: 15 May 2026 · 5 min read

Explainable AI (XAI) — What It Is, How It Works & Why It Is Becoming a Legal Requirement

What is Explainable AI?

A neural network approves or rejects 50,000 loan applications per day. It is 94% accurate — better than any human loan officer. But when an applicant is rejected, they ask why. The neural network has 47 layers and 12 million weights. No single weight means “this person is a default risk.” The answer — the real internal answer — is a mathematical operation too complex for any human to follow.

Explainable AI is the set of tools and techniques built to bridge this gap. Not to open the black box — that is often impossible — but to produce human-readable explanations that are accurate enough to be useful. Which features mattered most? In which direction? By how much? These approximate explanations allow loan officers to verify reasonableness, regulators to audit for discrimination, and applicants to understand and potentially challenge decisions.

KEY TECHNIQUES

SHAP (SHapley Additive exPlanations):
Based on Shapley values from cooperative game theory. For each prediction, SHAP calculates how much each feature contributed — positively or negatively — compared to the average prediction. Produces consistent, theoretically grounded explanations. Works with any model type. The industry standard for production XAI.

LIME (Local Interpretable Model-agnostic Explanations):
Perturbs the input (makes small changes) and observes how the model’s prediction changes. Fits a simple interpretable model (linear regression) to these perturbed examples in the local neighbourhood of the prediction. The simple model approximates the complex model’s behaviour locally. Fast and flexible but less consistent than SHAP.

Attention visualisation:
For transformer models, visualising which input tokens the model attended to when generating each output token. Gives intuition about what the model focused on — but attention is not the same as importance, and this can be misleading.

Counterfactual explanations:
Instead of explaining what happened, they explain what would have needed to change for a different outcome. “Your loan was rejected. If your debt-to-income ratio were 0.45 instead of 0.62, it would have been approved.” Actionable and natural for affected users.

Real-world examples

Not theory — what real teams actually shipped using this technique.

HSBC uses SHAP explanations in their credit decisioning system — when a loan is declined, the top three SHAP features (highest income variability, recent missed payment, high credit utilisation) are extracted and used to generate the adverse action notice required by US financial regulation.
NHS hospitals in England use XAI tools to explain AI-assisted sepsis predictions — the clinical team sees which vital signs and lab values drove the alert, allowing them to validate or override the AI recommendation with clinical judgment.
Airbnb’s pricing recommendation model uses SHAP to explain to hosts why a suggested price is higher or lower — “your listing price is 12% below market because your response rate is low and your photos score below average.”

Common pitfalls

Explanations are approximations — SHAP and LIME do not reveal the true internal reasoning of a neural network. They produce locally faithful approximations. Acting on them as if they are ground truth is risky.
Explanation gaming — if a model is required to provide explanations, bad actors can use those explanations to game the system. Knowing which features matter most tells them what to manipulate.
Explanation ≠ fairness — a model can produce plausible explanations for discriminatory decisions. XAI tools tell you what the model used, not whether what it used was fair or appropriate.
Over-reliance — when users see an explanation, they tend to trust the AI decision more, even when the explanation is wrong. Paradoxically, XAI can increase inappropriate deference to incorrect AI outputs.

Frequently asked questions

QUESTION 1 What is Explainable AI in simple terms?

ANSWER 1 Techniques that answer: why did the AI make this decision? When a model rejects a loan, XAI surfaces which factors drove it — enabling evaluation of whether the reasoning is fair and legally defensible.

QUESTION 2 What is the difference between interpretable and explainable AI?

ANSWER 2 Interpretable models show their reasoning directly (decision trees, linear regression). Explainable AI techniques (SHAP, LIME) approximate the reasoning of black-box models post-hoc.

QUESTION 3 What is SHAP?

ANSWER 3 A technique that assigns each feature a contribution score for a specific prediction — how much did this feature increase or decrease the output compared to average. Theoretically grounded and the industry standard.

QUESTION 4 Is Explainable AI legally required?

ANSWER 4 Increasingly yes. EU AI Act requires explanations for high-risk decisions. GDPR gives right to explanation for automated decisions. US financial regulators require adverse action notices explaining credit denials.

📬 Get one concept + one use case every Tuesday. Join the newsletter →