Explainable AI helps people understand why a model made a prediction. The goal is not to make every model simple; it is to make model behavior inspectable enough for debugging, trust, risk review, and communication.
Why it matters
Teams use explainability to debug bad features, find leakage, detect bias, explain decisions to users, satisfy governance needs, and decide when a human should review a case. In high-impact domains, a score without a reason is often not enough.
Key terms
- Global explanation: describes overall model behavior across many examples.
- Local explanation: describes one specific prediction.
- Feature importance: estimate of which inputs influence predictions most.
- SHAP: method inspired by cooperative game theory that assigns contribution values to features.
- LIME: method that explains one prediction by fitting a simple local surrogate model around it.
- Counterfactual: shows what minimal change could alter the model's decision.
- Saliency map: highlights image regions that influenced a vision model.
SHAP vs LIME
- SHAP gives feature contribution values that add up toward a prediction, but it can be more expensive.
- LIME perturbs inputs around one example and fits an easier local model, but results can vary with settings.
- Both are explanation tools, not proof of causality.
- Use explanations with validation, domain review, and fairness checks.
Visual explanation suggestion
Show a prediction score as a horizontal bar. Feature chips push the score left or right with positive and negative contribution arrows. Let learners toggle between a global importance chart and a local SHAP-style waterfall.
Common mistakes
- Treating feature importance as causation.
- Explaining only successful predictions and ignoring failures.
- Using explanations to justify an unfair model instead of improving it.
- Forgetting that correlated features can split or hide importance.
- Showing complex explanations to users without clear wording or actionability.
Interview-style questions
- What is the difference between local and global explainability?
- How does LIME explain an individual prediction?
- What does a SHAP value represent?
- Why should explanations not be treated as causal proof?
Related lessons
- Decision Trees
- Random Forests
- Model Evaluation Metrics
- AI Ethics & Bias
- Model Monitoring & Drift
Related project/template CTA
Add explainability reports to the MLOps Starter Kit or Invoice OCR AI Project so project users can inspect predictions and failure cases.