Explainable Artificial Intelligence (XAI) Understanding CNN Predictions Using Grad-CAM, LIME and SHAP
Introduction to Explainable Artificial Intelligence
Artificial Intelligence and Deep Learning models are increasingly used for image classification, medical diagnosis, financial analysis, recommendation systems, fraud detection, autonomous systems, and many other applications. These models can achieve high prediction accuracy, but understanding how they arrive at a particular prediction can be difficult.
For example, suppose a Convolutional Neural Network (CNN) receives an image of a shoe and predicts the class as Sandal. The model provides the prediction, but an important question remains:
Why did the model classify this image as a Sandal?
Did the model focus on the shape of the sole? Did it identify the open structure of the footwear? Did it focus on the actual object, or was the prediction influenced by an irrelevant part of the image?
Explainable Artificial Intelligence (XAI) helps answer such questions.
What is Explainable Artificial Intelligence?
Explainable Artificial Intelligence, commonly called XAI, refers to techniques that help humans understand and interpret the predictions or decisions made by Artificial Intelligence and Machine Learning models.
In simple words:
Artificial Intelligence tells us what the model predicted.
Explainable Artificial Intelligence helps us understand why the model made that prediction.
For example:
Input Image → CNN Model → Prediction: Sandal
Using Explainable AI:
Input Image → CNN Model → Prediction: Sandal → Explanation of the Prediction
The explanation may show which regions of the image were important for making the prediction.
Why Do We Need Explainable AI?
Many Machine Learning and Deep Learning models contain a large number of parameters and perform complex mathematical operations internally. Although these models can make accurate predictions, their internal decision-making process may not be directly understandable to humans.
This creates a problem when we need to know why a particular decision was made.
For example, consider a CNN used for medical image classification. If the model predicts that an X-ray contains an abnormality, a doctor may want to know which region of the X-ray influenced the prediction.
Similarly, if a model classifies an image incorrectly, the developer may want to understand which features caused the incorrect prediction.
Explainable AI therefore improves understanding, transparency, trust, debugging, validation, and responsible use of AI systems.
Black-Box Models
A black-box model is a model whose internal decision-making process is difficult for humans to interpret directly.
Deep Neural Networks and Convolutional Neural Networks are often described as black-box models because information passes through many layers and mathematical transformations before producing the final prediction.
For example:
Image → Convolution Layers → Feature Extraction → Hidden Representations → Classification → Prediction
We can observe the input and the final output, but understanding exactly why thousands or millions of internal calculations resulted in that prediction is difficult.
Explainable AI techniques help us investigate this black-box behaviour.
Explainability and Interpretability
The terms explainability and interpretability are closely related.
Interpretability refers to how easily humans can understand how a model works or how its features relate to its predictions.
Explainability focuses on providing understandable reasons for a model's particular prediction or behaviour.
For example, a simple decision tree may be relatively interpretable because we can follow its decision rules directly. A CNN is much more complex, so additional techniques such as Grad-CAM, LIME, or SHAP can be used to explain its predictions.
Explainable AI for Image Classification
In image classification, a CNN learns visual features from training images and uses those features to classify new images.
For example, suppose a CNN is trained using Fashion-MNIST images containing classes such as shirts, trousers, dresses, bags, sneakers, sandals, and ankle boots.
When a new image is given:
Input Image → CNN → Predicted Class: Sneaker
The CNN provides the class prediction. Explainable AI techniques can then be applied to understand which parts of the image influenced that prediction.
The general process is:
Input Image → Trained CNN → Prediction → XAI Technique → Explanation
In this practical, we will study three important Explainable AI techniques:
Grad-CAM
LIME
SHAP
Local and Global Explanations
Before understanding these techniques, it is useful to understand two important types of explanations: local explanations and global explanations.
A local explanation explains why the model made a particular prediction for one specific input.
For example:
Why did the CNN classify this particular image as a sneaker?
A global explanation attempts to explain the overall behaviour of the model.
For example:
Which features generally influence the CNN when distinguishing sneakers from sandals?
Grad-CAM and LIME are primarily used to explain individual predictions. SHAP can also explain individual predictions and, when SHAP values are aggregated across many samples, can provide broader information about model behaviour.
Feature Importance
A central idea in Explainable AI is feature importance.
Feature importance indicates how strongly different features contribute to a model's prediction.
In tabular data, features may be variables such as age, income, salary, or experience.
In image data, the features ultimately relate to visual information contained in different pixels or regions of the image.
An XAI technique attempts to answer:
Which parts of this image were important for the model's prediction?
Different Explainable AI methods answer this question in different ways.
Grad-CAM
Grad-CAM stands for Gradient-weighted Class Activation Mapping.
Grad-CAM is an Explainable AI technique commonly used with Convolutional Neural Networks to visualize the image regions that strongly influenced a particular class prediction.
It generates a heatmap that can be overlaid on the original image.
Suppose the CNN predicts:
Image → Sneaker
Grad-CAM can highlight the regions of the sneaker that contributed strongly to this prediction.
The basic idea is:
Input Image → CNN Prediction → Gradients of Selected Class → Convolutional Feature Maps → Heatmap
The heatmap provides a visual explanation of where the CNN was focusing when making its prediction.
How Grad-CAM Works
CNNs contain convolutional layers that learn visual patterns such as edges, shapes, textures, and object structures.
The deeper convolutional layers generally contain higher-level visual information. Grad-CAM examines the feature maps produced by a convolutional layer and uses gradients associated with the predicted class to determine which feature maps were important.
These contributions are combined to create a heatmap.
The heatmap is then resized and superimposed on the original image.
Conceptually:
Original Image + Grad-CAM Heatmap → Visual Explanation
The highlighted regions indicate areas that had stronger influence on the selected class prediction.
Understanding a Grad-CAM Heatmap
A Grad-CAM heatmap should be interpreted as a visual indication of the regions that influenced the model's prediction.
For example, if a CNN correctly identifies a shoe and the heatmap mainly highlights the shoe itself, it suggests that the model used relevant object regions for its decision.
If the heatmap mainly highlights unrelated background regions, it may indicate that the model learned inappropriate or misleading patterns.
Grad-CAM therefore helps us visually inspect whether the CNN appears to be focusing on meaningful regions.
Advantages of Grad-CAM
Grad-CAM is particularly useful for CNN-based image classification because its output is visually intuitive.
It can help identify important image regions, investigate incorrect predictions, compare predictions between images, and understand whether the model is focusing on meaningful visual information.
Grad-CAM is also convenient because it generally does not require retraining the original CNN.
Limitations of Grad-CAM
Grad-CAM provides a relatively coarse localization map rather than an exact pixel-level explanation.
Its explanation also depends on the selected convolutional layer. The highlighted region tells us where the model found influential information, but it does not always provide a complete explanation of the model's reasoning.
Therefore, Grad-CAM should be treated as an explanation tool rather than proof that the model reasons exactly like a human.
LIME
LIME stands for Local Interpretable Model-agnostic Explanations.
The word Local means that LIME explains an individual prediction.
The word Interpretable means that the explanation should be understandable to humans.
The term Model-agnostic means that LIME can be applied to different types of Machine Learning models because it mainly observes how the model's predictions change when the input is modified.
LIME can be used with tabular data, text data, and image data.
Basic Idea Behind LIME
LIME tries to understand a complex model by studying its behaviour around one particular input.
For an image, LIME divides the image into meaningful regions called superpixels.
It then creates multiple modified versions of the image by hiding or changing some of these regions.
Each modified image is passed through the trained model.
LIME observes how the prediction changes when different image regions are present or absent.
The process can be understood as:
Original Image → Divide into Regions → Modify Regions → Obtain Predictions → Identify Important Regions → Explanation
If removing a particular region causes the model's confidence in the predicted class to decrease significantly, that region is likely important for the prediction.
What is a Superpixel?
A superpixel is a group of neighbouring pixels that have similar visual characteristics and are treated as one meaningful image region.
Instead of changing thousands of individual pixels independently, LIME works with groups of related pixels.
For example, an image of a shoe may be divided into regions representing parts of the sole, upper portion, surrounding background, and other visual areas.
LIME can then investigate which of these regions support or oppose the model's prediction.
Example of LIME
Suppose the CNN predicts an image as:
Predicted Class: Ankle Boot
LIME generates several modified versions of the image by hiding different regions.
The model makes predictions for these modified images.
LIME then identifies the regions that have the strongest influence on the specific prediction being explained.
The final explanation highlights important image regions.
Therefore, LIME answers a question such as:
Which regions of this particular image influenced the model's Ankle Boot prediction?
Advantages of LIME
LIME is model-agnostic, which means it does not require detailed knowledge of the internal architecture of the original model.
It can explain individual predictions and provides visual explanations for images.
It is particularly useful when we want to understand one specific prediction made by a complex black-box model.
Limitations of LIME
LIME generates explanations by creating many modified samples around the original input. Therefore, it may require multiple model predictions and can take additional computational time.
The explanation can also vary depending on the image segmentation, number of generated samples, and random perturbations used.
LIME explains the model's behaviour around a particular input and should not automatically be interpreted as an explanation of the entire model.
SHAP
SHAP stands for SHapley Additive exPlanations.
SHAP is an Explainable AI approach based on Shapley values, a concept originating from cooperative game theory.
The central idea is to estimate how much each feature contributes to a model's prediction.
In simple terms, SHAP attempts to answer:
How much did each feature contribute to the final prediction?
For image classification, the features can be interpreted through pixels or image regions, depending on the SHAP explainer and representation being used.
Basic Idea Behind SHAP
Imagine that different features are members of a team and the model's prediction is the team's final result.
SHAP tries to determine how much each member contributed to that result.
For an image:
Image Features → Model Prediction
SHAP estimates the contribution of different features toward the selected prediction.
Some features may increase the model's support for a particular class, while other features may decrease it.
Therefore:
Positive Contribution → Pushes prediction toward the class
Negative Contribution → Pushes prediction away from the class
Shapley Value
A Shapley value represents the contribution of an individual feature to a prediction by considering its contribution across different combinations of features.
Suppose several features collectively produce a model prediction. Instead of evaluating a feature only once, the Shapley-value concept considers how its presence changes the result across different possible feature combinations.
This provides a principled way of distributing the prediction among contributing features.
In practical Deep Learning applications, SHAP libraries use specialized algorithms and approximations because calculating every possible feature combination directly would be computationally expensive.
SHAP for Image Classification
When SHAP is applied to a CNN image prediction, it can generate a visualization indicating how different image regions contribute to the selected class prediction.
For example:
Input Image → CNN → Prediction: Bag → SHAP → Feature Contribution Visualization
The visualization helps identify which regions support the Bag prediction and which regions may work against it.
This provides another perspective on the model's behaviour.
Advantages of SHAP
SHAP provides feature-contribution explanations grounded in Shapley-value theory.
It can explain individual predictions, and explanations from multiple samples can also be aggregated to study broader model behaviour.
SHAP is widely used with many types of Machine Learning and Deep Learning models.
Limitations of SHAP
SHAP can be computationally expensive, particularly when working with complex models and high-dimensional inputs such as images.
The exact implementation and interpretation also depend on the SHAP explainer being used.
For classroom practicals, the number of images or background/reference samples is often kept small so that the explanation can be generated within a reasonable time.
Difference Between Grad-CAM, LIME and SHAP
Although Grad-CAM, LIME, and SHAP are all Explainable AI techniques, they generate explanations differently.
Grad-CAM uses information from the internal convolutional layers and gradients of a CNN to create a heatmap showing influential image regions.
LIME modifies regions of the input image and observes how those modifications affect the model's prediction. It then builds a simple local explanation around that particular prediction.
SHAP estimates how different features contribute to the prediction using ideas derived from Shapley values.
A simple way to remember them is:
Grad-CAM → Where did the CNN focus?
LIME → Which image regions influence this particular prediction?
SHAP → How much do features contribute to the prediction?
Model-Specific and Model-Agnostic Explanations
Grad-CAM is closely associated with CNN architecture because it uses convolutional feature maps and gradients. Therefore, it is considered a model-specific explanation technique for suitable neural-network architectures.
LIME is model-agnostic because it mainly needs access to the model's prediction function.
SHAP includes different explainers. Some are model-agnostic, while others are designed to exploit particular model structures. Therefore, SHAP should not simply be described as always model-agnostic.
Using XAI with a CNN
For our practical, we will first train a simple CNN for image classification.
The CNN learns visual patterns from training images and predicts the class of an unseen test image.
The workflow is:
Training Images → CNN Training → Trained CNN
Then:
Test Image → Trained CNN → Predicted Class
After obtaining the prediction, we apply Explainable AI:
Predicted Image → Grad-CAM → Heatmap
Predicted Image → LIME → Important Image Regions
Predicted Image → SHAP → Feature Contribution Explanation
This allows us to compare how three different Explainable AI techniques explain the prediction made by the same CNN.
Why Use the Same Image for Grad-CAM, LIME and SHAP?
Using the same test image makes the comparison easier to understand.
Suppose the CNN predicts:
Actual Class: Ankle Boot
Predicted Class: Ankle Boot
We can then apply Grad-CAM, LIME, and SHAP to this same prediction.
Grad-CAM may show where the CNN concentrated its activation for the selected class. LIME may identify image regions that strongly influence the prediction. SHAP may show how image features contribute toward or away from the selected class.
Although the visualizations may look different, all three techniques are helping us investigate the same model prediction from different perspectives.
Explainable AI and Model Accuracy
Explainability and accuracy are different concepts.
A model can have high accuracy but still use undesirable or irrelevant patterns when making some predictions.
Similarly, an explanation does not automatically prove that the model is correct.
For example, a CNN may correctly classify an image but an explanation may reveal that the model relied heavily on an irrelevant background feature.
Therefore:
Accuracy tells us whether the prediction is correct.
Explainability helps us investigate why the model produced that prediction.
Both are important when evaluating AI systems.
Explainable AI and Responsible AI
Explainability is an important part of Responsible Artificial Intelligence because AI systems may affect people and real-world decisions.
Responsible AI commonly involves considerations such as transparency, fairness, accountability, privacy, safety, and explainability.
Explainability helps developers and users inspect model behaviour, identify possible errors, investigate unexpected predictions, and communicate how AI-assisted decisions are being made.
However, an explanation should not automatically be treated as proof that a model is fair, safe, or correct. It is one tool used to evaluate and understand the system.
Real-World Applications of Explainable AI
In healthcare, Explainable AI can help investigate which image regions influenced a medical-image classification model.
In banking and finance, XAI can help analyze which features contributed to predictions such as credit-risk assessments or fraud alerts.
In manufacturing, XAI can help understand which regions of product images influenced defect-detection models.
In autonomous and computer-vision systems, visual explanations can help investigate which parts of an image influenced object-recognition decisions.
In customer analytics, Explainable AI can help identify which factors contributed to predictions such as customer churn or purchase behaviour.
0 टिप्पण्या
कृपया तुमच्या प्रियजनांना लेख शेअर करा आणि तुमचा अभिप्राय जरूर नोंदवा. 🙏 🙏