mindspore_xai.explainer

Deep neural network explainers.

class mindspore_xai.explainer.Deconvolution(network)[source]

Provides Deconvolution explanation method.

Deconvolution method is a modified version of Gradient method. For the original ReLU operation in the network to be explained, Deconvolution modifies the propagation rule from directly backpropagating gradients to backpropagating positive gradients.

Note

The parsed network will be set to eval mode through network.set_grad(False) and network.set_train(False). If you want to train the network afterwards, please reset it back to training mode through the opposite operations. To use Deconvolution, the ReLU operations in the network must be implemented with mindspore.nn.Cell object rather than mindspore.ops.Operations.ReLU. Otherwise, the results will not be correct.

Parameters: network (Cell) – The black-box model to be explained.

Inputs:

inputs (Tensor) - The input data to be explained, a 4D tensor of shape \((N, C, H, W)\).
targets (Tensor, int, tuple, list) - The label of interest. It should be a 1D or scalar tensor, or an integer, or a tuple/list of integers. If it is a 1D tensor, tuple or list, its length should be the same as inputs.
ret (str): The return object type. ‘tensor’ means returns a Tensor object, ‘image’ means return a PIL.Image.Image list. Default: tensor.
show (bool, optional): Show the saliency images, None means automatically show the saliency images if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 4D tensor of shape \((N, 1, H, W)\). Or list[list[PIL.Image.Image]], the normalized saliency images if ret was set to ‘image’.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore import set_context, PYNATIVE_MODE
>>> from mindspore_xai.explainer import Deconvolution
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # The detail of LeNet5 is shown in model_zoo.official.cv.lenet.src.lenet.py
>>> net = LeNet5(10, num_channel=3)
>>> deconvolution = Deconvolution(net)
>>> # parse data and the target label to be explained and get the saliency map
>>> inputs = ms.Tensor(np.random.rand(1, 3, 32, 32), ms.float32)
>>> label = 5
>>> saliency = deconvolution(inputs, label)
>>> print(saliency.shape)
(1, 1, 32, 32)

class mindspore_xai.explainer.GradCAM(network, layer='')[source]

Provides GradCAM explanation method.

GradCAM generates saliency map at intermediate layer. The attribution is obtained as:

\[ \begin{align}\begin{aligned}\alpha_k^c = \frac{1}{Z} \sum_i \sum_j \frac{\partial{y^c}}{\partial{A_{i,j}^k}}\\attribution = ReLU(\sum_k \alpha_k^c A^k)\end{aligned}\end{align} \]

For more details, please refer to the original paper: GradCAM.

Note

The parsed network will be set to eval mode through network.set_grad(False) and network.set_train(False). If you want to train the network afterwards, please reset it back to training mode through the opposite operations.

Parameters

network (Cell) – The black-box model to be explained.
layer (str, optional) – The layer name to generate the explanation, usually chosen as the last convolutional layer for better practice. If it is ‘’, the explanation will be generated at the input layer. Default: ‘’.

Inputs:

inputs (Tensor) - The input data to be explained, a 4D tensor of shape \((N, C, H, W)\).
targets (Tensor, int, tuple, list) - The label of interest. It should be a 1D or scalar tensor, or an integer, or a tuple/list of integers. If it is a 1D tensor, tuple or list, its length should be \(N\).
ret (str): The return object type. ‘tensor’ means returns a Tensor object, ‘image’ means return a PIL.Image.Image list. Default: tensor.
show (bool, optional): Show the saliency images, None means automatically show the saliency images if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 4D tensor of shape \((N, 1, H, W)\), saliency maps. Or list[list[PIL.Image.Image]], the normalized saliency images if ret was set to ‘image’.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore_xai.explainer import GradCAM
>>> from mindspore import set_context, PYNATIVE_MODE
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # The detail of LeNet5 is shown in model_zoo.official.cv.lenet.src.lenet.py
>>> net = LeNet5(10, num_channel=3)
>>> # specify a layer name to generate explanation, usually the layer can be set as the last conv layer.
>>> layer_name = 'conv2'
>>> # init GradCAM with a trained network and specify the layer to obtain attribution
>>> gradcam = GradCAM(net, layer=layer_name)
>>> inputs = ms.Tensor(np.random.rand(1, 3, 32, 32), ms.float32)
>>> label = 5
>>> saliency = gradcam(inputs, label)
>>> print(saliency.shape)
(1, 1, 32, 32)

class mindspore_xai.explainer.Gradient(network)[source]

Provides Gradient explanation method.

Gradient is the simplest attribution method which uses the naive gradients of outputs w.r.t inputs as the explanation.

\[attribution = \frac{\partial{y}}{\partial{x}}\]

Note

The parsed network will be set to eval mode through network.set_grad(False) and network.set_train(False). If you want to train the network afterwards, please reset it back to training mode through the opposite operations.

Parameters: network (Cell) – The black-box model to be explained.

Inputs:

inputs (Tensor) - The input data to be explained, a 4D tensor of shape \((N, C, H, W)\).
targets (Tensor, int, tuple, list) - The label of interest. It should be a 1D or scalar tensor, or an integer, or a tuple/list of integers. If it is a 1D tensor, tuple or list, its length should be \(N\).
ret (str): The return object type. ‘tensor’ means returns a Tensor object, ‘image’ means return a PIL.Image.Image list. Default: tensor.
show (bool, optional): Show the saliency images, None means automatically show the saliency images if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 4D tensor of shape \((N, 1, H, W)\), saliency maps. Or list[list[PIL.Image.Image]], the normalized saliency images if ret was set to ‘image’.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore import set_context, PYNATIVE_MODE
>>> from mindspore_xai.explainer import Gradient
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # The detail of LeNet5 is shown in model_zoo.official.cv.lenet.src.lenet.py
>>> net = LeNet5(10, num_channel=3)
>>> gradient = Gradient(net)
>>> inputs = ms.Tensor(np.random.rand(1, 3, 32, 32), ms.float32)
>>> label = 5
>>> saliency = gradient(inputs, label)
>>> print(saliency.shape)
(1, 1, 32, 32)

class mindspore_xai.explainer.GuidedBackprop(network)[source]

Provides Guided-Backpropagation explanation method.

Guided-Backpropagation method is an extension of Gradient method. On top of the original ReLU operation in the network to be explained, Guided-Backpropagation introduces another ReLU operation to filter out the negative gradients during backpropagation.

Note

The parsed network will be set to eval mode through network.set_grad(False) and network.set_train(False). If you want to train the network afterwards, please reset it back to training mode through the opposite operations. To use GuidedBackprop, the ReLU operations in the network must be implemented with mindspore.nn.Cell object rather than mindspore.ops.Operations.ReLU. Otherwise, the results will not be correct.

Parameters: network (Cell) – The black-box model to be explained.

Inputs:

inputs (Tensor) - The input data to be explained, a 4D tensor of shape \((N, C, H, W)\).
targets (Tensor, int, tuple, list) - The label of interest. It should be a 1D or scalar tensor, or an integer, or an tuple/list of integers. If it is a 1D tensor, tuple or list, its length should be \(N\).
ret (str): The return object type. ‘tensor’ means returns a Tensor object, ‘image’ means return a PIL.Image.Image list. Default: tensor.
show (bool, optional): Show the saliency images, None means automatically show the saliency images if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 4D tensor of shape \((N, 1, H, W)\), saliency maps. Or list[list[PIL.Image.Image]], the normalized saliency images if ret was set to ‘image’.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore_xai.explainer import GuidedBackprop
>>> from mindspore import set_context, PYNATIVE_MODE
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # The detail of LeNet5 is shown in model_zoo.official.cv.lenet.src.lenet.py
>>> net = LeNet5(10, num_channel=3)
>>> gbp = GuidedBackprop(net)
>>> # feed data and the target label to be explained and get the saliency map
>>> inputs = ms.Tensor(np.random.rand(1, 3, 32, 32), ms.float32)
>>> label = 5
>>> saliency = gbp(inputs, label)
>>> print(saliency.shape)
(1, 1, 32, 32)

class mindspore_xai.explainer.LIMETabular(predictor, train_feat_stats, feature_names=None, categorical_features_indexes=None, class_names=None, num_perturbs=5000, max_features=10)[source]

Provides Lime Tabular explanation method.

Explains predictions on tabular (i.e. matrix) data. For numerical features, perturb them by sampling from a Normal(0,1) and doing the inverse operation of mean-centering and scaling, according to the means and stds in the training data. For categorical features, perturb by sampling according to the training distribution, and making a binary feature that is 1 when the value is the same as the instance being explained.

Parameters

predictor (Callable) – The black-box model to be explained, should be a callable function. For classification model, it accepts a 2D array/tensor of shape \((N, K)\) as input and outputs a 2D array/tensor of shape \((N, L)\). For regression model, it accepts a 2D array/tensor of shape \((N, K)\) as input and outputs a 1D array/tensor of shape \((N)\).
train_feat_stats (dict) – a dict object having the details of training data statistics. The stats can be generated using static method LIMETabular.to_feat_stats(training_data).
feature_names (list, optional) – list of names (strings) corresponding to the columns in the training data. Default: None.
categorical_features_indexes (list, optional) – list of indices (ints) corresponding to the categorical columns, their values MUST be integers. Other columns will be considered continuous. Default: None.
class_names (list, optional) – list of class names, ordered according to whatever the classifier is using. If not present, class names will be ‘0’, ‘1’, … Default: None.
num_perturbs (int, optional) – size of the neighborhood to learn the linear model. Default: 5000.
max_features (int, optional) – Maximum number of features present in explanation. Default: 10.

Inputs:

inputs (Tensor, numpy.ndarray) - The input data to be explained, a 2D float tensor or 2D float numpy array of shape \((N, K)\).
targets (Tensor, numpy.ndarray, list, int, optional) - The labels of interest to be explained. When targets is an integer, all the inputs will generate attribution map w.r.t this integer. When targets is a tensor, numpy array or list, it should be of shape \((N, L)\) (L being the number of labels for each sample), \((N,)\) or \(()\). For regression model, this parameter will be ignored. Default: 0.
show (bool, optional): Show the explanation figures, None means automatically show the explanation figures if it is running on JupyterLab. Default: None.

Outputs:

list[list[list[(str, float)]]], a 3-dimension list of tuple. The first dimension represents inputs. The second dimension represents targets. The third dimension represents features. The tuple represents feature description and weight.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> import mindspore.nn as nn
>>> from mindspore_xai.explainer import LIMETabular
>>> # Linear classification model
>>> class LinearNet(nn.Cell):
...     def __init__(self, num_inputs, num_class):
...         super(LinearNet, self).__init__()
...         self.fc = nn.Dense(num_inputs, num_class, activation=nn.Softmax())
...     def construct(self, x):
...         x = self.fc(x)
...         return x
>>> net = LinearNet(4, 3)
>>> # use iris data as example
>>> feature_names = ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
>>> class_names = ['setosa', 'versicolor', 'virginica']
>>> train = ms.Tensor(np.random.rand(10, 4), ms.float32)
>>> stats = LIMETabular.to_feat_stats(train, feature_names=feature_names)
>>> lime = LIMETabular(net, stats, feature_names=feature_names, class_names=class_names)
>>> inputs = ms.Tensor(np.random.rand(2, 4), ms.float32)
>>> targets = ms.Tensor([[1, 2], [1, 2]], ms.int32)
>>> exps = lime(inputs, targets)
>>> # output is a 3-dimension list of tuple
>>> print((len(exps), len(exps[0]), len(exps[0][0])))
(2, 2, 4)

static load_feat_stats(file)[source]

Load feature stats from disk.

Parameters: file (str, Path, IOBase) – File path or stream.
Returns: dict, training data stats

static save_feat_stats(stats, file)[source]

Save feature stats to disk.

Parameters

stats (dict) – training data stats.
file (str, Path, IOBase) – File path or stream.

static to_feat_stats(features, feature_names=None, categorical_features_indexes=None)[source]

Convert features to feature stats.

Parameters

features (Tensor, numpy.ndarray) – training data.
feature_names (list, optional) – feature names. Default: None.
categorical_features_indexes (list, optional) – list of indices (ints) corresponding to the categorical columns, their values MUST be integers. Other columns will be considered continuous. Default: None.

Returns

dict, training data stats

class mindspore_xai.explainer.Occlusion(network, activation_fn, perturbation_per_eval=32)[source]

Provides Occlusion explanation method.

Occlusion uses a sliding window to replace the pixels with a reference value (e.g. constant value), and computes the output difference w.r.t the original output. The output difference caused by perturbed pixels are assigned as feature importance to those pixels. For pixels involved in multiple sliding windows, the feature importance is the averaged differences from multiple sliding windows.

For more details, please refer to the original paper via: Visualizing and Understanding Convolutional Networks.

Note

Currently only single sample (\(N=1\)) at each call is supported.

Parameters

network (Cell) – The black-box model to be explained.
activation_fn (Cell) – The activation layer that transforms logits to prediction probabilities. For single label classification tasks, nn.Softmax is usually applied. As for multi-label classification tasks, nn.Sigmoid is usually be applied. Users can also pass their own customized activation_fn as long as when combining this function with network, the final output is the probability of the input.
perturbation_per_eval (int, optional) – Number of perturbations for each inference during inferring the perturbed samples. Within the memory capacity, usually the larger this number is, the faster the explanation is obtained. Default: 32.

Inputs:

inputs (Tensor) - The input data to be explained, a 4D tensor of shape \((N, C, H, W)\).
targets (Tensor, int, tuple, list) - The label of interest. It should be a 1D or scalar tensor, or an integer, or a tuple/list of integers. If it is a 1D tensor, tuple or list, its length should be \(N\).
ret (str): The return object type. ‘tensor’ means returns a Tensor object, ‘image’ means return a PIL.Image.Image list. Default: tensor.
show (bool, optional): Show the saliency images, None means automatically show the saliency images if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 4D tensor of shape \((N, 1, H, W)\), saliency maps. Or list[list[PIL.Image.Image]], the normalized saliency images if ret was set to ‘image’.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore_xai.explainer import Occlusion
>>> from mindspore import set_context, PYNATIVE_MODE
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # The detail of LeNet5 is shown in model_zoo.official.cv.lenet.src.lenet.py
>>> net = LeNet5(10, num_channel=3)
>>> # initialize Occlusion explainer with the pretrained model and activation function
>>> activation_fn = ms.nn.Softmax() # softmax layer is applied to transform logits to probabilities
>>> occlusion = Occlusion(net, activation_fn=activation_fn)
>>> input_x = ms.Tensor(np.random.rand(1, 3, 32, 32), ms.float32)
>>> label = ms.Tensor([1], ms.int32)
>>> saliency = occlusion(input_x, label)
>>> print(saliency.shape)
(1, 1, 32, 32)

class mindspore_xai.explainer.RISE(network, activation_fn, perturbation_per_eval=32)[source]

RISE: Randomized Input Sampling for Explanation of Black-box Model.

RISE is a perturbation-based method that generates attribution maps by sampling on multiple random binary masks. The original image \(I\) is randomly masked, and then fed into the black-box model to get predictions. The final attribution map is the weighted sum of these random masks \(M_i\) , with the weights being the corresponding output on the node of interest:

\[attribution = \sum_{i}f_c(I\odot M_i) M_i\]

For more details, please refer to the original paper via: RISE.

Parameters

network (Cell) – The black-box model to be explained.
activation_fn (Cell) – The activation layer that transforms logits to prediction probabilities. For single label classification tasks, nn.Softmax is usually applied. As for multi-label classification tasks, nn.Sigmoid is usually be applied. Users can also pass their own customized activation_fn as long as when combining this function with network, the final output is the probability of the input.
perturbation_per_eval (int, optional) – Number of perturbations for each inference during inferring the perturbed samples. Within the memory capacity, usually the larger this number is, the faster the explanation is obtained. Default: 32.

Inputs:

inputs (Tensor) - The input data to be explained, a 4D tensor of shape \((N, C, H, W)\).
targets (Tensor, int) - The labels of interest to be explained. When targets is an integer, all of the inputs will generates attribution map w.r.t this integer. When targets is a tensor, it should be of shape \((N, L)\) (L being the number of labels for each sample) or \((N,)\) \(()\).
ret (str): The return object type. ‘tensor’ means returns a Tensor object, ‘image’ means return a PIL.Image.Image list. Default: tensor.
show (bool, optional): Show the saliency images, None means automatically show the saliency images if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 4D tensor of shape \((N, L, H, W)\) when targets is a tensor of shape \((N, L)\), otherwise a tensor of shape \((N, 1, H, W)\), saliency maps. Or list[list[PIL.Image.Image]], the normalized saliency images if ret was set to ‘image’.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore_xai.explainer import RISE
>>> from mindspore import set_context, PYNATIVE_MODE
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # The detail of LeNet5 is shown in model_zoo.official.cv.lenet.src.lenet.py
>>> net = LeNet5(10, num_channel=3)
>>> # initialize RISE explainer with the pretrained model and activation function
>>> activation_fn = ms.nn.Softmax() # softmax layer is applied to transform logits to probabilities
>>> rise = RISE(net, activation_fn=activation_fn)
>>> # given an instance of RISE, saliency map can be generate
>>> inputs = ms.Tensor(np.random.rand(2, 3, 32, 32), ms.float32)
>>> # when `targets` is an integer
>>> targets = 5
>>> saliency = rise(inputs, targets)
>>> print(saliency.shape)
(2, 1, 32, 32)
>>> # `targets` can also be a 2D tensor
>>> targets = ms.Tensor([[5], [1]], ms.int32)
>>> saliency = rise(inputs, targets)
>>> print(saliency.shape)
(2, 1, 32, 32)

class mindspore_xai.explainer.RISEPlus(ood_net, network, activation_fn, perturbation_per_eval=32)[source]

Provides RISEPlus explanation method.

RISEPlus is a perturbation-based method that generates attribution maps by sampling on multiple random binary masks. An OoD detector is adopted to produce an ‘inlier score’, estimating the probability that a sample is generated from the distribution. Then the inlier score is aggregated to the weighted sum of the random masks, with the weights being the corresponding output on the node of interest:

\[attribution = \sum_{i}s_if_c(I\odot M_i) M_i\]

For more details, please refer to the original paper: Resisting Out-of-Distribution Data Problem in Perturbation of XAI .

Parameters

ood_net (OoDNet) – The OoD network for generating inlier score.
network (Cell) – The black-box model to be explained.
activation_fn (Cell) – The activation layer that transforms logits to prediction probabilities. For single label classification tasks, nn.Softmax is usually applied. As for multi-label classification tasks, nn.Sigmoid is usually be applied. Users can also pass their own customized activation_fn as long as when combining this function with network, the final output is the probability of the input.
perturbation_per_eval (int, optional) – Number of perturbations for each inference during inferring the perturbed samples. Within the memory capacity, usually the larger this number is, the faster the explanation is obtained. Default: 32.

Inputs:

inputs (Tensor) - The input data to be explained, a 4D tensor of shape \((N, C, H, W)\).
targets (Tensor, int) - The labels of interest to be explained. When targets is an integer, all of the inputs will generates attribution map w.r.t this integer. When targets is a tensor, it should be of shape \((N, L)\) (L being the number of labels for each sample) or \((N,)\) \(()\).
ret (str): The return object type. ‘tensor’ means returns a Tensor object, ‘image’ means return a PIL.Image.Image list. Default: tensor.
show (bool, optional): Show the saliency images, None means automatically show the saliency images if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 4D tensor of shape \((N, L, H, W)\) when targets is a tensor of shape \((N, L)\), otherwise a tensor of shape \((N, 1, H, W)\), saliency maps. Or list[list[PIL.Image.Image]], the normalized saliency images if ret was set to ‘image’.

Raises

TypeError – Be raised for any argument or input type problem.
ValueError – Be raised for any input value problem.

Supported Platforms:: Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> from mindspore import nn, set_context, PYNATIVE_MODE
>>> from mindspore.common.initializer import Normal
>>> from mindspore_xai.explainer import RISEPlus
>>> from mindspore_xai.tool.cv import OoDNet
>>>
>>>
>>> class MyLeNet5(nn.Cell):
...    def __init__(self, num_class, num_channel):
...        super(MyLeNet5, self).__init__()
...
...        # must add the following 2 attributes to your model
...        self.num_features = 84 # no. of features, int
...        self.output_features = False # output features flag, bool
...
...        self.conv1 = nn.Conv2d(num_channel, 6, 5, pad_mode='valid')
...        self.conv2 = nn.Conv2d(6, 16, 5, pad_mode='valid')
...        self.relu = nn.ReLU()
...        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)
...        self.flatten = nn.Flatten()
...        self.fc1 = nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02))
...        self.fc2 = nn.Dense(120, self.num_features, weight_init=Normal(0.02))
...        self.fc3 = nn.Dense(self.num_features, num_class, weight_init=Normal(0.02))
...
...    def construct(self, x):
...        x = self.conv1(x)
...        x = self.relu(x)
...        x = self.max_pool2d(x)
...        x = self.conv2(x)
...        x = self.relu(x)
...        x = self.max_pool2d(x)
...        x = self.flatten(x)
...        x = self.relu(self.fc1(x))
...        x = self.relu(self.fc2(x))
...
...        # return the features tensor if output_features is True
...        if self.output_features:
...            return x
...
...        x = self.fc3(x)
...        return x
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # prepare trained classifier
>>> net = MyLeNet5(10, num_channel=3)
>>> # prepare OoD network
>>> ood_net = OoDNet(net, 10)
>>> # initialize RISEPlus explainer with the pretrained model and activation function
>>> activation_fn = ms.nn.Softmax() # softmax layer is applied to transform logits to probabilities
>>> riseplus = RISEPlus(ood_net, net, activation_fn=activation_fn)
>>> # given an instance of RISEPlus, saliency map can be generate
>>> inputs = ms.Tensor(np.random.rand(2, 3, 32, 32), ms.float32)
>>> # when `targets` is an integer
>>> targets = 5
>>> saliency = riseplus(inputs, targets)
>>> print(saliency.shape)
(2, 1, 32, 32)

class mindspore_xai.explainer.SHAPGradient(network, features, feature_names=None, class_names=None, num_neighbours=200, max_features=10)[source]

Provides SHAP gradient explanation method.

Explains a network using expected gradients (an extension of integrated gradients).

Note

The parsed network will be set to eval mode through network.set_grad(False) and network.set_train(False). If you want to train the network afterwards, please reset it back to training mode through the opposite operations.

Parameters

network (Cell) – The mindspore cell to be explained. For classification, it accepts a 2D tensor of shape \((N, K)\) as input and outputs a 2D tensor of shape \((N, L)\). For regression, it accepts a 2D tensor of shape \((N, K)\) as input and outputs a 1D tensor of shape \((N)\).
features (Tensor) – 2D tensor of shape \((N, K)\) (N being the number of samples, K being the number of features). The background dataset to use for integrating out features, accept (whole or part of) training dataset.
feature_names (list, optional) – list of names (strings) corresponding to the columns in the training data. Default: None.
class_names (list, optional) – list of class names, ordered according to whatever the classifier is using. If not present, class names will be ‘0’, ‘1’, … Default: None.
num_neighbours (int, optional) – Number of subsets used for the estimation of the shap values. Default: 200.
max_features (int, optional) – Maximum number of features present in explanation. Default: 10.

Inputs:

inputs (Tensor) - The input data to be explained, a 2D float tensor of shape \((N, K)\).
targets (Tensor, numpy.ndarray, list, int, optional) - The labels of interest to be explained. When targets is an integer, all the inputs will generate attribution map w.r.t this integer. When targets is a tensor or numpy array or list, it should be of shape \((N, L)\) (L being the number of labels for each sample), \((N,)\) or \(()\). Default: 0.
show (bool, optional): Show the explanation figures, None means automatically show the explanation figures if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 3D tensor of shape \((N, L, K)\). The first dimension represents inputs. The second dimension represents targets. The third dimension represents feature weight.

Supported Platforms:

Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> import mindspore.nn as nn
>>> from mindspore import set_context, PYNATIVE_MODE
>>> from mindspore_xai.explainer import SHAPGradient
>>>
>>> set_context(mode=PYNATIVE_MODE)
>>> # Linear classification model
>>> class LinearNet(nn.Cell):
...     def __init__(self, num_inputs, num_class):
...         super(LinearNet, self).__init__()
...         self.fc = nn.Dense(num_inputs, num_class, activation=nn.Softmax())
...     def construct(self, x):
...         x = self.fc(x)
...         return x
>>> net = LinearNet(4, 3)
>>> # use iris data as example
>>> feature_names = ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
>>> class_names = ['setosa', 'versicolor', 'virginica']
>>> training_data = ms.Tensor(np.random.rand(10, 4), ms.float32)
>>> shap = SHAPGradient(net, training_data, feature_names=feature_names, class_names=class_names)
>>> inputs = ms.Tensor(np.random.rand(2, 4), ms.float32)
>>> targets = ms.Tensor([[1, 2], [1, 2]], ms.int32)
>>> exps = shap(inputs, targets)
>>> print(exps.shape)
(2, 2, 4)

class mindspore_xai.explainer.SHAPKernel(predictor, features, feature_names=None, class_names=None, num_neighbours=5000, max_features=10)[source]

Provides Kernel SHAP explanation method.

Uses the Kernel SHAP method to explain the output of any function.

Parameters

predictor (Callable) – The black-box model to be explained, should be a callable function. For classification model, it accepts a 2D array/tensor of shape \((N, K)\) as input and outputs a 2D array/tensor of shape \((N, L)\). For regression model, it accepts a 2D array/tensor of shape \((N, K)\) as input and outputs a 1D array/tensor of shape \((N)\).
features (Tensor, numpy.ndarray) – 2D tensor or 2D numpy array of shape \((N, K)\) (N being the number of samples, K being the number of features). The background dataset to use for integrating out features, accept (whole or part of) training dataset.
feature_names (list, optional) – list of names (strings) corresponding to the columns in the training data. Default: None.
class_names (list, optional) – list of class names, ordered according to whatever the classifier is using. If not present, class names will be ‘0’, ‘1’, … Default: None.
num_neighbours (int, optional) – Number of subsets used for the estimation of the shap values. Default: 5000.
max_features (int, optional) – Maximum number of features present in explanation. Default: 10.

Inputs:

inputs (Tensor, numpy.ndarray) - The input data to be explained, a 2D float tensor or 2D float numpy array of shape \((N, K)\).
targets (Tensor, numpy.ndarray, list, int, optional) - The labels of interest to be explained. When targets is an integer, all the inputs will generate attribution map w.r.t this integer. When targets is a tensor or numpy array or list, it should be of shape \((N, L)\) (L being the number of labels for each sample), \((N,)\) or \(()\). Default: 0.
show (bool, optional): Show the explanation figures, None means automatically show the explanation figures if it is running on JupyterLab. Default: None.

Outputs:

Tensor, a 3D tensor of shape \((N, L, K)\). The first dimension represents inputs. The second dimension represents targets. The third dimension represents feature weight.

Supported Platforms:

Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore as ms
>>> import mindspore.nn as nn
>>> from mindspore_xai.explainer import SHAPKernel
>>> # Linear classification model
>>> class LinearNet(nn.Cell):
...     def __init__(self, num_inputs, num_class):
...         super(LinearNet, self).__init__()
...         self.fc = nn.Dense(num_inputs, num_class, activation=nn.Softmax())
...     def construct(self, x):
...         x = self.fc(x)
...         return x
>>> net = LinearNet(4, 3)
>>> # use iris data as example
>>> feature_names = ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
>>> class_names = ['setosa', 'versicolor', 'virginica']
>>> training_data = ms.Tensor(np.random.rand(10, 4), ms.float32)
>>> shap = SHAPKernel(net, training_data, feature_names=feature_names, class_names=class_names)
>>> inputs = ms.Tensor(np.random.rand(2, 4), ms.float32)
>>> targets = ms.Tensor([[1, 2], [1, 2]], ms.int32)
>>> exps = shap(inputs, targets)
>>> print(exps.shape)
(2, 2, 4)