mindspore.nn.probability

mindspore.nn.probability.bijector

Bijector.

The high-level components(Bijectors) used to construct the probabilistic network.

class mindspore.nn.probability.bijector.Bijector(is_constant_jacobian=False, is_injective=True, name=None, dtype=None, param=None)

Bijecotr class.

Parameters

• is_constant_jacobian (bool) – if the bijector has constant derivative. Default: False.

• is_injective (bool) – if the bijector is an one-to-one mapping. Default: True.

• name (str) – name of the bijector. Default: None.

• dtype (mindspore.dtype) – type of the distribution the bijector can operate on. Default: None.

• param (dict) – parameters used to initialize the bijector. Default: None.

construct(name, *args, **kwargs)

Override construct in Cell.

Note

Names of supported functions include: ‘forward’, ‘inverse’, ‘forward_log_jacobian’, ‘inverse_log_jacobian’.

Parameters

• name (str) – name of the function.

• *args (list) – list of positional arguments needed for the function.

• **kwargs (dictionary) – dictionary of keyword arguments needed for the function.

forward(*args, **kwargs)

Forward transformation: transform the input value to another distribution.

forward_log_jacobian(*args, **kwargs)

Logarithm of the derivative of the forward transformation.

inverse(*args, **kwargs)

Inverse transformation: transform the input value back to the original distribution.

inverse_log_jacobian(*args, **kwargs)

Logarithm of the derivative of the inverse transformation.

class mindspore.nn.probability.bijector.Exp(name='Exp')

Exponential Bijector. This Bijector performs the operation: Y = exp(x).

Parameters

name (str) – name of the bijector. Default: ‘Exp’.

Examples

>>> # To initialize a Exp bijector
>>> import mindspore.nn.probability.bijector as msb
>>> n = msb.Exp()
>>>
>>> # To use Exp distribution in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.e1 = msb.Exp()
>>>
>>>     def construct(self, value):
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'forward' with the name of the function
>>>         ans1 = self.s1.forward(value)
>>>         ans2 = self.s1.inverse(value)
>>>         ans3 = self.s1.forward_log_jacobian(value)
>>>         ans4 = self.s1.inverse_log_jacobian(value)

class mindspore.nn.probability.bijector.PowerTransform(power=0, name='PowerTransform', param=None)

Power Bijector. This Bijector performs the operation: Y = g(X) = (1 + X * c)^(1 / c), X >= -1 / c, where c >= 0 is the power.

The power transform maps inputs from [-1/c, inf] to [0, inf].

This bijector is equivalent to the Exp bijector when c=0

Raises

ValueError – If the power is less than 0 or is not known statically.

Parameters

• power (int or float) – scale factor. Default: 0.

• name (str) – name of the bijector. Default: ‘PowerTransform’.

Examples

>>> # To initialize a PowerTransform bijector of power 0.5
>>> import mindspore.nn.probability.bijector as msb
>>> n = msb.PowerTransform(0.5)
>>>
>>> # To use PowerTransform distribution in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.p1 = msb.PowerTransform(0.5)
>>>
>>>     def construct(self, value):
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'forward' with the name of the function
>>>         ans1 = self.s1.forward(value)
>>>         ans2 = self.s1.inverse(value)
>>>         ans3 = self.s1.forward_log_jacobian(value)
>>>         ans4 = self.s1.inverse_log_jacobian(value)

class mindspore.nn.probability.bijector.ScalarAffine(scale=1.0, shift=0.0, name='ScalarAffine')

Scalar Affine Bijector. This Bijector performs the operation: Y = a * X + b, where a is the scale factor and b is the shift factor.

Parameters

• scale (float) – scale factor. Default: 1.0.

• shift (float) – shift factor. Default: 0.0.

• name (str) – name of the bijector. Default: ‘ScalarAffine’.

Examples

>>> # To initialize a ScalarAffine bijector of scale 1 and shift 2
>>> scalaraffine = nn.probability.bijector.ScalarAffine(1, 2)
>>>
>>> # To use ScalarAffine bijector in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.s1 = nn.probability.bijector.ScalarAffine(1, 2)
>>>
>>>     def construct(self, value):
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'forward' with the name of the function
>>>         ans1 = self.s1.forward(value)
>>>         ans2 = self.s1.inverse(value)
>>>         ans3 = self.s1.forward_log_jacobian(value)
>>>         ans4 = self.s1.inverse_log_jacobian(value)

class mindspore.nn.probability.bijector.Softplus(sharpness=1.0, name='Softplus')

Softplus Bijector. This Bijector performs the operation, where k is the sharpness factor.

\[Y = \frac{\log(1 + e ^ {kX})}{k}\]

Parameters

• sharpness (float) – scale factor. Default: 1.0.

• name (str) – name of the bijector. Default: ‘Softplus’.

Examples

>>> # To initialize a Softplus bijector of sharpness 2
>>> softplus = nn.probability.bijector.Softfplus(2)
>>>
>>> # To use ScalarAffine bijector in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.sp1 = nn.probability.bijector.Softflus(2)
>>>
>>>     def construct(self, value):
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'forward' with the name of the function
>>>         ans1 = self.sp1.forward(value)
>>>         ans2 = self.sp1.inverse(value)
>>>         ans3 = self.sp1.forward_log_jacobian(value)
>>>         ans4 = self.sp1.inverse_log_jacobian(value)

mindspore.nn.probability.bnn_layers

Bayesian Layer.

The high-level components(Cells) used to construct the bayesian neural network.

class mindspore.nn.probability.bnn_layers.ConvReparam(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_prior_fn=NormalPrior, weight_posterior_fn=<lambda name, shape: NormalPosterior(name=name, shape=shape)>, bias_prior_fn=NormalPrior, bias_posterior_fn=<lambda name, shape: NormalPosterior(name=name, shape=shape)>)

Convolutional variational layers with Reparameterization.

See more details in paper Auto-Encoding Variational Bayes.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – The data type is int or tuple with 2 integers. Specifies the height and width of the 2D convolution window. Single int means the value if for both height and width of the kernel. A tuple of 2 ints means the first value is for the height and the other is for the width of the kernel.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number that represents the height and width of movement are both strides, or a tuple of two int numbers that represent height and width of movement respectively. Default: 1.

• pad_mode (str) –

  Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

  • same: Adopts the way of completion. Output height and width will be the same as the input. Total number of padding will be calculated for horizontal and vertical direction and evenly distributed to top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side. If this mode is set, padding must be 0.

  • valid: Adopts the way of discarding. The possibly largest height and width of output will be return without padding. Extra pixels will be discarded. If this mode is set, padding must be 0.

  • pad: Implicit paddings on both sides of the input. The number of padding will be padded to the input Tensor borders. padding should be greater than or equal to 0.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the input. Default: 0.

• dilation (Union[int, tuple[int]]) – The data type is int or tuple with 2 integers. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value should be greater or equal to 1 and bounded by the height and width of the input. Default: 1.

• group (int) – Split filter into groups, in_ channels and out_channels should be divisible by the number of groups. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_prior_fn – prior distribution for weight. It should return a mindspore distribution instance. Default: NormalPrior. (which creates an instance of standard normal distribution). The current version only supports normal distribution.

• weight_posterior_fn – posterior distribution for sampling weight. It should be a function handle which returns a mindspore distribution instance. Default: lambda name, shape: NormalPosterior(name=name, shape=shape). The current version only supports normal distribution.

• bias_prior_fn – prior distribution for bias vector. It should return a mindspore distribution. Default: NormalPrior(which creates an instance of standard normal distribution). The current version only supports normal distribution.

• bias_posterior_fn – posterior distribution for sampling bias vector. It should be a function handle which returns a mindspore distribution instance. Default: lambda name, shape: NormalPosterior(name=name, shape=shape). The current version only supports normal distribution.

Inputs:

• input (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Examples

>>> net = ConvReparam(120, 240, 4, has_bias=False)
>>> input = Tensor(np.ones([1, 120, 1024, 640]), mindspore.float32)
>>> net(input).shape
(1, 240, 1024, 640)

class mindspore.nn.probability.bnn_layers.DenseReparam(in_channels, out_channels, activation=None, has_bias=True, weight_prior_fn=NormalPrior, weight_posterior_fn=<lambda name, shape: NormalPosterior(name=name, shape=shape)>, bias_prior_fn=NormalPrior, bias_posterior_fn=<lambda name, shape: NormalPosterior(name=name, shape=shape)>)

Dense variational layers with Reparameterization.

See more details in paper Auto-Encoding Variational Bayes.

Applies dense-connected layer for the input. This layer implements the operation as:

\[\text{outputs} = \text{activation}(\text{inputs} * \text{kernel} + \text{bias}),\]

where \(\text{activation}\) is the activation function passed as the activation argument (if passed in), \(\text{activation}\) is a weight matrix with the same data type as the inputs created by the layer, \(\text{weight}\) is a weight matrix sampling from posterior distribution of weight, and \(\text{bias}\) is a bias vector with the same data type as the inputs created by the layer (only if has_bias is True). The bias vector is sampling from posterior distribution of \(\text{bias}\).

Parameters

• in_channels (int) – The number of input channel.

• out_channels (int) – The number of output channel .

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• activation (str, Cell) – Regularizer function applied to the output of the layer. The type of activation can be str (eg. ‘relu’) or Cell (eg. nn.ReLU()). Note that if the type of activation is Cell, it must have been instantiated. Default: None.

• weight_prior_fn – prior distribution for weight. It should return a mindspore distribution instance. Default: NormalPrior. (which creates an instance of standard normal distribution). The current version only supports normal distribution.

• weight_posterior_fn – posterior distribution for sampling weight. It should be a function handle which returns a mindspore distribution instance. Default: lambda name, shape: NormalPosterior(name=name, shape=shape). The current version only supports normal distribution.

• bias_prior_fn – prior distribution for bias vector. It should return a mindspore distribution. Default: NormalPrior(which creates an instance of standard normal distribution). The current version only supports normal distribution.

• bias_posterior_fn – posterior distribution for sampling bias vector. It should be a function handle which returns a mindspore distribution instance. Default: lambda name, shape: NormalPosterior(name=name, shape=shape). The current version only supports normal distribution.

Inputs:

• input (Tensor) - Tensor of shape \((N, in\_channels)\).

Outputs:

Tensor of shape \((N, out\_channels)\).

Examples

>>> net = DenseReparam(3, 4)
>>> input = Tensor(np.random.randint(0, 255, [2, 3]), mindspore.float32)
>>> net(input).shape
(2, 4)

class mindspore.nn.probability.bnn_layers.WithBNNLossCell(backbone, loss_fn, dnn_factor, bnn_factor)

Generate WithLossCell suitable for BNN.

Parameters

• backbone (Cell) – The target network.

• loss_fn (Cell) – The loss function used to compute loss.

• dnn_factor (int, float) – The coefficient of backbone’s loss, which is computed by loss functin. Default: 1.

• bnn_factor (int, float) – The coefficient of kl loss, which is kl divergence of Bayesian layer. Default: 1.

Inputs:

• data (Tensor) - Tensor of shape \((N, \ldots)\).

• label (Tensor) - Tensor of shape \((N, \ldots)\).

Outputs:

Tensor, a scalar tensor with shape \(()\).

Examples

>>> net = Net()
>>> loss_fn = nn.SoftmaxCrossEntropyWithLogits(is_grad=False, sparse=True)
>>> net_with_criterion_object = WithBNNLossCell(net, loss_fn)
>>> net_with_criterion = net_with_criterion_object()
>>>
>>> batch_size = 2
>>> data = Tensor(np.ones([batch_size, 3, 64, 64]).astype(np.float32) * 0.01)
>>> label = Tensor(np.ones([batch_size, 1, 1, 1]).astype(np.int32))
>>>
>>> net_with_criterion(data, label)

class mindspore.nn.probability.bnn_layers.NormalPosterior(name, shape, dtype=mindspore.float32, loc_mean=0, loc_std=0.1, untransformed_scale_mean=- 5, untransformed_scale_std=0.1)

Build Normal distributions with trainable parameters.

Parameters

• name (str) – Name prepended to trainable parameter.

• shape (list, tuple) – Shape of the mean and standard deviation.

• dtype (mindspore.dtype) – The argument is used to define the data type of the output tensor. Default: mindspore.float32.

• loc_mean (int, float) – Mean of distribution to initialize trainable parameters. Default: 0.

• loc_std (int, float) – Standard deviation of distribution to initialize trainable parameters. Default: 0.1.

• untransformed_scale_mean (int, float) – Mean of distribution to initialize trainable parameters. Default: -5.

• untransformed_scale_std (int, float) – Standard deviation of distribution to initialize trainable parameters. Default: 0.1.

Returns

Cell, a normal distribution.

std_trans(std_pre)

Transform std_pre to prevent its value being zero.

class mindspore.nn.probability.bnn_layers.NormalPrior(dtype=mindspore.float32, mean=0, std=0.1)

To initialize a normal distribution of mean 0 and standard deviation 0.1.

Parameters

• dtype (mindspore.dtype) – The argument is used to define the data type of the output tensor. Default: mindspore.float32.

• mean (int, float) – Mean of normal distribution. Default: 0.

• std (int, float) – Standard deviation of normal distribution. Default: 0.1.

Returns

Cell, a normal distribution.

mindspore.nn.probability.distribution

Distribution.

The high-level components(Distributions) used to construct the probabilistic network.

class mindspore.nn.probability.distribution.Bernoulli(probs=None, seed=0, dtype=mindspore.int32, name='Bernoulli')

Bernoulli Distribution.

Parameters

• probs (float, list, numpy.ndarray, Tensor, Parameter) – probability of 1 as outcome.

• seed (int) – seed to use in sampling. Default: 0.

• dtype (mindspore.dtype) – type of the distribution. Default: mstype.int32.

• name (str) – name of the distribution. Default: Bernoulli.

Note

probs should be proper probabilities (0 < p < 1). Dist_spec_args is probs.

Examples

>>> # To initialize a Bernoulli distribution of prob 0.5
>>> import mindspore.nn.probability.distribution as msd
>>> b = msd.Bernoulli(0.5, dtype=mstype.int32)
>>>
>>> # The following creates two independent Bernoulli distributions
>>> b = msd.Bernoulli([0.5, 0.5], dtype=mstype.int32)
>>>
>>> # A Bernoulli distribution can be initilized without arguments
>>> # In this case, probs must be passed in through args during function calls.
>>> b = msd.Bernoulli(dtype=mstype.int32)
>>>
>>> # To use Bernoulli in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.b1 = msd.Bernoulli(0.5, dtype=mstype.int32)
>>>         self.b2 = msd.Bernoulli(dtype=mstype.int32)
>>>
>>>     # All the following calls in construct are valid
>>>     def construct(self, value, probs_b, probs_a):
>>>
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'prob' with the name of the function
>>>         ans = self.b1.prob(value)
>>>         # Evaluate with the respect to distribution b
>>>         ans = self.b1.prob(value, probs_b)
>>>
>>>         # probs must be passed in during function calls
>>>         ans = self.b2.prob(value, probs_a)
>>>
>>>         # Functions 'sd', 'var', 'entropy' have the same usage as 'mean'
>>>         # Will return 0.5
>>>         ans = self.b1.mean()
>>>         # Will return probs_b
>>>         ans = self.b1.mean(probs_b)
>>>
>>>         # probs must be passed in during function calls
>>>         ans = self.b2.mean(probs_a)
>>>
>>>         # Usage of 'kl_loss' and 'cross_entropy' are similar
>>>         ans = self.b1.kl_loss('Bernoulli', probs_b)
>>>         ans = self.b1.kl_loss('Bernoulli', probs_b, probs_a)
>>>
>>>         # Additional probs_a must be passed in through
>>>         ans = self.b2.kl_loss('Bernoulli', probs_b, probs_a)
>>>
>>>         # Sample
>>>         ans = self.b1.sample()
>>>         ans = self.b1.sample((2,3))
>>>         ans = self.b1.sample((2,3), probs_b)
>>>         ans = self.b2.sample((2,3), probs_a)

property probs

Returns the probability for the outcome is 1.

class mindspore.nn.probability.distribution.Categorical(probs=None, logits=None, seed=0, dtype=mindspore.int32, name='Categorical')

Creates a categorical distribution parameterized by either probs or logits (but not both).

Parameters

• probs (Tensor, list, numpy.ndarray, Parameter, float) – event probabilities.

• logits (Tensor, list, numpy.ndarray, Parameter, float) – event log-odds.

• seed (int) – seed to use in sampling. Default: 0.

• dtype (mindspore.dtype) – type of the distribution. Default: mstype.int32.

• name (str) – name of the distribution. Default: Categorical.

Note

probs must be non-negative, finite and have a non-zero sum, and it will be normalized to sum to 1.

Examples

>>> # To initialize a Categorical distribution of prob is [0.5, 0.5]
>>> import mindspore.nn.probability.distribution as msd
>>> b = msd.Categorical(probs = [0.5, 0.5], dtype=mstype.int32)
>>>
>>> # To use Categorical in a network
>>> class net(Cell):
>>>     def __init__(self, probs):
>>>         super(net, self).__init__():
>>>         self.ca = msd.Categorical(probs=probs, dtype=mstype.int32)
>>>     # All the following calls in construct are valid
>>>     def construct(self, value):
>>>
>>>         # Similar calls can be made to logits
>>>         ans = self.ca.probs
>>>         # value should be Tensor
>>>         ans = self.ca.log_prob(value)
>>>
>>>         # Usage of enumerate_support
>>>         ans = self.ca.enumerate_support()
>>>
>>>         # Usage of entropy
>>>         ans = self.ca.entropy()
>>>
>>>         # Sample
>>>         ans = self.ca.sample()
>>>         ans = self.ca.sample((2,3))
>>>         ans = self.ca.sample((2,))

enumerate_support(expand=True)

Enumerate categories.

property logits

Returns the logits.

property probs

Returns the probability.

class mindspore.nn.probability.distribution.Distribution(seed, dtype, name, param)

Base class for all mathematical distributions.

Parameters

• seed (int) – random seed used in sampling.

• dtype (mindspore.dtype) – type of the distribution.

• name (str) – Python str name prefixed to Ops created by this class. Default: subclass name.

• param (dict) – parameters used to initialize the distribution.

Note

Derived class should override operations such as ,_mean, _prob, and _log_prob. Required arguments, such as value for _prob, should be passed in through args or kwargs. dist_spec_args which specify a new distribution are optional.

dist_spec_args are unique for each type of distribution. For example, mean and sd are the dist_spec_args for a Normal distribution, while rate is the dist_spec_args for exponential distribution.

For all functions, passing in dist_spec_args, is optional. Passing in the additional dist_spec_args will make the result to be evaluated with new distribution specified by the dist_spec_args. But it won’t change the original distribuion.

cdf(*args, **kwargs)

Evaluate the cdf at given value.

Note

Args must include value. dist_spec_args are optional.

construct(name, *args, **kwargs)

Override construct in Cell.

Note

Names of supported functions include: ‘prob’, ‘log_prob’, ‘cdf’, ‘log_cdf’, ‘survival_function’, ‘log_survival’ ‘var’, ‘sd’, ‘entropy’, ‘kl_loss’, ‘cross_entropy’, ‘sample’.

Parameters

• name (str) – name of the function.

• *args (list) – list of positional arguments needed for the function.

• **kwargs (dictionary) – dictionary of keyword arguments needed for the function.

cross_entropy(*args, **kwargs)

Evaluate the cross_entropy between distribution a and b.

Note

Args must include type of the distribution, parameters of distribution b. Parameters for distribution a are optional.

entropy(*args, **kwargs)

Evaluate the entropy.

Note

dist_spec_args are optional.

kl_loss(*args, **kwargs)

Evaluate the KL divergence, i.e. KL(a||b).

Note

Args must include type of the distribution, parameters of distribution b. Parameters for distribution a are optional.

log_cdf(*args, **kwargs)

Evaluate the log cdf at given value.

Note

Args must include value. dist_spec_args are optional.

log_prob(*args, **kwargs)

Evaluate the log probability(pdf or pmf) at the given value.

Note

Args must include value. dist_spec_args are optional.

log_survival(*args, **kwargs)

Evaluate the log survival function at given value.

Note

Args must include value. dist_spec_args are optional.

mean(*args, **kwargs)

Evaluate the mean.

Note

dist_spec_args are optional.

mode(*args, **kwargs)

Evaluate the mode.

Note

dist_spec_args are optional.

prob(*args, **kwargs)

Evaluate the probability (pdf or pmf) at given value.

Note

Args must include value. dist_spec_args are optional.

sample(*args, **kwargs)

Sampling function.

Note

Shape of the sample is default to (). dist_spec_args are optional.

sd(*args, **kwargs)

Evaluate the standard deviation.

Note

dist_spec_args are optional.

survival_function(*args, **kwargs)

Evaluate the survival function at given value.

Note

Args must include value. dist_spec_args are optional.

var(*args, **kwargs)

Evaluate the variance.

Note

dist_spec_args are optional.

class mindspore.nn.probability.distribution.Exponential(rate=None, seed=0, dtype=mindspore.float32, name='Exponential')

Example class: Exponential Distribution.

Parameters

• rate (float, list, numpy.ndarray, Tensor, Parameter) – inverse scale.

• seed (int) – seed to use in sampling. Default: 0.

• dtype (mindspore.dtype) – type of the distribution. Default: mstype.float32.

• name (str) – name of the distribution. Default: Exponential.

Note

rate should be strictly greater than 0. Dist_spec_args is rate.

Examples

>>> # To initialize an Exponential distribution of rate 0.5
>>> import mindspore.nn.probability.distribution as msd
>>> e = msd.Exponential(0.5, dtype=mstype.float32)
>>>
>>> # The following creates two independent Exponential distributions
>>> e = msd.Exponential([0.5, 0.5], dtype=mstype.float32)
>>>
>>> # An Exponential distribution can be initilized without arguments
>>> # In this case, rate must be passed in through args during function calls
>>> e = msd.Exponential(dtype=mstype.float32)
>>>
>>> # To use Exponential in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.e1 = msd.Exponential(0.5, dtype=mstype.float32)
>>>         self.e2 = msd.Exponential(dtype=mstype.float32)
>>>
>>>     # All the following calls in construct are valid
>>>     def construct(self, value, rate_b, rate_a):
>>>
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'prob' with the name of the function
>>>         ans = self.e1.prob(value)
>>>         # Evaluate with the respect to distribution b
>>>         ans = self.e1.prob(value, rate_b)
>>>
>>>         # Rate must be passed in during function calls
>>>         ans = self.e2.prob(value, rate_a)
>>>
>>>         # Functions 'sd', 'var', 'entropy' have the same usage as'mean'
>>>         # Will return 2
>>>         ans = self.e1.mean()
>>>         # Will return 1 / rate_b
>>>         ans = self.e1.mean(rate_b)
>>>
>>>         # Rate must be passed in during function calls
>>>         ans = self.e2.mean(rate_a)
>>>
>>>         # Usage of 'kl_loss' and 'cross_entropy' are similar
>>>         ans = self.e1.kl_loss('Exponential', rate_b)
>>>         ans = self.e1.kl_loss('Exponential', rate_b, rate_a)
>>>
>>>         # Additional rate must be passed in
>>>         ans = self.e2.kl_loss('Exponential', rate_b, rate_a)
>>>
>>>         # Sample
>>>         ans = self.e1.sample()
>>>         ans = self.e1.sample((2,3))
>>>         ans = self.e1.sample((2,3), rate_b)
>>>         ans = self.e2.sample((2,3), rate_a)

property rate

Return rate of the distribution.

class mindspore.nn.probability.distribution.Geometric(probs=None, seed=0, dtype=mindspore.int32, name='Geometric')

Geometric Distribution. It represents k+1 Bernoulli trials needed to get one success, k is the number of failures.

Parameters

• probs (float, list, numpy.ndarray, Tensor, Parameter) – probability of success.

• seed (int) – seed to use in sampling. Default: 0.

• dtype (mindspore.dtype) – type of the distribution. Default: mstype.int32.

• name (str) – name of the distribution. Default: Geometric.

Note

probs should be proper probabilities (0 < p < 1). Dist_spec_args is probs.

Examples

>>> # To initialize a Geometric distribution of prob 0.5
>>> import mindspore.nn.probability.distribution as msd
>>> n = msd.Geometric(0.5, dtype=mstype.int32)
>>>
>>> # The following creates two independent Geometric distributions
>>> n = msd.Geometric([0.5, 0.5], dtype=mstype.int32)
>>>
>>> # A Geometric distribution can be initilized without arguments
>>> # In this case, probs must be passed in through args during function calls.
>>> n = msd.Geometric(dtype=mstype.int32)
>>>
>>> # To use Geometric in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.g1 = msd.Geometric(0.5, dtype=mstype.int32)
>>>         self.g2 = msd.Geometric(dtype=mstype.int32)
>>>
>>>     # Tthe following calls are valid in construct
>>>     def construct(self, value, probs_b, probs_a):
>>>
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'prob' with the name of the function
>>>         ans = self.g1.prob(value)
>>>         # Evaluate with the respect to distribution b
>>>         ans = self.g1.prob(value, probs_b)
>>>
>>>         # Probs must be passed in during function calls
>>>         ans = self.g2.prob(value, probs_a)
>>>
>>>         # Functions 'sd', 'var', 'entropy' have the same usage as 'mean'
>>>         # Will return 1.0
>>>         ans = self.g1.mean()
>>>         # Another possible usage
>>>         ans = self.g1.mean(probs_b)
>>>
>>>         # Probs must be passed in during function calls
>>>         ans = self.g2.mean(probs_a)
>>>
>>>         # Usage of 'kl_loss' and 'cross_entropy' are similar
>>>         ans = self.g1.kl_loss('Geometric', probs_b)
>>>         ans = self.g1.kl_loss('Geometric', probs_b, probs_a)
>>>
>>>         # Additional probs must be passed in
>>>         ans = self.g2.kl_loss('Geometric', probs_b, probs_a)
>>>
>>>         # Sample
>>>         ans = self.g1.sample()
>>>         ans = self.g1.sample((2,3))
>>>         ans = self.g1.sample((2,3), probs_b)
>>>         ans = self.g2.sample((2,3), probs_a)

property probs

Returns the probability of success of the Bernoulli trail.

class mindspore.nn.probability.distribution.Normal(mean=None, sd=None, seed=0, dtype=mindspore.float32, name='Normal')

Normal distribution.

Parameters

• mean (int, float, list, numpy.ndarray, Tensor, Parameter) – mean of the Normal distribution.

• sd (int, float, list, numpy.ndarray, Tensor, Parameter) – stddev of the Normal distribution.

• seed (int) – seed to use in sampling. Default: 0.

• dtype (mindspore.dtype) – type of the distribution. Default: mstype.float32.

• name (str) – name of the distribution. Default: Normal.

Note

Standard deviation should be greater than zero. Dist_spec_args are mean and sd.

Examples

>>> # To initialize a Normal distribution of mean 3.0 and standard deviation 4.0
>>> import mindspore.nn.probability.distribution as msd
>>> n = msd.Normal(3.0, 4.0, dtype=mstype.float32)
>>>
>>> # The following creates two independent Normal distributions
>>> n = msd.Normal([3.0, 3.0], [4.0, 4.0], dtype=mstype.float32)
>>>
>>> # A Normal distribution can be initilize without arguments
>>> # In this case, mean and sd must be passed in through args.
>>> n = msd.Normal(dtype=mstype.float32)
>>>
>>> # To use Normal in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.n1 = msd.Nomral(0.0, 1.0, dtype=mstype.float32)
>>>         self.n2 = msd.Normal(dtype=mstype.float32)
>>>
>>>     # The following calls are valid in construct
>>>     def construct(self, value, mean_b, sd_b, mean_a, sd_a):
>>>
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'prob' with the name of the function
>>>         ans = self.n1.prob(value)
>>>         # Evaluate with the respect to distribution b
>>>         ans = self.n1.prob(value, mean_b, sd_b)
>>>
>>>         # mean and sd must be passed in during function calls
>>>         ans = self.n2.prob(value, mean_a, sd_a)
>>>
>>>         # Functions 'sd', 'var', 'entropy' have the same usage as 'mean'
>>>         # will return [0.0]
>>>         ans = self.n1.mean()
>>>         # will return mean_b
>>>         ans = self.n1.mean(mean_b, sd_b)
>>>
>>>         # mean and sd must be passed during function calls
>>>         ans = self.n2.mean(mean_a, sd_a)
>>>
>>>         # Usage of 'kl_loss' and 'cross_entropy' are similar
>>>         ans = self.n1.kl_loss('Normal', mean_b, sd_b)
>>>         ans = self.n1.kl_loss('Normal', mean_b, sd_b, mean_a, sd_a)
>>>
>>>         # Additional mean and sd must be passed
>>>         ans = self.n2.kl_loss('Normal', mean_b, sd_b, mean_a, sd_a)
>>>
>>>         # Sample
>>>         ans = self.n1.sample()
>>>         ans = self.n1.sample((2,3))
>>>         ans = self.n1.sample((2,3), mean_b, sd_b)
>>>         ans = self.n2.sample((2,3), mean_a, sd_a)

class mindspore.nn.probability.distribution.TransformedDistribution(bijector, distribution, dtype, seed=0, name='transformed_distribution')

Transformed Distribution. This class contains a bijector and a distribution and transforms the original distribution to a new distribution through the operation defined by the bijector.

Parameters

• bijector (Bijector) – transformation to perform.

• distribution (Distribution) – The original distribution.

• name (str) – name of the transformed distribution. Default: transformed_distribution.

Note

The arguments used to initialize the original distribution cannot be None. For example, mynormal = nn.Normal(dtype=dtyple.float32) cannot be used to initialized a TransformedDistribution since mean and sd are not specified.

Examples

>>> # To initialize a transformed distribution, e.g. lognormal distribution,
>>> # using Normal distribution as the base distribution, and Exp bijector as the bijector function.
>>> import mindspore.nn.probability.distribution as msd
>>> import mindspore.nn.probability.bijector as msb
>>> ln = msd.TransformedDistribution(msb.Exp(),
>>>                                  msd.Normal(0.0, 1.0, dtype=mstype.float32),
>>>                                  dtype=mstype.float32)
>>>
>>> # To use a transformed distribution in a network
>>> class net(Cell):
>>>     def __init__(self):
>>>         super(net, self).__init__():
>>>         self.ln = msd.TransformedDistribution(msb.Exp(),
>>>                                               msd.Normal(0.0, 1.0, dtype=mstype.float32),
>>>                                               dtype=mstype.float32)
>>>
>>>     def construct(self, value):
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'sample' with the name of the function
>>>         ans = self.ln.sample(shape=(2, 3))

class mindspore.nn.probability.distribution.Uniform(low=None, high=None, seed=0, dtype=mindspore.float32, name='Uniform')

Example class: Uniform Distribution.

Parameters

• low (int, float, list, numpy.ndarray, Tensor, Parameter) – lower bound of the distribution.

• high (int, float, list, numpy.ndarray, Tensor, Parameter) – upper bound of the distribution.

• seed (int) – seed to use in sampling. Default: 0.

• dtype (mindspore.dtype) – type of the distribution. Default: mstype.float32.

• name (str) – name of the distribution. Default: Uniform.

Note

low should be stricly less than high. Dist_spec_args are high and low.

Examples

>>> # To initialize a Uniform distribution of mean 3.0 and standard deviation 4.0
>>> import mindspore.nn.probability.distribution as msd
>>> u = msd.Uniform(0.0, 1.0, dtype=mstype.float32)
>>>
>>> # The following creates two independent Uniform distributions
>>> u = msd.Uniform([0.0, 0.0], [1.0, 2.0], dtype=mstype.float32)
>>>
>>> # A Uniform distribution can be initilized without arguments
>>> # In this case, high and low must be passed in through args during function calls.
>>> u = msd.Uniform(dtype=mstype.float32)
>>>
>>> # To use Uniform in a network
>>> class net(Cell):
>>>     def __init__(self)
>>>         super(net, self).__init__():
>>>         self.u1 = msd.Uniform(0.0, 1.0, dtype=mstype.float32)
>>>         self.u2 = msd.Uniform(dtype=mstype.float32)
>>>
>>>     # All the following calls in construct are valid
>>>     def construct(self, value, low_b, high_b, low_a, high_a):
>>>
>>>         # Similar calls can be made to other probability functions
>>>         # by replacing 'prob' with the name of the function
>>>         ans = self.u1.prob(value)
>>>         # Evaluate with the respect to distribution b
>>>         ans = self.u1.prob(value, low_b, high_b)
>>>
>>>         # High and low must be passed in during function calls
>>>         ans = self.u2.prob(value, low_a, high_a)
>>>
>>>         # Functions 'sd', 'var', 'entropy' have the same usage as 'mean'
>>>         # Will return 0.5
>>>         ans = self.u1.mean()
>>>         # Will return (low_b + high_b) / 2
>>>         ans = self.u1.mean(low_b, high_b)
>>>
>>>         # High and low must be passed in during function calls
>>>         ans = self.u2.mean(low_a, high_a)
>>>
>>>         # Usage of 'kl_loss' and 'cross_entropy' are similar
>>>         ans = self.u1.kl_loss('Uniform', low_b, high_b)
>>>         ans = self.u1.kl_loss('Uniform', low_b, high_b, low_a, high_a)
>>>
>>>         # Additional high and low must be passed
>>>         ans = self.u2.kl_loss('Uniform', low_b, high_b, low_a, high_a)
>>>
>>>         # Sample
>>>         ans = self.u1.sample()
>>>         ans = self.u1.sample((2,3))
>>>         ans = self.u1.sample((2,3), low_b, high_b)
>>>         ans = self.u2.sample((2,3), low_a, high_a)

property high

Return upper bound of the distribution.

property low

Return lower bound of the distribution.

mindspore.nn.probability.dpn

Deep Probability Network(dpn).

Deep probability network such as BNN and VAE network.

class mindspore.nn.probability.dpn.ConditionalVAE(encoder, decoder, hidden_size, latent_size, num_classes)

Conditional Variational Auto-Encoder (CVAE).

The difference with VAE is that CVAE uses labels information. see more details in Learning Structured Output Representation using Deep Conditional Generative Models.

Note

When define the encoder and decoder, the shape of the encoder’s output tensor and decoder’s input tensor should be \((N, hidden\_size)\). The latent_size should be less than or equal to the hidden_size.

Parameters

• encoder (Cell) – The DNN model defined as encoder.

• decoder (Cell) – The DNN model defined as decoder.

• hidden_size (int) – The size of encoder’s output tensor.

• latent_size (int) – The size of the latent space.

• num_classes (int) – The number of classes.

Inputs:

• input_x (Tensor) - the same shape as the input of encoder, the shape is \((N, C, H, W)\).

• input_y (Tensor) - the tensor of the target data, the shape is \((N,)\).

Outputs:

• output (tuple) - (recon_x(Tensor), x(Tensor), mu(Tensor), std(Tensor)).

construct(x, y)

The input are x and y, so the WithLossCell method needs to be rewritten when using cvae interface.

generate_sample(sample_y, generate_nums, shape)

Randomly sample from latent space to generate sample.

Parameters

• sample_y (Tensor) – Define the label of sample. Tensor of shape (generate_nums, ) and type mindspore.int32.

• generate_nums (int) – The number of samples to generate.

• shape (tuple) – The shape of sample, it should be (generate_nums, C, H, W) or (-1, C, H, W).

Returns

Tensor, the generated sample.

reconstruct_sample(x, y)

Reconstruct sample from original data.

Parameters

• x (Tensor) – The input tensor to be reconstructed, the shape is (N, C, H, W).

• y (Tensor) – The label of the input tensor, the shape is (N,).

Returns

Tensor, the reconstructed sample.

class mindspore.nn.probability.dpn.VAE(encoder, decoder, hidden_size, latent_size)

Variational Auto-Encoder (VAE).

The VAE defines a generative model, Z is sampled from the prior, then used to reconstruct X by a decoder. see more details in Auto-Encoding Variational Bayes.

Note

When define the encoder and decoder, the shape of the encoder’s output tensor and decoder’s input tensor should be \((N, hidden\_size)\). The latent_size should be less than or equal to the hidden_size.

Parameters

• encoder (Cell) – The DNN model defined as encoder.

• decoder (Cell) – The DNN model defined as decoder.

• hidden_size (int) – The size of encoder’s output tensor.

• latent_size (int) – The size of the latent space.

Inputs:

• input (Tensor) - the same shape as the input of encoder, the shape is \((N, C, H, W)\).

Outputs:

• output (Tuple) - (recon_x(Tensor), x(Tensor), mu(Tensor), std(Tensor)).

generate_sample(generate_nums, shape)

Randomly sample from latent space to generate sample.

Parameters

• generate_nums (int) – The number of samples to generate.

• shape (tuple) – The shape of sample, it should be (generate_nums, C, H, W) or (-1, C, H, W).

Returns

Tensor, the generated sample.

reconstruct_sample(x)

Reconstruct sample from original data.

Parameters

x (Tensor) – The input tensor to be reconstructed, the shape is (N, C, H, W).

Returns

Tensor, the reconstructed sample.

mindspore.nn.probability.infer

Infer algorithms in Probabilistic Programming.

class mindspore.nn.probability.infer.ELBO(latent_prior='Normal', output_prior='Normal')

The Evidence Lower Bound (ELBO).

Variational inference minimizes the Kullback-Leibler (KL) divergence from the variational distribution to the posterior distribution. It maximizes the evidence lower bound (ELBO), a lower bound on the logarithm of the marginal probability of the observations log p(x). The ELBO is equal to the negative KL divergence up to an additive constant. see more details in Variational Inference: A Review for Statisticians.

Parameters

• latent_prior (str) – The prior distribution of latent space. Default: Normal. - Normal: The prior distribution of latent space is Normal.

• output_prior (str) – The distribution of output data. Default: Normal. - Normal: If the distribution of output data is Normal, the reconstruct loss is MSELoss.

Inputs:

• input_data (Tuple) - (recon_x(Tensor), x(Tensor), mu(Tensor), std(Tensor)).

• target_data (Tensor) - the target tensor of shape \((N,)\).

Outputs:

Tensor, loss float tensor.

class mindspore.nn.probability.infer.SVI(net_with_loss, optimizer)

Stochastic Variational Inference(SVI).

Variational inference casts the inference problem as an optimization. Some distributions over the hidden variables that is indexed by a set of free parameters, and then optimize the parameters to make it closest to the posterior of interest. see more details in Variational Inference: A Review for Statisticians.

Parameters

• net_with_loss (Cell) – Cell with loss function.

• optimizer (Cell) – Optimizer for updating the weights.

get_train_loss()

Returns

numpy.dtype, the loss after training.

run(train_dataset, epochs=10)

Optimize the parameters by training the probability network, and return the trained network.

Parameters

• epochs (int) – Total number of iterations on the data. Default: 10.

• train_dataset (Dataset) – A training dataset iterator.

Outputs:

Cell, the trained probability network.

mindspore.nn.probability.toolbox

Uncertainty toolbox.

class mindspore.nn.probability.toolbox.UncertaintyEvaluation(model, train_dataset, task_type, num_classes=None, epochs=1, epi_uncer_model_path=None, ale_uncer_model_path=None, save_model=False)

Toolbox for Uncertainty Evaluation.

Parameters

• model (Cell) – The model for uncertainty evaluation.

• train_dataset (Dataset) – A dataset iterator to train model.

• task_type (str) – Option for the task types of model - regression: A regression model. - classification: A classification model.

• num_classes (int) – The number of labels of classification. If the task type is classification, it must be set; if not classification, it need not to be set. Default: None.

• epochs (int) – Total number of iterations on the data. Default: 1.

• epi_uncer_model_path (str) – The save or read path of the epistemic uncertainty model. Default: None.

• ale_uncer_model_path (str) – The save or read path of the aleatoric uncertainty model. Default: None.

• save_model (bool) – Save the uncertainty model or not, if True, the epi_uncer_model_path and ale_uncer_model_path should not be None. If False, give the path of the uncertainty model, it will load the model to evaluate, if not given the path, it will not save or load the uncertainty model. Default: False.

Examples

>>> network = LeNet()
>>> param_dict = load_checkpoint('checkpoint_lenet.ckpt')
>>> load_param_into_net(network, param_dict)
>>> ds_train = create_dataset('workspace/mnist/train')
>>> evaluation = UncertaintyEvaluation(model=network,
>>>                                    train_dataset=ds_train,
>>>                                    task_type='classification',
>>>                                    num_classes=10,
>>>                                    epochs=1,
>>>                                    epi_uncer_model_path=None,
>>>                                    ale_uncer_model_path=None,
>>>                                    save_model=False)
>>> epistemic_uncertainty = evaluation.eval_epistemic_uncertainty(eval_data)
>>> aleatoric_uncertainty = evaluation.eval_aleatoric_uncertainty(eval_data)
>>> epistemic_uncertainty.shape
(32, 10)
>>> aleatoric_uncertainty.shape
(32,)

eval_aleatoric_uncertainty(eval_data)

Evaluate the aleatoric uncertainty of inference results, which also called data uncertainty.

Parameters

eval_data (Tensor) – The data samples to be evaluated, the shape should be (N,C,H,W).

Returns

numpy.dtype, the aleatoric uncertainty of inference results of data samples.

eval_epistemic_uncertainty(eval_data)

Evaluate the epistemic uncertainty of inference results, which also called model uncertainty.

Parameters

eval_data (Tensor) – The data samples to be evaluated, the shape should be (N,C,H,W).

Returns

numpy.dtype, the epistemic uncertainty of inference results of data samples.

mindspore.nn.probability.transforms

Transforms.

The high-level components used to transform model between DNN and BNN.

class mindspore.nn.probability.transforms.TransformToBNN(trainable_dnn, dnn_factor=1, bnn_factor=1)

Transform Deep Neural Network (DNN) model to Bayesian Neural Network (BNN) model.

Parameters

• trainable_dnn (Cell) – A trainable DNN model (backbone) wrapped by TrainOneStepCell.

• dnn_factor ((int, float) – The coefficient of backbone’s loss, which is computed by loss function. Default: 1.

• bnn_factor (int, float) – The coefficient of kl loss, which is kl divergence of Bayesian layer. Default: 1.

Examples

>>> class Net(nn.Cell):
>>>     def __init__(self):
>>>         super(Net, self).__init__()
>>>         self.conv = nn.Conv2d(3, 64, 3, has_bias=False, weight_init='normal')
>>>         self.bn = nn.BatchNorm2d(64)
>>>         self.relu = nn.ReLU()
>>>         self.flatten = nn.Flatten()
>>>         self.fc = nn.Dense(64*224*224, 12) # padding=0
>>>
>>>     def construct(self, x):
>>>         x = self.conv(x)
>>>         x = self.bn(x)
>>>         x = self.relu(x)
>>>         x = self.flatten(x)
>>>         out = self.fc(x)
>>>         return out
>>>
>>> net = Net()
>>> criterion = nn.SoftmaxCrossEntropyWithLogits(is_grad=False, sparse=True)
>>> optim = Momentum(params=net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> net_with_loss = WithLossCell(network, criterion)
>>> train_network = TrainOneStepCell(net_with_loss, optim)
>>> bnn_transformer = TransformToBNN(train_network, 60000, 0.1)

transform_to_bnn_layer(dnn_layer_type, bnn_layer_type, get_args=None, add_args=None)

Transform a specific type of layers in DNN model to corresponding BNN layer.

Parameters

• dnn_layer_type (Cell) – The type of DNN layer to be transformed to BNN layer. The optional values are nn.Dense, nn.Conv2d.

• bnn_layer_type (Cell) – The type of BNN layer to be transformed to. The optional values are DenseReparam, ConvReparam.

• get_args – The arguments gotten from the DNN layer. Default: None.

• add_args (dict) – The new arguments added to BNN layer. Note that the arguments in add_args should not duplicate arguments in get_args. Default: None.

Returns

Cell, a trainable model wrapped by TrainOneStepCell, whose sprcific type of layer is transformed to the corresponding bayesian layer.

Examples

>>> net = Net()
>>> criterion = nn.SoftmaxCrossEntropyWithLogits(is_grad=False, sparse=True)
>>> optim = Momentum(params=net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> net_with_loss = WithLossCell(network, criterion)
>>> train_network = TrainOneStepCell(net_with_loss, optim)
>>> bnn_transformer = TransformToBNN(train_network, 60000, 0.1)
>>> train_bnn_network = bnn_transformer.transform_to_bnn_layer(Dense, DenseReparam)

transform_to_bnn_model(get_dense_args=<function TransformToBNN.<lambda>>, get_conv_args=<function TransformToBNN.<lambda>>, add_dense_args=None, add_conv_args=None)

Transform the whole DNN model to BNN model, and wrap BNN model by TrainOneStepCell.

Parameters

• get_dense_args – The arguments gotten from the DNN full connection layer. Default: lambda dp: {“in_channels”: dp.in_channels, “out_channels”: dp.out_channels, “has_bias”: dp.has_bias}.

• get_conv_args – The arguments gotten from the DNN convolutional layer. Default: lambda dp: {“in_channels”: dp.in_channels, “out_channels”: dp.out_channels, “pad_mode”: dp.pad_mode, “kernel_size”: dp.kernel_size, “stride”: dp.stride, “has_bias”: dp.has_bias}.

• add_dense_args (dict) – The new arguments added to BNN full connection layer. Note that the arguments in add_dense_args should not duplicate arguments in get_dense_args. Default: None.

• add_conv_args (dict) – The new arguments added to BNN convolutional layer. Note that the arguments in add_conv_args should not duplicate arguments in get_conv_args. Default: None.

Returns

Cell, a trainable BNN model wrapped by TrainOneStepCell.

Examples

>>> net = Net()
>>> criterion = nn.SoftmaxCrossEntropyWithLogits(is_grad=False, sparse=True)
>>> optim = Momentum(params=net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> net_with_loss = WithLossCell(network, criterion)
>>> train_network = TrainOneStepCell(net_with_loss, optim)
>>> bnn_transformer = TransformToBNN(train_network, 60000, 0.1)
>>> train_bnn_network = bnn_transformer.transform_to_bnn_model()