mindspore.ops.gelu

mindspore.ops.gelu(input, approximate='none')[source]

Gaussian Error Linear Units activation function.

Warning

After version 2.9.0, the approximate argument will become keyword-only.

GeLU is described in the paper Gaussian Error Linear Units (GELUs). See also BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

When approximate argument is 'none', GeLU is defined as follows:

\[GELU(x_i) = x_i*P(X < x_i),\]

where \(P\) is the cumulative distribution function of the standard Gaussian distribution, \(x_i\) is the input element.

When approximate argument is 'tanh', GeLU is estimated with:

\[GELU(x_i) = 0.5 * x_i * (1 + \tanh(\sqrt{\frac{2}{\pi}} * (x_i + 0.044715 * x_i^3)))\]

GELU Activation Function Graph:

../../_images/GELU.png
Parameters

  • input (Tensor) – The input of the activation function GeLU.

  • approximate (str, optional) – the gelu approximation algorithm to use. Acceptable values are 'none' and 'tanh' . Default: 'none' .

Returns

Tensor, with the same type and shape as input.

Raises

  • TypeError – If input is not a Tensor.

  • ValueError – If approximate value is neither 'none' nor 'tanh'.

Supported Platforms:

Ascend GPU CPU

Examples

>>> import mindspore
>>> from mindspore import Tensor, ops
>>> x = Tensor([1.0, 2.0, 3.0], mindspore.float32)
>>> result = ops.gelu(x, approximate='none')
>>> print(result)
[0.8413447 1.9544997 2.9959505]