tfa.activations.gelu

Gaussian Error Linear Unit.

Computes gaussian error linear:

$$ \mathrm{gelu}(x) = x \Phi(x), $$

where

$$ \Phi(x) = \frac{1}{2} \left[ 1 + \mathrm{erf}(\frac{x}{\sqrt{2} }) \right]$ $$

when approximate is False; or

$$ \Phi(x) = \frac{x}{2} \left[ 1 + \tanh(\sqrt{\frac{2}{\pi} } \cdot (x + 0.044715 \cdot x^3)) \right] $$

when approximate is True.

See Gaussian Error Linear Units (GELUs) and BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.

Note that approximate will default to False from TensorFlow version 2.4 onwards. Consider using tf.nn.gelu instead.

Usage:

x = tf.constant([-1.0, 0.0, 1.0])
tfa.activations.gelu(x, approximate=False)
<tf.Tensor: shape=(3,), dtype=float32, numpy=array([-0.15865529,  0.        ,  0.8413447 ], dtype=float32)>
tfa.activations.gelu(x, approximate=True)
<tf.Tensor: shape=(3,), dtype=float32, numpy=array([-0.15880796,  0.        ,  0.841192  ], dtype=float32)>

x A Tensor. Must be one of the following types: float16, float32, float64.
approximate bool, whether to enable approximation.

A Tensor. Has the same type as x.