<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/rezero_transformer.py#L29-L360) |

Transformer layer with ReZero.

    tfm.nlp.layers.ReZeroTransformer(
        num_attention_heads,
        inner_dim=768,
        inner_activation=https://www.tensorflow.org/api_docs/python/tfm/utils/activations/gelu,
        dropout_rate=0.0,
        attention_dropout_rate=0.0,
        output_range=None,
        kernel_initializer='glorot_uniform',
        bias_initializer='zeros',
        kernel_regularizer=None,
        bias_regularizer=None,
        activity_regularizer=None,
        kernel_constraint=None,
        bias_constraint=None,
        use_layer_norm=False,
        share_rezero=True,
        num_kv_heads=None,
        src_block_size=None,
        tgt_block_size=None,
        linformer_dim=None,
        linformer_shared_kv_projection=True,
        use_sigmoid_attn=False,
        sigmoid_attn_bias=None,
        **kwargs
    )

This layer implements the Transformer from "Attention Is All You Need".
(<https://arxiv.org/abs/1706.03762>).
The residual connection implements the ReZero method.
(<https://arxiv.org/abs/2003.04887>)

<br />

<br />

<br />

| ## Args ||
|---|---|
| `num_attention_heads` | Number of attention heads. |
| `inner_dim` | The output dimension of the first Dense layer in a two-layer feedforward network. |
| `inner_activation` | The activation for the first Dense layer in a two-layer feedforward network. |
| `dropout_rate` | Dropout probability for the post-attention and output dropout. |
| `attention_dropout_rate` | Dropout probability for within the attention layer. |
| `output_range` | the sequence output range, \[0, output_range) by slicing the target sequence. `None` means the target sequence is not sliced. |
| `kernel_initializer` | Initializer for dense layer kernels. |
| `bias_initializer` | Initializer for dense layer biases. |
| `kernel_regularizer` | Regularizer for dense layer kernels. |
| `bias_regularizer` | Regularizer for dense layer biases. |
| `activity_regularizer` | Regularizer for dense layer activity. |
| `kernel_constraint` | Constraint for dense layer kernels. |
| `bias_constraint` | Constraint for dense layer kernels. |
| `use_layer_norm` | If add layer_norm on top of the ReZero. |
| `share_rezero` | If attention layer and FFN layer share the same alpha. |
| `num_kv_heads` | Number of key-value heads for multi-query attention. Refer to `multi_query_attention.MultiHeadAttention` for more details. |
| `src_block_size` | Source block size. Refer to `block_sparse_attention.MultiHeadAttention` for more details. |
| `tgt_block_size` | Target block size. Refer to `block_sparse_attention.MultiHeadAttention` for more details. |

<br />

## Methods

### `call`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/rezero_transformer.py#L315-L360)

    call(
        inputs, output_range: Optional[tf.Tensor] = None
    ) -> tf.Tensor

This is where the layer's logic lives.

The `call()` method may not create state (except in its first
invocation, wrapping the creation of variables or other resources in
[`tf.init_scope()`](https://www.tensorflow.org/api_docs/python/tf/init_scope)). It is recommended to create state, including
[`tf.Variable`](https://www.tensorflow.org/api_docs/python/tf/Variable) instances and nested `Layer` instances,
in `__init__()`, or in the `build()` method that is
called automatically before `call()` executes for the first time.

<br />

<br />

<br />

| Args ||
|---|---|
| `inputs` | Input tensor, or dict/list/tuple of input tensors. The first positional `inputs` argument is subject to special rules: <br /> - `inputs` must be explicitly passed. A layer cannot have zero arguments, and `inputs` cannot be provided via the default value of a keyword argument. - NumPy array or Python scalar values in `inputs` get cast as tensors. - TF-Keras mask metadata is only collected from `inputs`. - Layers are built (`build(input_shape)` method) using shape info from `inputs` only. - `input_spec` compatibility is only checked against `inputs`. - Mixed precision input casting is only applied to `inputs`. If a layer has tensor arguments in `*args` or `**kwargs`, their casting behavior in mixed precision should be handled manually. - The SavedModel input specification is generated using `inputs` only. - Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for `inputs` and not for tensors in positional and keyword arguments. |
| `*args` | Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above. |
| `**kwargs` | Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - `training`: Boolean scalar tensor of Python boolean indicating whether the `call` is meant for training or inference. - `mask`: Boolean input mask. If the layer's `call()` method takes a `mask` argument, its default value will be set to the mask generated for `inputs` by the previous layer (if `input` did come from a layer that generated a corresponding mask, i.e. if it came from a TF-Keras layer with masking support). |

<br />

<br />

<br />

| Returns ||
|---|---|
| A tensor or list/tuple of tensors. ||

<br />

### `reset_rezero`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/rezero_transformer.py#L310-L313)

    reset_rezero()