![]() |
Transformer encoder.
tfm.nlp.models.TransformerEncoder(
num_layers=6,
num_attention_heads=8,
intermediate_size=2048,
activation='relu',
dropout_rate=0.0,
attention_dropout_rate=0.0,
use_bias=False,
norm_first=True,
norm_epsilon=1e-06,
intermediate_dropout=0.0,
**kwargs
)
Transformer encoder is made up of N identical layers. Each layer is composed of the sublayers:
- Self-attention layer
- Feedforward network (which is 2 fully-connected layers)
Methods
call
call(
encoder_inputs, attention_mask=None
)
Return the output of the encoder.
Args | |
---|---|
encoder_inputs
|
A tensor with shape (batch_size, input_length,
hidden_size) .
|
attention_mask
|
A mask for the encoder self-attention layer with shape
(batch_size, input_length, input_length) .
|
Returns | |
---|---|
Output of encoder which is a float32 tensor with shape
(batch_size, input_length, hidden_size) .
|