Have a question? Connect with the community at the TensorFlow Forum Visit Forum


Masked language model network head for BERT modeling.

This layer implements a masked language model based on the provided transformer based encoder. It assumes that the encoder network being passed has a "get_embedding_table()" method.



embedding_table The embedding table from encoder network.
activation The activation, if any, for the dense layer.
initializer The initializer for the dense layer. Defaults to a Glorot uniform initializer.
output The output style for this layer. Can be either 'logits' or 'predictions'.



This is where the layer's logic lives.

Note here that call() method in tf.keras is little bit different from keras API. In keras API, you can pass support masking for layers as additional arguments. Whereas tf.keras has compute_mask() method to support masking.

inputs Input tensor, or list/tuple of input tensors.
*args Additional positional arguments. Currently unused.
**kwargs Additional keyword arguments. Currently unused.

A tensor or list/tuple of tensors.