Have a question? Connect with the community at the TensorFlow Forum Visit Forum

tfnlp.layers.RelativePositionBias

Relative position embedding via per-head bias in T5 style.

Reference implementation in MeshTF: https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/transformer_layers.py#L1000

This layer implements the relative position bias used in "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer" (https://arxiv.org/abs/1910.10683)

Methods

call

View source

Implements the forward pass.

Args
query query input tensor shape [batch, query length, hidden size].
key key input tensor shape [batch, key length, hidden size].

Returns
A tensor in shape of [batch, heads, query length, key length].