<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L605-L738) |

A bert tokenizer keras layer using text.FastWordpieceTokenizer.

    tfm.nlp.layers.FastWordpieceBertTokenizer(
        *,
        vocab_file: str,
        lower_case: bool,
        tokenize_with_offsets: bool = False,
        **kwargs
    )

See details: "Fast WordPiece Tokenization" (<https://arxiv.org/abs/2012.15524>)

<br />

<br />

<br />

| ## Args ||
|---|---|
| `vocab_file` | A Python string with the path of the vocabulary file. This is a text file with newline-separated wordpiece tokens. This layer loads a list of tokens from it to create text.FastWordpieceTokenizer. |
| `lower_case` | A Python boolean forwarded to text.BasicTokenizer. If true, input text is converted to lower case (where applicable) before tokenization. This must be set to match the way in which the vocab_file was created. |
| `tokenize_with_offsets` | A Python boolean. If true, this layer calls FastWordpieceTokenizer.tokenize_with_offsets() instead of plain .tokenize() and outputs a triple of (tokens, start_offsets, limit_offsets) insead of just tokens. |
| `**kwargs` | standard arguments to Layer(). |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `vocab_size` | <br /> <br /> |

<br />

## Methods

### `call`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L707-L738)

    call(
        inputs: tf.Tensor
    )

Calls text.BertTokenizer on inputs.

<br />

<br />

<br />

| Args ||
|---|---|
| `inputs` | A string Tensor of shape \[batch_size\]. |

<br />

<br />

| Returns ||
|---|---|
| One or three of RaggedTensors if tokenize_with_offsets is False or True, respectively. These are ||
| `tokens` | A RaggedTensor of shape \[batch_size, (words), (pieces_per_word)\] and type int32. tokens\[i,j,k\] contains the k-th wordpiece of the j-th word in the i-th input. start_offsets, limit_offsets: If tokenize_with_offsets is True, RaggedTensors of type int64 with the same indices as tokens. Element \[i,j,k\] contains the byte offset at the start, or past the end, resp., for the k-th wordpiece of the j-th word in the i-th input. |

<br />

### `get_special_tokens_dict`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L655-L672)

    get_special_tokens_dict()

Returns dict of token ids, keyed by standard names for their purpose.

<br />

<br />

| Returns ||
|---|---|
| A dict from Python strings to Python integers. Each key is a standard name for a special token describing its use. (For example, "padding_id" is what BERT traditionally calls "\[PAD\]" but others may call "".) The corresponding value is the integer token id. If a special token is not found, its entry is omitted from the dict. <br /> The supported keys and tokens are: - start_of_sequence_id: looked up from "\[CLS\]" - end_of_segment_id: looked up from "\[SEP\]" - padding_id: looked up form "\[PAD\]" - mask_id: looked up from "\[MASK\]" - vocab_size: one past the largest token id used ||

<br />