<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L61-L235) |

Wraps TF.Text's BertTokenizer with pre-defined vocab as a Keras Layer.

    tfm.nlp.layers.BertTokenizer(
        *,
        vocab_file: str,
        lower_case: Optional[bool] = None,
        tokenize_with_offsets: bool = False,
        tokenizer_kwargs: Optional[Mapping[Text, Any]] = None,
        **kwargs
    )

<br />

<br />

<br />

| ## Args ||
|---|---|
| `vocab_file` | A Python string with the path of the vocabulary file. This is a text file with newline-separated wordpiece tokens. This layer initializes a lookup table from it that gets used with [`text.BertTokenizer`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer). |
| `lower_case` | Optional boolean forwarded to [`text.BertTokenizer`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer). If true, input text is converted to lower case (where applicable) before tokenization. This must be set to match the way in which the `vocab_file` was created. If passed, this overrides whatever value may have been passed in `tokenizer_kwargs`. |
| `tokenize_with_offsets` | A Python boolean. If true, this layer calls [`text.BertTokenizer.tokenize_with_offsets()`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer#tokenize_with_offsets) instead of plain [`text.BertTokenizer.tokenize()`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer#tokenize) and outputs a triple of `(tokens, start_offsets, limit_offsets)` insead of just tokens. |
| `tokenizer_kwargs` | Optional mapping with keyword arguments to forward to [`text.BertTokenizer`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer)'s constructor. |
| `**kwargs` | Standard arguments to `Layer()`. |

<br />

<br />

<br />

<br />

| ## Raises ||
|---|---|
| `ImportError` | If importing `tensorflow_text` failed. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|---|
| `tokenize_with_offsets` | If true, calls [`text.BertTokenizer.tokenize_with_offsets()`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer#tokenize_with_offsets) instead of plain [`text.BertTokenizer.tokenize()`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer#tokenize) and outputs a triple of `(tokens, start_offsets, limit_offsets)`. |
| `raw_table_access` | An object with methods `.lookup(keys) and`.size()`that operate on the raw lookup table of tokens. It can be used to look up special token synbols like`\[MASK\]`. </td> </tr><tr> <td>`vocab_size\` | <br /> <br /> |

<br />

## Methods

### `call`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L138-L173)

    call(
        inputs: tf.Tensor
    )

Calls [`text.BertTokenizer`](https://www.tensorflow.org/text/api_docs/python/text/BertTokenizer) on inputs.

<br />

<br />

<br />

| Args ||
|---|---|
| `inputs` | A string Tensor of shape `(batch_size,)`. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| One or three of `RaggedTensors` if `tokenize_with_offsets` is False or True, respectively. These are tokens: A `RaggedTensor` of shape `[batch_size, (words), (pieces_per_word)]` and type int32. `tokens[i,j,k]` contains the k-th wordpiece of the j-th word in the i-th input. start_offsets, limit_offsets: If `tokenize_with_offsets` is True, RaggedTensors of type int64 with the same indices as tokens. Element `[i,j,k]` contains the byte offset at the start, or past the end, resp., for the k-th wordpiece of the j-th word in the i-th input. ||

<br />

### `get_special_tokens_dict`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L179-L196)

    get_special_tokens_dict()

Returns dict of token ids, keyed by standard names for their purpose.

<br />

<br />

| Returns ||
|---|---|
| A dict from Python strings to Python integers. Each key is a standard name for a special token describing its use. (For example, "padding_id" is what BERT traditionally calls "\[PAD\]" but others may call "".) The corresponding value is the integer token id. If a special token is not found, its entry is omitted from the dict. <br /> The supported keys and tokens are: - start_of_sequence_id: looked up from "\[CLS\]" - end_of_segment_id: looked up from "\[SEP\]" - padding_id: looked up form "\[PAD\]" - mask_id: looked up from "\[MASK\]" - vocab_size: one past the largest token id used ||

<br />