<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L238-L437) |

Wraps `tf_text.SentencepieceTokenizer` as a Keras Layer.

    tfm.nlp.layers.SentencepieceTokenizer(
        *,
        lower_case: bool,
        model_file_path: Optional[str] = None,
        model_serialized_proto: Optional[str] = None,
        tokenize_with_offsets: bool = False,
        nbest_size: int = 0,
        alpha: float = 1.0,
        strip_diacritics: bool = False,
        **kwargs
    )

<br />

<br />

<br />

| ## Args ||
|---|---|
| `lower_case` | A Python boolean indicating whether to lowercase the string before tokenization. NOTE: New models are encouraged to build `*_cf` (case folding) normalization into the Sentencepiece model itself and avoid this extra step. |
| `model_file_path` | A Python string with the path of the sentencepiece model. Exactly one of `model_file_path` and `model_serialized_proto` can be specified. In either case, the Keras model config for this layer will store the actual proto (not a filename passed here). |
| `model_serialized_proto` | The sentencepiece model serialized proto string. |
| `tokenize_with_offsets` | A Python boolean. If true, this layer calls [`SentencepieceTokenizer.tokenize_with_offsets()`](https://www.tensorflow.org/text/api_docs/python/text/SentencepieceTokenizer#tokenize_with_offsets) instead of plain `.tokenize()` and outputs a triple of `(tokens, start_offsets, limit_offsets)` insead of just tokens. Note that when following `strip_diacritics` is set to True, returning offsets is not supported now. |
| `nbest_size` | A scalar for sampling: nbest_size = {0,1}: No sampling is performed. (default) nbest_size \> 1: samples from the nbest_size results. nbest_size \< 0: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. |
| `alpha` | A scalar for a smoothing parameter. Inverse temperature for probability rescaling. |
| `strip_diacritics` | Whether to strip diacritics or not. Note that stripping diacritics requires additional text normalization and dropping bytes, which makes it impossible to keep track of the offsets now. Hence when `strip_diacritics` is set to True, we don't yet support `tokenize_with_offsets`. NOTE: New models are encouraged to put this into custom normalization rules for the Sentencepiece model itself to avoid this extra step and the limitation regarding offsets. |
| `**kwargs` | standard arguments to `Layer()`. |

<br />

<br />

<br />

<br />

| ## Raises ||
|---|---|
| `ImportError` | if importing tensorflow_text failed. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `tokenize_with_offsets` | If true, calls [`SentencepieceTokenizer.tokenize_with_offsets()`](https://www.tensorflow.org/text/api_docs/python/text/SentencepieceTokenizer#tokenize_with_offsets) instead of plain `.tokenize()` and outputs a triple of `(tokens, start_offsets, limit_offsets)`. |
| `vocab_size` | <br /> <br /> |

<br />

## Methods

### `call`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L330-L371)

    call(
        inputs: tf.Tensor
    )

Calls [`text.SentencepieceTokenizer`](https://www.tensorflow.org/text/api_docs/python/text/SentencepieceTokenizer) on inputs.

<br />

<br />

<br />

| Args ||
|---|---|
| `inputs` | A string Tensor of shape `(batch_size,)`. |

<br />

<br />

| Returns ||
|---|---|
| One or three of RaggedTensors if tokenize_with_offsets is False or True, respectively. These are ||
| `tokens` | A RaggedTensor of shape `[batch_size, (pieces)]` and type `int32`. `tokens[i,j]` contains the j-th piece in the i-th input. start_offsets, limit_offsets: If `tokenize_with_offsets` is True, RaggedTensors of type `int64` with the same indices as tokens. Element `[i,j]` contains the byte offset at the start, or past the end, resp., for the j-th piece in the i-th input. |

<br />

### `get_special_tokens_dict`

[View source](https://github.com/tensorflow/models/blob/v2.20.0/official/nlp/modeling/layers/text_layers.py#L377-L394)

    get_special_tokens_dict()

Returns dict of token ids, keyed by standard names for their purpose.

<br />

<br />

| Returns ||
|---|---|
| A dict from Python strings to Python integers. Each key is a standard name for a special token describing its use. (For example, "padding_id" is what Sentencepiece calls "" but others may call "\[PAD\]".) The corresponding value is the integer token id. If a special token is not found, its entry is omitted from the dict. <br /> The supported keys and tokens are: - start_of_sequence_id: looked up from "\[CLS\]" - end_of_segment_id: looked up from "\[SEP\]" - padding_id: looked up from "" - mask_id: looked up from "\[MASK\]" - vocab_size: one past the largest token id used ||

<br />