<br />

|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/deprecated/text/text_encoder.py#L358-L437) |

Splits a string into tokens, and joins them back.

    tfds.deprecated.text.Tokenizer(
        alphanum_only=True, reserved_tokens=None
    )

<br />

<br />

<br />

| ## Args ||
|---|---|
| `alphanum_only` | `bool`, if `True`, only parse out alphanumeric tokens (non-alphanumeric characters are dropped); otherwise, keep all characters (individual tokens will still be either all alphanumeric or all non-alphanumeric). |
| `reserved_tokens` | `list<str>`, a list of strings that, if any are in `s`, will be preserved as whole tokens, even if they contain mixed alphanumeric/non-alphanumeric characters. |

<br />

<br />

<br />

<br />

| ## Attributes ||
|---|---|
| `alphanum_only` | <br /> <br /> |
| `reserved_tokens` | <br /> <br /> |

<br />

## Methods

### `join`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/deprecated/text/text_encoder.py#L413-L419)

    join(
        tokens
    )

Joins tokens into a string.

### `load_from_file`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/deprecated/text/text_encoder.py#L433-L437)

    @classmethod
    load_from_file(
        filename_prefix
    )

### `save_to_file`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/deprecated/text/text_encoder.py#L425-L431)

    save_to_file(
        filename_prefix
    )

### `tokenize`

[View source](https://github.com/tensorflow/datasets/blob/v4.9.10/tensorflow_datasets/core/deprecated/text/text_encoder.py#L390-L411)

    tokenize(
        s
    )

Splits a string into tokens.