|---|
| [![](https://www.tensorflow.org/images/GitHub-Mark-32px.png) View source on GitHub](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L194-L575) |

Text tokenization utility class.

    tf.keras.preprocessing.text.Tokenizer(
        num_words=None,
        filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
        lower=True,
        split=' ',
        char_level=False,
        oov_token=None,
        analyzer=None,
        **kwargs
    )

> [!WARNING]
> **Deprecated:** [`tf.keras.preprocessing.text.Tokenizer`](https://www.tensorflow.org/versions/r2.15/api_docs/python/tf/keras/preprocessing/text/Tokenizer) does not operate on tensors and is not recommended for new code. Prefer [`tf.keras.layers.TextVectorization`](https://www.tensorflow.org/versions/r2.15/api_docs/python/tf/keras/layers/TextVectorization) which provides equivalent functionality through a layer which accepts [`tf.Tensor`](https://www.tensorflow.org/versions/r2.15/api_docs/python/tf/Tensor) input. See the [text loading tutorial](https://www.tensorflow.org/tutorials/load_data/text) for an overview of the layer and text handling in tensorflow.

This class allows to vectorize a text corpus, by turning each
text into either a sequence of integers (each integer being the index
of a token in a dictionary) or into a vector where the coefficient
for each token could be binary, based on word count, based on tf-idf...

By default, all punctuation is removed, turning the texts into
space-separated sequences of words
(words may include the `'` character). These sequences are then
split into lists of tokens. They will then be indexed or vectorized.

`0` is a reserved index that won't be assigned to any word.

<br />

<br />

<br />

| ## Args ||
|---|---|
| `num_words` | the maximum number of words to keep, based on word frequency. Only the most common `num_words-1` words will be kept. |
| `filters` | a string where each element is a character that will be filtered from the texts. The default is all punctuation, plus tabs and line breaks, minus the `'` character. |
| `lower` | boolean. Whether to convert the texts to lowercase. |
| `split` | str. Separator for word splitting. |
| `char_level` | if True, every character will be treated as a token. |
| `oov_token` | if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls |
| `analyzer` | function. Custom analyzer to split the text. The default analyzer is text_to_word_sequence |

<br />

## Methods

### `fit_on_sequences`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L329-L343)

    fit_on_sequences(
        sequences
    )

Updates internal vocabulary based on a list of sequences.

Required before using `sequences_to_matrix`
(if `fit_on_texts` was never called).

<br />

<br />

<br />

| Args ||
|---|---|
| `sequences` | A list of sequence. A "sequence" is a list of integer word indices. |

<br />

### `fit_on_texts`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L269-L327)

    fit_on_texts(
        texts
    )

Updates internal vocabulary based on a list of texts.

In the case where texts contains lists,
we assume each entry of the lists to be a token.

Required before using `texts_to_sequences` or `texts_to_matrix`.

<br />

<br />

<br />

| Args ||
|---|---|
| `texts` | can be a list of strings, a generator of strings (for memory-efficiency), or a list of list of strings. |

<br />

### `get_config`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L526-L555)

    get_config()

Returns the tokenizer configuration as Python dictionary.

The word count dictionaries used by the tokenizer get serialized
into plain JSON, so that the configuration can be read by other
projects.

<br />

<br />

<br />

| Returns ||
|---|---|
| A Python dictionary with the tokenizer configuration. ||

<br />

### `sequences_to_matrix`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L466-L524)

    sequences_to_matrix(
        sequences, mode='binary'
    )

Converts a list of sequences into a Numpy matrix.

<br />

<br />

<br />

| Args ||
|---|---|
| `sequences` | list of sequences (a sequence is a list of integer word indices). |
| `mode` | one of "binary", "count", "tfidf", "freq" |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A Numpy matrix. ||

<br />

<br />

<br />

<br />

| Raises ||
|---|---|
| `ValueError` | In case of invalid `mode` argument, or if the Tokenizer requires to be fit to sample data. |

<br />

### `sequences_to_texts`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L407-L419)

    sequences_to_texts(
        sequences
    )

Transforms each sequence into a list of text.

Only top `num_words-1` most frequent words will be taken into account.
Only words known by the tokenizer will be taken into account.

<br />

<br />

<br />

| Args ||
|---|---|
| `sequences` | A list of sequences (list of integers). |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A list of texts (strings) ||

<br />

### `sequences_to_texts_generator`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L421-L451)

    sequences_to_texts_generator(
        sequences
    )

Transforms each sequence in `sequences` to a list of texts(strings).

Each sequence has to a list of integers.
In other words, sequences should be a list of sequences

Only top `num_words-1` most frequent words will be taken into account.
Only words known by the tokenizer will be taken into account.

<br />

<br />

<br />

| Args ||
|---|---|
| `sequences` | A list of sequences. |

<br />

<br />

<br />

<br />

| Yields ||
|---|---|
| Yields individual texts. ||

<br />

### `texts_to_matrix`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L453-L464)

    texts_to_matrix(
        texts, mode='binary'
    )

Convert a list of texts to a Numpy matrix.

<br />

<br />

<br />

| Args ||
|---|---|
| `texts` | list of strings. |
| `mode` | one of "binary", "count", "tfidf", "freq". |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A Numpy matrix. ||

<br />

### `texts_to_sequences`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L345-L357)

    texts_to_sequences(
        texts
    )

Transforms each text in texts to a sequence of integers.

Only top `num_words-1` most frequent words will be taken into account.
Only words known by the tokenizer will be taken into account.

<br />

<br />

<br />

| Args ||
|---|---|
| `texts` | A list of texts (strings). |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A list of sequences. ||

<br />

### `texts_to_sequences_generator`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L359-L405)

    texts_to_sequences_generator(
        texts
    )

Transforms each text in `texts` to a sequence of integers.

Each item in texts can also be a list,
in which case we assume each item of that list to be a token.

Only top `num_words-1` most frequent words will be taken into account.
Only words known by the tokenizer will be taken into account.

<br />

<br />

<br />

| Args ||
|---|---|
| `texts` | A list of texts (strings). |

<br />

<br />

<br />

<br />

| Yields ||
|---|---|
| Yields individual sequences. ||

<br />

### `to_json`

[View source](https://github.com/keras-team/keras/tree/v2.15.0/keras/preprocessing/text.py#L557-L575)

    to_json(
        **kwargs
    )

Returns a JSON string containing the tokenizer configuration.

To load a tokenizer from a JSON string, use
[`keras.preprocessing.text.tokenizer_from_json(json_string)`](https://www.tensorflow.org/versions/r2.15/api_docs/python/tf/keras/preprocessing/text/tokenizer_from_json).

<br />

<br />

<br />

| Args ||
|---|---|
| `**kwargs` | Additional keyword arguments to be passed to `json.dumps()`. |

<br />

<br />

<br />

<br />

| Returns ||
|---|---|
| A JSON string containing the tokenizer configuration. ||

<br />