tf.keras.preprocessing.text.text_to_word_sequence 

tf.keras.preprocessing.text.text_to_word_sequence(
    text,
    filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
    lower=True,
    split=' '
)

Defined in tensorflow/python/keras/preprocessing/text.py.

Converts a text to a sequence of words (or tokens).

Arguments:

  • text: Input text (string).
  • filters: list (or concatenation) of characters to filter out, such as punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n', includes basic punctuation, tabs, and newlines.
  • lower: boolean, whether to convert the input to lowercase.
  • split: string, separator for word splitting.

Returns:

A list of words (or tokens).