Missed TensorFlow Dev Summit? Check out the video playlist. Watch recordings

tfds.features.text.ByteTextEncoder

View source on GitHub

Byte-encodes text.

Inherits From: TextEncoder

tfds.features.text.ByteTextEncoder(
    additional_tokens=None
)

Args:

  • additional_tokens: list<str>, list of additional tokens. These will be assigned vocab ids [1, 1+len(additional_tokens)]. Useful for things like "end-of-string" tokens (e.g. "").

Attributes:

  • additional_tokens
  • vocab_size: Size of the vocabulary. Decode produces ints [1, vocab_size).

Methods

decode

View source

decode(
    ids
)

Decodes a list of integers into text.

encode

View source

encode(
    s
)

Encodes text into a list of integers.

load_from_file

View source

@classmethod
load_from_file(
    cls, filename_prefix
)

Load from file. Inverse of save_to_file.

save_to_file

View source

save_to_file(
    filename_prefix
)

Store to file. Inverse of load_from_file.