Apply to speak at TensorFlow World. Deadline April 23rd. Propose talk

tfds.features.text.ByteTextEncoder

Class ByteTextEncoder

Inherits From: TextEncoder

Defined in core/features/text/text_encoder.py.

Byte-encodes text.

__init__

__init__(additional_tokens=None)

Constructs ByteTextEncoder.

Args:

  • additional_tokens: list<str>, list of additional tokens. These will be assigned vocab ids [1, 1+len(additional_tokens)]. Useful for things like "end-of-string" tokens (e.g. "").

Properties

additional_tokens

vocab_size

Methods

decode

decode(ids)

encode

encode(s)

load_from_file

@classmethod
load_from_file(
    cls,
    filename_prefix
)

save_to_file

save_to_file(filename_prefix)