TensorFlow 2.0 Beta is available Learn more

tfds.features.text.ByteTextEncoder

Class ByteTextEncoder

Byte-encodes text.

Inherits From: TextEncoder

View source

__init__

View source

__init__(additional_tokens=None)

Constructs ByteTextEncoder.

Args:

  • additional_tokens: list<str>, list of additional tokens. These will be assigned vocab ids [1, 1+len(additional_tokens)]. Useful for things like "end-of-string" tokens (e.g. "").

Properties

additional_tokens

vocab_size

Methods

decode

View source

decode(ids)

encode

View source

encode(s)

load_from_file

View source

@classmethod
load_from_file(
    cls,
    filename_prefix
)

save_to_file

View source

save_to_file(filename_prefix)