|View source on GitHub|
Maps strings from a vocabulary to integer indices.
tf.keras.layers.experimental.preprocessing.StringLookup( max_tokens=None, num_oov_indices=1, mask_token='', oov_token='[UNK]', vocabulary=None, encoding=None, invert=False, output_mode=index_lookup.INT, sparse=False, pad_to_max_tokens=False, **kwargs )
Used in the notebooks
|Used in the guide||Used in the tutorials|
This layer translates a set of arbitrary strings into an integer output via a table-based vocabulary lookup.
The vocabulary for the layer can be supplied on construction or learned via
adapt(), the layer will analyze a data set, determine the
frequency of individual strings tokens, and create a vocabulary from them. If
the vocabulary is capped in size, the most frequent tokens will be used to
create the vocabulary and all others will be treated as out-of-vocabulary
There are two possible output modes for the layer.
output_mode is "int",
input strings are converted to their index in the vocabulary (an integer).
output_mode is "binary", "count", or "tf-idf", input strings
are encoded into an array where each dimension corresponds to an element in
The vocabulary can optionally contain a mask token as well as an OOV token
(which can optionally occupy multiple indices in the vocabulary, as set
The position of these tokens in the vocabulary is fixed. When
"int", the vocabulary will begin with the mask token at index 0, followed by
OOV indices, followed by the rest of the vocabulary. When
"binary", "count", or "tf-idf" the vocabulary will begin with OOV indices and
instances of the mask token will be dropped.
||The maximum size of the vocabulary for this layer. If None, there is no cap on the size of the vocabulary. Note that this size includes the OOV and mask tokens. Default to None.|
The number of out-of-vocabulary tokens to use. If this
value is more than 1, OOV inputs are hashed to determine their OOV value.
If this value is 0, OOV inputs will map to -1 when
A token that represents masked inputs. When
Only used when
||An optional list of tokens, or a path to a text file containing a vocabulary to load into this layer. The file should contain one token per line. If the list or file contains the same token multiple times, an error will be thrown.|
Only valid when
||Specification for the output of the layer. Defaults to "int". Values can be "int", "binary", "count", or "tf-idf" configuring the layer as follows: "int": Return the raw integer indices of the input tokens. "binary": Outputs a single int array per sample, of either vocab_size or max_tokens size, containing 1s in all elements where the token mapped to that index exists at least once in the sample. "count": Like "binary", but the int array contains a count of the number of times the token at that index appeared in the sample. "tf-idf": As "binary", but the TF-IDF algorithm is applied to find the value in each token slot.|
Only applicable when |