text.normalize_utf8_with_offsets_map
Stay organized with collections
Save and categorize content based on your preferences.
Normalizes each UTF-8 string in the input tensor using the specified rule.
text.normalize_utf8_with_offsets_map(
input, normalization_form='NFKC', name=None
)
Returns normalized strings and an offset map used by another operation to map
post-normalized string offsets to pre-normalized string offsets.
See http://unicode.org/reports/tr15/
Examples:
# input: <string>[num_strings]
normalize_utf8_with_offsets_map(["株式会社", "KADOKAWA"])
# output: <string>[num_strings], <variant>[num_strings]
NormalizeUTF8WithOffsetsMap(output=<tf.Tensor: shape=(2,), dtype=string,
numpy=
array([b'\xe6\xa0\xaa\xe5\xbc\x8f\xe4\xbc\x9a\xe7\xa4\xbe', b'KADOKAWA'],
dtype=object)>, offsets_map=<tf.Tensor: shape=(2,), dtype=variant,
numpy=<unprintable>>)
Args |
input
|
A Tensor or RaggedTensor of type string. (Must be UTF-8.)
|
normalization_form
|
One of the following string values ('NFC', 'NFKC',
'NFD', 'NFKD'). Default is 'NFKC'. NOTE: NFD and NFKD for
normalize_utf8_with_offsets_map will not be available until the
tf.text release w/ ICU 69 (scheduled after 4/2021).
|
name
|
The name for this op (optional).
|
Returns |
A tuple of (results, offsets_map) where:
|
results
|
A Tensor or RaggedTensor of type string, with normalized
contents.
|
offsets_map
|
A Tensor or RaggedTensor of type variant , used to map
the post-normalized string offsets to pre-normalized string offsets. It
has the same shape as the results tensor. offsets_map is an input to
find_source_offsets op.
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-12-20 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-12-20 UTC."],[],[]]