Splitting

tf.string_split(source, delimiter=' ')

Split elements of source based on delimiter into a SparseTensor.

Let N be the size of source (typically N will be the batch size). Split each element of source based on delimiter and return a SparseTensor containing the splitted tokens. Empty tokens are ignored.

If delimiter is an empty string, each element of the source is split into individual strings, each containing one byte. (This includes splitting multibyte sequences of UTF-8.) If delimiter contains multiple bytes, it is treated as a set of delimiters with each considered a potential split point.

For example: N = 2, source[0] is 'hello world' and source[1] is 'a b c', then the output will be

st.indices = [0, 0; 0, 1; 1, 0; 1, 1; 1, 2] st.shape = [2, 3] st.values = ['hello', 'world', 'a', 'b', 'c']

Args:
  • source: 1-D string Tensor, the strings to split.
  • delimiter: 0-D string Tensor, the delimiter character, the string should be length 0 or 1.
Raises:
  • ValueError: If delimiter is not a string.
Returns:

A SparseTensor of rank 2, the strings split according to the delimiter. The first column of the indices corresponds to the row in source and the second column corresponds to the index of the split component in this row.


tf.substr(input, pos, len, name=None)

Return substrings from Tensor of strings.

For each string in the input Tensor, creates a substring starting at index pos with a total length of len.

If len defines a substring that would extend beyond the length of the input string, then as many characters as possible are used.

If pos is negative or specifies a character index larger than any of the input strings, then an InvalidArgumentError is thrown.

pos and len must have the same shape, otherwise a ValueError is thrown on Op creation.

NOTE: Substr supports broadcasting up to two dimensions. More about broadcasting here


Examples

Using scalar pos and len:

input = [b'Hello', b'World']
position = 1
length = 3

output = [b'ell', b'orl']

Using pos and len with same shape as input:

input = [[b'ten', b'eleven', b'twelve'],
         [b'thirteen', b'fourteen', b'fifteen'],
         [b'sixteen', b'seventeen', b'eighteen']]
position = [[1, 2, 3],
            [1, 2, 3],
            [1, 2, 3]]
length =   [[2, 3, 4],
            [4, 3, 2],
            [5, 5, 5]]

output = [[b'en', b'eve', b'lve'],
          [b'hirt', b'urt', b'te'],
          [b'ixtee', b'vente', b'hteen']]

Broadcasting pos and len onto input:

input = [[b'ten', b'eleven', b'twelve'],
         [b'thirteen', b'fourteen', b'fifteen'],
         [b'sixteen', b'seventeen', b'eighteen'],
         [b'nineteen', b'twenty', b'twentyone']]
position = [1, 2, 3]
length =   [1, 2, 3]

output = [[b'e', b'ev', b'lve'],
          [b'h', b'ur', b'tee'],
          [b'i', b've', b'hte'],
          [b'i', b'en', b'nty']]

Broadcasting input onto pos and len:

input = b'thirteen'
position = [1, 5, 7]
length =   [3, 2, 1]

output = [b'hir', b'ee', b'n"]
Args:
  • input: A Tensor of type string. Tensor of strings
  • pos: A Tensor. Must be one of the following types: int32, int64. Scalar defining the position of first character in each substring
  • len: A Tensor. Must have the same type as pos. Scalar defining the number of characters to include in each substring
  • name: A name for the operation (optional).
Returns:

A Tensor of type string. Tensor of substrings