Join the SIG TFX-Addons community and help make TFX even better!


Find the token count of each document/row.

tokens is either a RaggedTensor or SparseTensor, representing tokenized strings. This function simply returns size of each row, so the dtype is not constrained to string.


sparse = tf.SparseTensor(indices=[[0, 0], [0, 1], [2, 2]],
                         values=['a', 'b', 'c'], dense_shape=(4, 4))
<tf.Tensor: shape=(4,), dtype=int64, numpy=array([2, 0, 1, 0])>

tokens either (1) a SparseTensor, or (2) a RaggedTensor with ragged rank of 1, non-ragged rank of 1 of dtype tf.string containing tokens to be counted
name (Optional) A name for this operation.

A one-dimensional Tensor the token counts of each row.

ValueError if tokens is neither sparse nor ragged