TF 2.0 is out! Get hands-on practice at TF World, Oct 28-31. Use code TF20 for 20% off select passes. Register now

tfds.features.Tensor

View source

Class Tensor

FeatureConnector for generic data of arbitrary shape and type.

Inherits From: FeatureConnector

__init__

View source

__init__(
    shape,
    dtype
)

Construct a Tensor feature.

Properties

dtype

Return the dtype (or dict of dtype) of this FeatureConnector.

shape

Return the shape (or dict of shape) of this FeatureConnector.

Methods

decode_example

View source

decode_example(tfexample_data)

Decode the feature dict to TF compatible input.

Args:

  • tfexample_data: Data or dictionary of data, as read by the tf-example reader. It correspond to the tf.Tensor() (or dict of tf.Tensor()) extracted from the tf.train.Example, matching the info defined in get_serialized_info().

Returns:

  • tensor_data: Tensor or dictionary of tensor, output of the tf.data.Dataset object

encode_example

View source

encode_example(example_data)

See base class for details.

get_serialized_info

View source

get_serialized_info()

Return the shape/dtype of features after encoding (for the adapter).

The FileAdapter then use those information to write data on disk.

This function indicates how this feature is encoded on file internally. The DatasetBuilder are written on disk as tf.train.Example proto.

Ex:

return {
    'image': tfds.features.TensorInfo(shape=(None,), dtype=tf.uint8),
    'height': tfds.features.TensorInfo(shape=(), dtype=tf.int32),
    'width': tfds.features.TensorInfo(shape=(), dtype=tf.int32),
}

FeatureConnector which are not containers should return the feature proto directly:

return tfds.features.TensorInfo(shape=(64, 64), tf.uint8)

If not defined, the retuned values are automatically deduced from the get_tensor_info function.

Returns:

  • features: Either a dict of feature proto object, or a feature proto object

get_tensor_info

View source

get_tensor_info()

See base class for details.

load_metadata

View source

load_metadata(
    data_dir,
    feature_name
)

Restore the feature metadata from disk.

If a dataset is re-loaded and generated files exists on disk, this function will restore the feature metadata from the saved file.

Args:

  • data_dir: str, path to the dataset folder to which save the info (ex: ~/datasets/cifar10/1.2.0/)
  • feature_name: str, the name of the feature (from the FeaturesDict key)

save_metadata

View source

save_metadata(
    data_dir,
    feature_name
)

Save the feature metadata on disk.

This function is called after the data has been generated (by _download_and_prepare) to save the feature connector info with the generated dataset.

Some dataset/features dynamically compute info during _download_and_prepare. For instance:

  • Labels are loaded from the downloaded data
  • Vocabulary is created from the downloaded data
  • ImageLabelFolder compute the image dtypes/shape from the manual_dir

After the info have been added to the feature, this function allow to save those additional info to be restored the next time the data is loaded.

By default, this function do not save anything, but sub-classes can overwrite the function.

Args:

  • data_dir: str, path to the dataset folder to which save the info (ex: ~/datasets/cifar10/1.2.0/)
  • feature_name: str, the name of the feature (from the FeaturesDict key)